Slope panorama three-dimensional reconstruction method based on unmanned aerial vehicle group collaborative aerial photography

Abstract

The invention relates to the technical field of side slope monitoring and three-dimensional reconstruction, and discloses a side slope panoramic three-dimensional reconstruction method based on unmanned aerial vehicle group collaborative aerial photography, which comprises the following steps: S1, unmanned aerial vehicle group task planning, S2, collaborative aerial photography data acquisition, S3, multi-source data preprocessing, denoising, time synchronization and coordinate calibration of the acquired multi-source data, and S4, multi-source data preprocessing. The method comprises the following steps: S1, preprocessing data to obtain a standardized data set, S4, generating and registering point clouds, generating initial point clouds through an image matching algorithm based on the preprocessed data, and fusing data of multiple unmanned aerial vehicles by adopting a point cloud registration algorithm, and S5, constructing a panoramic three-dimensional model, performing surface reconstruction and texture mapping on the fused point clouds, and generating the panoramic three-dimensional model of the slope. The master-slave unmanned aerial vehicle group can flexibly divide the aerial photography area according to the slope terrain (such as multiple inflection points and high and steep areas), the slave unmanned aerial vehicle goes deep into the local area which is difficult to cover by the master unmanned aerial vehicle, 100% dead-corner-free data acquisition of the slope is realized, and the problem of missed photography of the traditional single unmanned aerial vehicle is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of slope monitoring and three-dimensional reconstruction, in particular to a slope panoramic three-dimensional reconstruction method based on cooperative aerial photography of a UAV group. BACKGROUND

[0002] Slope is a common geological structure in engineering construction, and its stability directly affects engineering safety. At present, slope monitoring and modeling mainly rely on aerial photography technology, that is, a camera is carried by a UAV to collect slope images, and then a three-dimensional model is generated based on image data to provide data support for slope stability analysis and risk warning. In the prior art, a single UAV is the mainstream equipment for slope aerial photography, which is easy to operate and has low cost, and has been widely used in small and medium-sized slope modeling, such as daily inspection of highway slopes and local modeling of small mines.

[0003] The existing technology still has the following problems:

[0004] 1. Single UAV aerial photography has low efficiency and limited coverage. For a large slope with a length of more than 5 km and a height of more than 300 m, a single UAV needs to fly back and forth more than 10 times, and the single operation time is more than 8 hours, which is difficult to meet the demand of engineering for rapid slope modeling.

[0005] 2. The single-source data splicing error is large, and the three-dimensional model precision is insufficient. The single UAV only relies on visible light image data collection, and the image matching is easily affected by light changes and single slope surface texture (such as bare rock slope), resulting in fault or distortion in the overlapping area of point cloud splicing. SUMMARY

[0006] The purpose of the present application is to provide a slope panoramic three-dimensional reconstruction method based on cooperative aerial photography of a UAV group to solve the problems in the background art.

[0007] To solve the above technical problems, the present application is realized by the following technical scheme:

[0008] The present application is a slope panoramic three-dimensional reconstruction method based on cooperative aerial photography of a UAV group, comprising the following steps:

[0009] S1. UAV group task planning, according to the topographic features of the slope and the reconstruction accuracy requirement, the aerial photography area is divided and the cooperative flight path of each UAV is generated.

[0010] S2. Cooperative aerial photography data collection, the UAV group executes the aerial photography task according to the planned flight path, and synchronously collects the image data, position data and attitude data of the slope.

[0011] S3. Multi-source data preprocessing, the collected multi-source data is denoised, time-synchronized and coordinate-calibrated to obtain a standardized data set.

[0012] S4. Point cloud generation and registration, based on preprocessed data, generate initial point cloud through image matching algorithm, and realize fusion of multi-unmanned aerial vehicle data by using point cloud registration algorithm.

[0013] S5. Panoramic three-dimensional model construction, surface reconstruction and texture mapping are performed on the fused point cloud to generate a panoramic three-dimensional model of the slope.

[0014] Further, the unmanned aerial vehicle group task planning in step S1 further comprises: determining the role division of the master unmanned aerial vehicle and the slave unmanned aerial vehicle, the master unmanned aerial vehicle is responsible for global route control, and the slave unmanned aerial vehicle is responsible for aerial photography supplement of local detail area.

[0015] Further, the cooperative aerial photography data collection in step S2 adopts a master-slave communication protocol, the master unmanned aerial vehicle sends position correction instructions to the slave unmanned aerial vehicle in real time through the RTK positioning module, ensures seamless connection of aerial photography areas of each unmanned aerial vehicle, and the position synchronization accuracy is ≤0.5 m.

[0016] Further, the image data collected in step S2 comes from a double-lens camera carried by the unmanned aerial vehicle, including a visible light high-definition camera and a structured light camera, wherein the structured light camera is used to obtain depth information of the slope surface.

[0017] Further, the multi-source data preprocessing in step S3 comprises: removing image noise by using a Gaussian filtering algorithm, aligning timestamps of image and position data based on IMU (inertial measurement unit) data, and eliminating lens distortion error by camera intrinsic parameter calibration.

[0018] Further, the image matching algorithm in step S4 is a bundle adjustment method (Bundle Adjustment), which is used to optimize the matching accuracy of image feature points, and the point cloud registration algorithm is an iterative closest point algorithm (ICP), which is used to eliminate spatial deviation between point clouds of multiple unmanned aerial vehicles.

[0019] Further, the surface reconstruction in step S5 adopts a Poisson surface reconstruction algorithm, generates a continuous slope surface model based on the fused point cloud, and the texture mapping adopts an image projection technology, which is used to attach the preprocessed high-definition image to the surface model to restore the real appearance of the slope.

[0020] Further, step S1 further comprises fault redundancy planning: when any slave unmanned aerial vehicle fails, the master unmanned aerial vehicle automatically re-divides the aerial photography area of the remaining unmanned aerial vehicles to ensure that the aerial photography task is not interrupted.

[0021] Further, it further comprises step S6: three-dimensional model precision verification, by laying slope ground control points, comparing the coordinates of corresponding points in the three-dimensional model with the ground measured coordinates, if the error is >0.1 m, then returning to step S4 to re-perform point cloud registration.

[0022] Further, the panoramic three-dimensional model generated in step S5 supports dynamic updating: periodically controlling the UAV group to locally re-shoot the slope, and fusing the newly collected data into the original model to realize dynamic monitoring of the slope deformation.

[0023] The present application has the following advantages:

[0024] (1) The UAV group shortens the operation time to 1 / 3 of the traditional single UAV through parallel aerial photography, for example, for a 5km long large mine slope, 5 UAVs are used for cooperative operation, and only 1.5 hours are needed to complete data collection, meeting the engineering rapid modeling requirements.

[0025] (2) The master-slave UAV group can flexibly divide the aerial photography area according to the slope terrain (such as multiple inflection points and high and steep areas), and the UAVs can penetrate into the local area that is difficult for the master UAV to cover, realizing 100% data collection of the slope without dead angle and avoiding the missing shooting problem of the traditional single UAV.

[0026] (3) The present application fuses visible light images, structured light depth data and IMU attitude data, and the multi-source data is complementary to reduce the matching error caused by single texture and light, and the three-dimensional model coordinate error is controlled within 0.1m, which is four to six times higher than the single UAV model accuracy.

[0027] (4) The present application can obtain at least three different angle image data in high and steep slope area through multi-UAV multi-view data collection, and the traditional single UAV model can be filled in the hollow when generating point cloud, and the steepness, depression and other detailed features of the slope can be accurately restored.

[0028] (5) When any slave UAV fails, the master UAV automatically reallocates the flight path without interrupting the task, and the task completion rate is improved by 100% compared with the task termination caused by single UAV failure.

[0029] (6) The panoramic three-dimensional model can be dynamically updated through periodic local re-shooting, real-time capturing of slope deformation (such as crack expansion and local collapse), providing long-term data support for slope stability warning, and breaking through the limitation of traditional static modeling.

[0030] Of course, any product implementing the present application does not necessarily need to achieve all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0032] Figure 1 The method flow chart of the present application. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the present application.

[0034] Please refer to Figure 1 The present application is a slope panoramic three-dimensional reconstruction method based on unmanned aerial vehicle group cooperative aerial photography, which comprises the following steps:

[0035] S1. Unmanned aerial vehicle group task planning, according to the topographic features of the slope and the reconstruction accuracy requirement, the aerial photography area is divided and the cooperative flight path of each unmanned aerial vehicle is generated, wherein the unmanned aerial vehicle group task planning further comprises: determining the role division of the master unmanned aerial vehicle and the slave unmanned aerial vehicle, the master unmanned aerial vehicle is responsible for the global flight path control, and the slave unmanned aerial vehicle is responsible for the aerial photography supplement of the local detail area, in addition, the fault redundancy planning is also included: when any slave unmanned aerial vehicle fails, the master unmanned aerial vehicle automatically redivides the aerial photography area of the remaining unmanned aerial vehicles to ensure that the aerial photography task is not interrupted.

[0036] S2. Cooperative aerial photography data acquisition, the unmanned aerial vehicle group executes the aerial photography task according to the planned flight path, and synchronously acquires the image data, position data and attitude data of the slope, wherein the cooperative aerial photography data acquisition adopts a master-slave communication protocol, the master unmanned aerial vehicle sends position correction instructions to the slave unmanned aerial vehicles in real time through the RTK positioning module, so as to ensure the seamless connection of the aerial photography areas of each unmanned aerial vehicle, and the position synchronization accuracy is ≤0.5m, in addition, the acquired image data comes from the double-lens camera carried by the unmanned aerial vehicle, including a visible light high-definition camera and a structured light camera, wherein the structured light camera is used to obtain the depth information of the slope surface.

[0037] S3. Multi-source data preprocessing, the collected multi-source data is denoised, time-synchronized and coordinate-calibrated to obtain a standardized data set, the multi-source data preprocessing comprises: removing image noise by using a Gaussian filtering algorithm, aligning the timestamps of the image and the position data based on the IMU inertial measurement unit data, and eliminating lens distortion error through camera intrinsic parameter calibration.

[0038] S4. Point cloud generation and registration, based on preprocessed data, generate initial point cloud through image matching algorithm, and then use point cloud registration algorithm to realize fusion of multi-unmanned aerial vehicle data, wherein the image matching algorithm is bundle adjustment, which is used to optimize the matching accuracy of image feature points; the point cloud registration algorithm is iterative closest point algorithm (ICP), which is used to eliminate the spatial offset between multi-unmanned aerial vehicle point clouds.

[0039] S5. Panoramic three-dimensional model construction, surface reconstruction and texture mapping are performed on the fused point cloud to generate a panoramic three-dimensional model of the slope, the surface reconstruction adopts Poisson surface reconstruction algorithm, and a continuous slope surface model is generated based on the fused point cloud, the texture mapping adopts image projection technology, and the preprocessed high-definition image is attached to the surface model to restore the real appearance of the slope, in addition, the generated panoramic three-dimensional model supports dynamic updating: periodically control the unmanned aerial vehicle group to locally re-shoot the slope, and integrate the newly collected data into the original model to realize dynamic monitoring of the slope deformation.

[0040] In use, first, the ground control center completes the task planning of the unmanned aerial vehicle group according to the terrain data of the slope and the accuracy requirement, and clearly defines the flight route and division of labor of the master and slave unmanned aerial vehicles, laying the foundation for cooperative aerial photography.

[0041] Secondly, the master unmanned aerial vehicle controls the slave unmanned aerial vehicles to synchronously collect multi-source data (visible light, structured light, IMU) according to the flight route through RTK positioning and real-time communication, ensures that the data covers no dead angle and is positionally synchronous, and then denoises, time-synchronizes and coordinates calibrates the collected multi-source data to eliminate data errors and obtain standardized data.

[0042] Then, initial point cloud is generated through bundle adjustment, and multi-unmanned aerial vehicle point cloud fusion is realized through ICP algorithm, solving the problem of large splicing error of single-source data.

[0043] Finally, surface reconstruction and texture mapping are performed based on the fused point cloud to generate a panoramic three-dimensional model, and the accuracy is verified through ground control points, and the model is dynamically updated when necessary, through the cooperation of the unmanned aerial vehicle group and the complementation of multi-source data, the efficiency and accuracy bottleneck of traditional single unmanned aerial vehicle is broken through, and an efficient and reliable technical solution is provided for slope modeling and monitoring.

[0044] Embodiment

[0045] The present application also provides the following embodiments: taking 8km long mine slope panoramic three-dimensional reconstruction of a 5-unmanned aerial vehicle group (1 master and 4 slaves) as an example, the specific execution process of the method of the present application is described in detail:

[0046] 1. UAV fleet configuration: the master UAV is selected as DJI M300RTK, equipped with a visible light high-definition camera (resolution 5472x3648) and an RTK positioning module (positioning accuracy 1 cm+1 ppm); 4 slave UAVs are selected as DJI Mavic3Enterprise, each equipped with a structured light camera (depth measurement range 0.510 m) and an IMU inertial measurement unit (attitude accuracy 0.1°); the ground control center uses an industrial computer, installed with self-developed UAV fleet cooperative control software and three-dimensional reconstruction software.

[0047] 2. UAV fleet task planning (step S1):

[0048] Import the DEM terrain data of the mine slope (resolution 10 m), and set the reconstruction accuracy requirement to 0.15 m;

[0049] Divide the 8 km long and 500 m high slope into 5 aerial photography areas, each area is about 1.6 km², and the overlap degree between areas is 5%;

[0050] Generate master UAV flight route: flight height 150 m, route spacing 8 m, fly along the slope strike; generate slave UAV flight route: 2 slave UAVs are responsible for the top area of the slope (flight height 120 m), 2 are responsible for the bottom area of the slope (flight height 100 m), and the routes are perpendicular to the master UAV flight route to achieve multi-view coverage;

[0051] Set the master-slave communication frequency to 1 Hz, and the master UAV sends position correction instructions to the slave UAV every 1 second.

[0052] 3. Cooperative aerial photography data collection (step S2):

[0053] The ground control center sends the take-off command, the master UAV first rises to a height of 150 m, and after completing the RTK positioning initialization, instructs the 4 slave UAVs to rise;

[0054] The UAV fleet flies according to the planned route, the master UAV takes a visible light image every 0.5 seconds, the slave UAV synchronously takes a visible light image and a set of structured light depth data every 0.3 seconds, and the IMU records the attitude data (roll angle, pitch angle, yaw angle) of each UAV in real time;

[0055] During the collection process, one of the slave UAVs reports a low battery alarm (remaining battery 15%), the master UAV automatically divides half of the bottom area it is responsible for to the adjacent slave UAV, and the adjusted route still maintains an area overlap degree of 5%, the task is not interrupted;

[0056] After the aerial photography task is completed, the UAV fleet returns in turn, and the collected data is transmitted to the ground control center in real time through the 4G link, and the total collection time is 2.5 hours.

[0057] 4. Multi-source data preprocessing (step S3):

[0058] Image denoising: all visible light images are processed using a 3x3 Gaussian filter algorithm to eliminate noise caused by flight jitter;

[0059] Time synchronization: based on the timestamp of the IMU data, align the image data, position data (latitude, longitude, height), and attitude data at the same time, with an error of within 10ms;

[0060] Coordinate calibration: through camera intrinsic calibration board (checkerboard calibration board, precision 0.01m), eliminate the lens distortion of master-slave unmanned aerial vehicle cameras, get the mapping relationship between standardized image coordinate system and geodetic coordinate system.

[0061] 5. Point cloud generation and registration (step S4):

[0062] Initial point cloud generation: using bundle adjustment method to match feature points of preprocessed image data, extract SIFT feature points (about 2000 per image) of each image, generate initial point cloud of each aerial area by triangulation principle, point cloud density is about 80 points / m²;

[0063] Point cloud registration: using ICP algorithm to fuse the initial point clouds of the five regions, taking the master unmanned aerial vehicle point cloud as the reference, iteratively optimizing the spatial position of the slave unmanned aerial vehicle point cloud, until the average distance error between point clouds is <0.05m, to get the global fused point cloud.

[0064] 6. Panoramic three-dimensional model construction (step S5):

[0065] Surface reconstruction: using Poisson surface reconstruction algorithm to grid the fused point cloud, generating a triangular mesh model with a grid resolution of 0.1m, restoring the surface relief features of the slope;

[0066] Texture mapping: according to the mapping relationship between the image coordinate system and the grid model, the preprocessed high-definition visible light images are attached to the triangular mesh surface to generate a panoramic three-dimensional model with real texture, with a model resolution of 5472x3648.

[0067] 7. Three-dimensional model accuracy verification (step S6):

[0068] Lay out 10 control points on the slope ground (use total station to measure, coordinate accuracy 0.005m);

[0069] Extract the coordinates of the 10 control points in the three-dimensional model and compare them with the ground measured coordinates, with an average error of 0.12m, meeting the accuracy requirement of 0.15m, and the model verification is passed.

[0070] According to the above embodiments, the present application further relates to the following examples for further detailed description:

[0071] Example 1

[0072] For panoramic three-dimensional reconstruction of highway slope, for a section of slope of a certain expressway, the length is 2km, the height is 300m, the slope surface is mainly bare rock, there are 3 high and steep areas (slope > 60°), a panoramic three-dimensional model with a precision of 0.1m needs to be constructed for slope crack monitoring, specifically:

[0073] UAV group configuration: 1 master and 2 slave UAVs, the master UAV is equipped with RTK and high-definition camera, and the slave UAV is equipped with structured light camera;

[0074] Task planning: the slope is divided into 3 areas, the master UAV is responsible for the middle gentle area, and the slave UAV is responsible for the high and steep area on both sides, the flight height is 100m, and the route interval is 5m;

[0075] Data collection: total time 1.5 hours, 2000 visible light images and 1500 structured light data are collected;

[0076] Model results: the three-dimensional model has no "hollow", the cracks (width 5mm) in the high and steep area are clearly visible, the average error of the control points is 0.08m, and the precision is improved by 60% compared with the traditional single UAV (error 0.2m)

[0077] Example 2

[0078] For panoramic three-dimensional reconstruction of water conservancy slope, for a reservoir dam slope, the length is 1.5km, the height is 200m, the slope surface is covered with part of vegetation, a dynamic updated panoramic three-dimensional model needs to be constructed for monitoring the deformation of the slope during the reservoir impoundment period, specifically:

[0079] UAV group configuration: 1 master and 3 slave UAVs, the master UAV is equipped with a millimeter wave radar module (for penetrating vegetation), and the slave UAV is equipped with a thermal imaging camera (for identifying bare rock areas under vegetation coverage);

[0080] Task planning: 4 aerial photography areas are divided, the master UAV route is along the dam trend, and the slave UAV route is perpendicular to the dam, the flight height is 80m, and the route interval is 6m;

[0081] Dynamic updating: after the first modeling, every month, 2 slave UAVs are used to re-shoot the key areas (dam body and slope junction) of the slope, and the new data is integrated into the original model;

[0082] Application effect: continuous 6-month monitoring found that there was a 0.05m transverse displacement at the slope junction, timely deformation warning was issued, and danger was avoided, the dynamic updating efficiency of the model was improved by 70% compared with the traditional single UAV.

[0083] The preferred embodiments of the application disclosed above are only to facilitate the elucidation of the application. The preferred embodiments do not describe all the details of the application and limit the application to the specific embodiments described. Obviously, many modifications and variations can be made in light of the teachings above. The description is chosen and described in order to best explain the principles of the application and its practical application to thereby enable others skilled in the art to best utilize the application and get the best results from the application. The application is only limited by the claims and their full scope and equivalents.

Claims

1. A method for slope panoramic three-dimensional reconstruction based on cooperation of a UAV group, characterized in that, The method comprises the following steps: S1. UAV group task planning, according to the terrain characteristics of the slope and the reconstruction accuracy requirement, the aerial photography area is divided and the cooperative flight path of each UAV is generated; S2. Cooperative aerial photography data collection, the UAV group executes the aerial photography task according to the planned flight path, and synchronously collects image data, position data and attitude data of the slope; S3. Multi-source data preprocessing, the collected multi-source data is denoised, time-synchronized and coordinate-calibrated to obtain a standardized data set; S4. Point cloud generation and registration, based on the preprocessed data, an initial point cloud is generated through an image matching algorithm, and a point cloud registration algorithm is used to realize the fusion of multi-UAV data; S5. Panoramic three-dimensional model construction, the fused point cloud is surface reconstructed and texture mapped to generate a panoramic three-dimensional model of the slope.

2. The slope panoramic three-dimensional reconstruction method based on UAV swarm cooperative aerial photography according to claim 1, characterized in that, The UAV group task planning in step S1 further comprises: determining the role division of the master UAV and the slave UAV, the master UAV is responsible for global flight path control, and the slave UAV is responsible for aerial photography supplement of local detail area. 3.The slope panoramic three-dimensional reconstruction method based on UAV swarm cooperative aerial photography according to claim 1, characterized in that, The cooperative aerial photography data collection in step S2 adopts a master-slave communication protocol, the master UAV sends position correction instructions to the slave UAV in real time through an RTK positioning module, ensures seamless connection of aerial photography areas of each UAV, and the position synchronization accuracy is ≤0.5 m.

4. The slope panoramic three-dimensional reconstruction method based on UAV swarm cooperative aerial photography according to claim 1, characterized in that, The image data collected in step S2 is collected from a double-lens camera carried by the UAV, including a visible light high-definition camera and a structured light camera, wherein the structured light camera is used to obtain the depth information of the slope surface. 5.The slope panoramic three-dimensional reconstruction method based on UAV swarm cooperative aerial photography according to claim 1, characterized in that, The multi-source data preprocessing in step S3 comprises: removing image noise by using a Gaussian filtering algorithm, aligning the time stamps of the image and the position data based on IMU (inertial measurement unit) data, and eliminating lens distortion error through camera intrinsic parameter calibration. 6.The slope panoramic three-dimensional reconstruction method based on UAV swarm cooperative aerial photography according to claim 1, characterized in that, The image matching algorithm in step S4 is a bundle adjustment method, which is used to optimize the matching accuracy of image feature points, and the point cloud registration algorithm is an iterative closest point algorithm (ICP), which is used to eliminate the spatial deviation between multi-UAV point clouds.

7. The slope panoramic three-dimensional reconstruction method based on UAV swarm cooperative aerial photography according to claim 1, characterized in that, The surface reconstruction in step S5 adopts a Poisson surface reconstruction algorithm, generates a continuous slope surface model based on the fused point cloud, and the texture mapping adopts an image projection technology, which is used to paste the preprocessed high-definition image to the surface model and restore the real appearance of the slope. 8.The method of claim 2, wherein, Step S1 further comprises fault redundancy planning: when any slave UAV fails, the master UAV automatically redivides the aerial photography area of the remaining UAVs to ensure that the aerial photography task is not interrupted. 9.The slope panoramic three-dimensional reconstruction method based on UAV swarm cooperative aerial photography according to claim 1, characterized in that, It further comprises step S6: three-dimensional model accuracy verification, by laying out slope ground control points, comparing the coordinates of the corresponding points in the three-dimensional model with the ground measured coordinates, if the error is >0.1 m, then returning to step S4 to re-perform point cloud registration. 10.The slope panoramic three-dimensional reconstruction method based on the UAV swarm cooperative aerial photography according to claim 1, characterized in that, The panoramic three-dimensional model generated in step S5 supports dynamic updating: periodically controlling the UAV group to re-photograph the local slope, integrating the newly collected data into the original model, and realizing dynamic monitoring of slope deformation.

Images