A rapid modeling method and system for an unfamiliar environment based on multi-UAV collaboration

Abstract

The present invention discloses a method and system for rapid modeling of unfamiliar environments based on multi-UAV collaboration, which at least includes a flight path planning module, a position information acquisition module, an image rapid stitching module, and a 3D modeling processing module. The 3D model includes a basic 3D model and a fine 3D model. The UAV collects data during the aerial photography flight path according to the aerial photography flight path planned by the flight path planning module through the position information acquisition module, stitches the aerial photography images through the image rapid stitching module, and then constructs a 3D model through the 3D modeling processing module to achieve rapid modeling of unfamiliar environments. This method and system use a UAV equipped with an oblique camera and a hybrid modeling method based on real-time aerial photography images, a basic 3D model, and a fine 3D model to achieve rapid mapping of unfamiliar areas, improve the ability of rapid 3D modeling of unfamiliar and complex environments, and have far-reaching significance for the development of environmental rapid perception ability in future multi-scenarios.

Description

Technical Field

[0001] The present invention belongs to the technical field of 3D modeling, and mainly relates to a rapid modeling method and system for an unfamiliar environment based on multi-UAV collaboration. Background Art

[0002] Environmental modeling technology has a very broad application prospect in the fields of urban construction, geographical mapping, travel tourism, etc. Its core is to collect environmental data and generate a 3D environmental model through processing. Currently, common means mainly include manual surveying and mapping modeling, laser scanning modeling, aerial photogrammetry modeling, oblique photogrammetry modeling, etc.; Western countries started earlier in environmental modeling, and are not only very mature in modeling systems and modeling software, but also integrate high-precision positioning of the global navigation satellite system to generate an overall 3D model of the environmental area; the United States has even applied this technology to military applications, using the high-resolution 3D terrain environment of the combat area to support ground training and mission rehearsal of troops.

[0003] The UAV aerial photography method based on satellite navigation and positioning has the advantages of strong mobility, high acquisition efficiency, accurate information perception, etc., and is widely used in 3D modeling in the current urban environment. However, in an unfamiliar environment, there are factors such as complex terrain and more uncertainties, which limit the existing modeling methods and systems, and it is difficult to meet the requirements of rapid and accurate acquisition in the mode environment. Related technologies urgently need to be broken through. Summary of the Invention

[0004] The present invention precisely aims at the problem that the existing modeling methods and systems are difficult to meet the requirements of rapid and accurate acquisition in the mode environment, and provides a rapid modeling method and system for an unfamiliar environment based on multi-UAV collaboration, which at least includes a route planning module, a position information acquisition module, an image rapid stitching module, and a 3D modeling processing module. The route planning module is used to realize the route planning of UAV aerial photography and the route planning of multi-UAV collaborative aerial photography; the position information acquisition module is used to realize the data acquisition of the real-time shooting position of the UAV; the image rapid stitching module is used to realize the real-time stitching of the aerial photography images transmitted back in real time; the 3D modeling processing module uses oblique photogrammetry data to realize the construction of the 3D model; the 3D model includes a basic 3D model and a fine 3D model; the UAV collects data during the aerial photography route according to the aerial photography route planned by the route planning module through the position information acquisition module, stitches the aerial photography images through the image rapid stitching module, and then constructs the 3D model through the 3D modeling processing module to realize the rapid modeling of the unfamiliar environment. This method and system use UAVs equipped with oblique cameras, and based on the hybrid modeling method of real-time aerial photography images, basic 3D models, and fine 3D models, realize the rapid mapping of unfamiliar areas, can improve the rapid 3D modeling ability of unfamiliar and complex environments, and have far-reaching significance for the development of the future rapid environmental perception ability in multiple scenarios.

[0005] To achieve the above object, the technical solution adopted by the present invention is: A rapid modeling system for unfamiliar environments based on multi-UAV collaboration, which at least includes a route planning module, a position information acquisition module, an image rapid stitching module, and a 3D modeling processing module.

[0006] The route planning module is used to implement the route planning for UAV aerial photography and the route planning for collaborative aerial photography of multiple UAVs.

[0007] The position information acquisition module is used to collect data on the real-time shooting position of the UAV.

[0008] The image rapid stitching module is used to perform real-time stitching of the aerial photography images transmitted back in real time.

[0009] The 3D modeling processing module uses oblique photogrammetry data to construct a 3D model; the 3D model includes a basic 3D model and a fine 3D model.

[0010] The UAV, according to the aerial photography route planned by the route planning module, collects data during the UAV aerial photography route through the position information acquisition module, stitches the aerial photography images through the image rapid stitching module, and then constructs a 3D model through the 3D modeling processing module to achieve rapid modeling of unfamiliar environments.

[0011] To achieve the above object, the technical solution further adopted by the present invention is: A rapid modeling method for unfamiliar environments based on multi-UAV collaboration, including the following steps:

[0012] S1: Use a single UAV to perform scanning shooting on the route during the route planning process to collect aerial photography image data of the target area.

[0013] S2: During the image acquisition process, perform Laplacian image fusion based on sift feature point detection and matching to achieve incremental update of the UAV aerial photography images and generate a quasi-real-time two-dimensional image.

[0014] S3: Perform grid calculation in the 3D modeling processing module based on the main module - worker thread module mode to achieve cluster processing of the modeling tasks.

[0015] S4: After the acquisition is completed, use the image dataset quickly collected by a single UAV to construct a modeling task, and quickly generate a basic 3D model based on step S3.

[0016] S5: In the basic 3D model, use the selected unfamiliar key target facilities to identify feature points, mark the important ground features as regions of interest, and generate corresponding point, line, and surface vector constraint information.

[0017] S6: Based on the vector constraint information of the basic 3D model identification and the aerial photography mission, use the MTSP algorithm to plan the adaptive flight path of the multi-aircraft aerial photography drones;

[0018] S7: Based on the adaptive flight path, multiple drones cooperate to complete the acquisition of the oblique photogrammetry data of the target area;

[0019] S8: Use the oblique photogrammetry data collected by multiple drones in cooperation to perform step S3 to complete the construction of the fine 3D model of the target area.

[0020] As an improvement of the present invention, in the step S1, when a single drone performs scanning shooting, the requirement for the image overlap degree is 60% - 65%.

[0021] As another improvement of the present invention, in the main module of the step S3, the 3D modeling task sequence is managed by managing the task sequence. The task sequence is directly managed through the Windows file explorer. The task sequence directory includes the following subdirectories: tasks cancelled by the user, completed tasks, list of all engines connected to the task sequence, tasks failed due to errors at the engine end, tasks waiting to be processed, tasks currently being processed; in the worker thread module, the main processing content includes aerial triangulation calculation or 3D reconstruction process.

[0022] As yet another improvement of the present invention, the planning of the adaptive flight path of the MTSP algorithm in the step S6 specifically includes: combining the oblique photography trajectory and the front-view trajectory point set into a point set S, inputting the number of drones k, and the calculation process: ① converting the longitude and latitude high point set into xyz coordinates; ② using K-means clustering to divide the point set into categories and limit the number of points in each category; ③ modeling each category of point set as a TSP problem and solving the shortest path using the improved greedy algorithm; ④ storing the shortest path of each category of point set as an ordered point set, that is, the path assigned to each drone; ⑤ obtaining the multi-aircraft path and downloading the executable JSON file.

[0023] Compared with the prior art, the present invention provides a method and system for rapid modeling in an unfamiliar environment based on multi-drone cooperation. Aiming at the 3D modeling requirements of unfamiliar environment areas, through the hybrid modeling method using aerial photography real-time images, basic 3D models, and fine 3D models, rapid modeling of complex unfamiliar areas is realized, the post-processing time of 3D models is reduced, the construction speed of unfamiliar environments is accelerated, and the rapid generation of 3D real scenes is supported.

[0024] In addition, the rapid modeling system for unfamiliar environments based on multi-UAV collaboration according to the present invention realizes the route optimization of multi-UAV collaborative aerial photography based on the feature identification and area of interest identification of the basic 3D model obtained from aerial photography, greatly reducing the number of redundant photos taken; real-time image stitching realizes the real-time stitching of images to quickly obtain the real scene of the unfamiliar area; the 3D modeling software module uses a cluster method to realize the rapid construction of 3D models, greatly shortening the modeling time. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flowchart of the steps of a method for rapid modeling of unfamiliar environments based on multi-UAV collaboration according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] The present invention will be further illustrated below in conjunction with the drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and not to limit the scope of the present invention.

[0027] Embodiment 1

[0028] A rapid modeling system for unfamiliar environments based on multi-UAV collaboration includes at least a route planning module, a position information acquisition module, an image rapid stitching module, and a 3D modeling processing module. The route planning module is used to implement the route planning of UAV aerial photography and the route planning of multi-UAV collaborative aerial photography; the position information acquisition module is used to implement the data acquisition of the real-time shooting position of the UAV; the image rapid stitching module is used to implement the real-time stitching of the aerial photography images transmitted back in real time; the 3D modeling processing module uses oblique photogrammetry data to implement the construction of 3D models; the 3D models include a basic 3D model and a fine 3D model. The basic 3D model is a 3D model constructed using the images collected by a single UAV during the rough flight stage, with an image resolution of (8-10 cm) and an image heading and side overlap of 60%-65%; the fine 3D model is a model constructed using the dataset jointly constructed by the images collected by multiple UAVs during the fine flight stage and the aerial photography images collected during the rough flight. The resolution of the images collected during the fine flight is (3-5 cm), and the image heading and side overlap of the images collected during the fine flight are 80%-85%.

[0029] Take the unfamiliar area to be collected on the map in the assigned task as the target area, and take the target area as an aerial photography task. First, dispatch a single unmanned aerial vehicle (UAV) for aerial photography to quickly collect the scenes of the target area and transmit the aerial photography data back to the image rapid stitching module in real time. Based on the real-time transmitted image data, perform real-time stitching to generate a quasi-real-time two-dimensional image of the target area. After the aerial photography is completed, use the three-dimensional modeling processing module cluster for processing to quickly generate a basic three-dimensional model. Based on the basic three-dimensional model, perform feature marking and region of interest recognition, and plan the collaborative adaptive aerial photography routes of multiple UAVs. Finally, generate a fine three-dimensional model of the target area based on the data of multi-UAV collaborative oblique photography to achieve rapid modeling of unfamiliar environments.

[0030] Embodiment 2

[0031] A method for rapid modeling of unfamiliar environments based on multi-UAV collaboration, as Figure 1 shown, includes the following steps:

[0032] S1: Use a single UAV to perform scanning shooting along the planned route, without requiring image overlap, and quickly collect aerial photography image data of the target area;

[0033] Before the UAV aerial photography, take the unfamiliar area to be collected on the map in the assigned task as the target area, install a high-precision positioning enhancement module on the UAV, and use the rapid modeling system for unfamiliar environments to perform initial route planning (without requiring overlap), and quickly collect aerial photography image data of the target area.

[0034] S2: During the image acquisition process, perform Laplacian image fusion based on sift feature point detection and matching to achieve incremental update of the UAV aerial photography images and generate a quasi-real-time two-dimensional image;

[0035] During the UAV aerial photography process, the UAV transmits the real-time aerial photography image data back to the specified directory of the server through the image transmission module. The real-time image stitching module in the rapid modeling system for unfamiliar environments obtains the real-time aerial photography images from the specified directory, and uses the sift feature point detection, feature point matching, and Laplacian image fusion algorithms to achieve real-time stitching of the aerial photography images. Along with the real-time transmission of the UAV aerial photography data, perform incremental update on the stitched UAV aerial photography images to generate a quasi-real-time two-dimensional image.

[0036] Image stitching is essentially to project each aerial photography image onto the space of the final stitched large image through a transformation matrix. Only by solving the transformation matrix for each image can the final stitching be completed. The point pairs on two images with different perspectives can be expressed by a projective transformation, that is: x1 = Hx2, where the matrix H is called the homography matrix, x1 and x2 are represented in the form of 3×1 homogeneous coordinates, and the homography matrix H is a 3×3 matrix with 8 degrees of freedom, because H33 If it is fixed as 1, then the transformation process can be written as follows:

[0037]

[0038] However, in the application scenario of stitching, it cannot be simply considered in this way. First of all, the point pairs here should be the matching feature points on the corresponding matching map. Usually, the number of matching points is very large, and here it is not simply considering the stitching of two pictures, but the stitching of several pictures. The optimal pairing result of the first two pictures may not be the optimal for the subsequent pictures. Therefore, in image stitching, the pairing error between all pairing points should be minimized, rather than directly solving a unique solution.

[0039] Suppose there are M pairs of matching feature points among N pictures. The transformation matrix of the i-th picture is vectorized into X i (Reshape the affine transformation matrix H i into a 1×9 vector), let Based on this, the objective function E(X) to be optimized is defined as:

[0040] E(X) = E cor (X) + λE reg (X)

[0041] Where E cor (X) is the registration error, and its main purpose is to minimize the distance difference after the corresponding registration feature points are projected onto the stitched large picture through the affine transformation matrix. The specific definition is as follows:

[0042]

[0043] E reg (X) in the objective function is a regularization term, and λ is a weight parameter. The purpose of setting this term is mainly to ensure that the final transformation distortion is as small as possible and ensure that its transformation is as close as possible to an Euclidean transformation, which is defined as the following formula:

[0044]

[0045]

[0046] Finally, by minimizing E(X), the optimal transformation matrix of each image can be obtained. By solving the transformation matrix for each picture and projecting each aerial photo into the geographic space, the final stitched large picture is finally completed.

[0047] S3: In the 3D modeling processing module, grid calculation is performed based on the master-module - worker-thread module mode to achieve the cluster processing of the modeling task;

[0048] Using the main module - worker thread module mode in the 3D modeling processing module, the modeling tasks are divided into several basic tasks, and a task sequence is submitted. The worker thread extracts the waiting tasks in the sequence. A cluster is built using multiple computers and associated with the task sequence to achieve multi - machine collaborative processing of 3D modeling tasks.

[0049] In the main module, the 3D modeling task sequence is managed by managing the task sequence. The task sequence is directly managed through the Windows File Explorer. The tasks are displayed as XML files in the task sequence. The task sequence directory includes the following sub - directories: tasks cancelled by users, completed tasks, list of all engines connected to the task sequence, tasks failed due to errors on the engine side, tasks waiting to be processed, and tasks currently being processed.

[0050] In the worker thread, the application engine in the cluster - processing computer runs in the background of the computer without interacting with the user. When the computer is idle, the engine extracts the waiting tasks in the task sequence according to the priority and submission date. By reading the task sequence defined in the main module, it processes different basic tasks, stores the engine processing results in the storage location defined in the main module, and updates the task content in the task sequence at the same time. The main processing content includes aerial triangulation calculation or 3D reconstruction process.

[0051] S4: After the acquisition is completed, use the image data set quickly acquired by a single UAV output in S1 to construct a 3D model reconstruction task, and quickly generate a basic 3D model based on step S3;

[0052] After a single UAV completes the aerial photography task along the route, the 3D modeling software module in the fast construction system for unfamiliar environments reads all the aerial photography data, performs aerial triangulation encryption, region of interest editing, and adaptive chunking of the image data based on the 3D modeling software cluster, completes the rapid reconstruction of the 3D scene, and generates a basic 3D model. The modeling task in the present invention is based on the image data set acquired in S1, and uses the cluster - processing mode in step S3 to complete the construction of the basic model. The basic requirement is the image data of a two - square - kilometer area, and the basic model is constructed by 4 graphic workstations with 64G of memory. The model construction time is less than 4h. The resolution of the basic 3D model depends on the ground resolution of the images acquired by UAV aerial photography (8 - 10 cm). The image modeling effect depends on the heading and side - looking overlap during the aerial photography process. The heading and side - looking overlap during acquisition are 60% - 65%.

[0053] S5: In the basic 3D model, use the selection of unfamiliar key target facilities to identify feature points, identify regions of interest for important ground features, and generate corresponding point, line, and surface vector constraint information;

[0054] Based on the basic 3D model data, feature identification and area of interest identification of the reconstructed 3D scene are carried out based on the target recognition algorithm. That is, key target facilities and important target areas are selected and identified in the 3D reconstruction scene, and the position information of the feature identification and the area of interest identification is stored to generate corresponding point, line, and plane vector constraint information.

[0055] The target recognition algorithm is a key area recognition technology based on DeepLab v3+. DeepLab v3+ is a semantic segmentation network proposed by Google and is the third enhanced version of its DeepLab series of semantic segmentation networks. When training the semantic segmentation network model for building recognition, the dataset used is the dataset of typical urban building instances in China. This dataset uses high-resolution remote sensing images as the data source and is constructed by combining manual annotation and interactive annotation. Since this dataset is an instance dataset, its label assigns different labels to each building. Here, it is binarized to only label the background and buildings without distinguishing specific buildings. All connected regions are obtained through edge detection, and then the largest circumscribed rectangle result is found among this series of edge detection results as the key building recognition area, and the recognition area is used as the key shooting area in the precise flight stage.

[0056] S6: Based on the vector constraint information (key shooting area) identified from the basic 3D model and the aerial photography task, the MTSP algorithm is used to plan the adaptive flight path of multiple aerial photography drones with constraints.

[0057] Taking the identification of the basic 3D model as the constraint information, the flight path planning module combines the constraint information based on the UAV formation planning algorithm and uses the MTSP algorithm to adaptively plan the flight path of the target area, that is, to achieve focused shooting of the key shooting area and key shooting targets (increasing the image overlap and resolution), and rapid shooting of general areas, greatly reducing the shooting of redundant photos.

[0058] The mTSP can generally be defined as follows: Given a set of nodes, there are m salespersons on a single node. The remaining nodes (cities) to be visited are called intermediate nodes. Then, the mTSP involves finding routes for all m salespersons, who all start and end at the initial node, such that each intermediate node is visited only once and the total cost of visiting all nodes is minimized. The cost metric can be defined in terms of distance, time, etc. In the scenario of this project, the flight path points can also be regarded as city nodes, and the drones can be used as traveling salesmen to solve the problem. The difference is that the current discussions are mostly based on two-dimensional plane work, and the edges between each node are directed paths, while the characteristic of drone flight is that the distance between each node only depends on the spatial distance, and theoretically, it is possible to directly reach between different nodes. Applying K-clustering to the multi-drone collaborative problem of mTSP, all trajectory points are divided into k regions, and the number of regions k is the number of drones, which successfully weakens an mTSP problem into multiple TSP problems. Next, there are many choices for solving the remaining multiple TSP problems. Just select a more suitable and approximately optimal algorithm through experiments.

[0059] Process of multi-drone trajectory allocation: Combine the oblique photography trajectory and the orthographic trajectory point set into point set S, input the number of drones k, and the calculation process is as follows: ① Convert the longitude and latitude high point set into xyz coordinates; ② Use K-clustering to divide the point set into categories and limit the number of points in each category; ③ Model each category of point set as a TSP problem and solve the shortest path using the improved greedy algorithm; ④ Store the shortest path of each category of point set as an ordered point set, that is, the path assigned to each drone; ⑤ Obtain the multi-drone path and download the executable JSON file, which is the aerial photography route in the precise flight stage of multi-drones.

[0060] S7: Based on the adaptive flight path, multiple drones cooperate to complete the acquisition of oblique photography measurement data in the target area;

[0061] Based on the aerial photography route, multiple drones complete the shooting of their respective responsible aerial photography tasks according to the sub-tasks of aerial photography. The image heading overlap and side overlap need to be set to 80% - 85%, and the image ground resolution is (3 - 5 cm). The data interaction method during the acquisition process refers to step S1.

[0062] S8: Use the oblique photography measurement data collected by multiple drones in cooperation to perform step S3 to complete the construction of a fine three-dimensional model of the target area.

[0063] Use the images taken by multiple drones for aerial photography combined with the image data set quickly collected by a single drone output in S1 to construct a three-dimensional reconstruction project, generate the corresponding three-dimensional modeling task, decompose the modeling task to construct the model, and the modeling steps refer to step S3 to complete the construction of a fine three-dimensional model of the target area.

[0064] A rapid modeling method and system for unfamiliar environments based on multi-UAV collaboration proposed by the present invention uses UAVs equipped with oblique cameras and a hybrid modeling method based on aerial photography real-time images, basic 3D models, and fine 3D models to achieve rapid mapping of unfamiliar areas, improve the rapid 3D modeling ability of unfamiliar and complex environments, and have far-reaching significance for the development of rapid environmental perception ability in future multi-scenario applications.

[0065] It should be noted that the above content only illustrates the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. For those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements all fall within the protection scope of the claims of the present invention.

Claims

1. A rapid modeling method for an unfamiliar environment based on multi-UAV collaboration, characterized in that It includes the following steps: S1: Use a single drone to conduct a scanning shot of the route during the route planning process to collect aerial image data of the target area; S2: During the image collection process, perform Laplacian image fusion based on sift feature point detection and matching to achieve incremental update of the drone aerial images and generate a quasi-real-time two-dimensional image; S3: In the three-dimensional modeling processing module, perform grid calculations based on the main module - worker thread module mode to achieve cluster processing of the modeling tasks; in the main module, manage the three-dimensional modeling task sequence by managing the task sequence. The task sequence is directly managed through the Windows file explorer. The task sequence directory includes the following subdirectories: tasks cancelled by the user, completed tasks, list of all engines connected to the task sequence, tasks failed due to errors at the engine end, tasks waiting to be processed, and tasks currently being processed; In the worker thread module, the processing content includes aerial triangulation calculation or three-dimensional reconstruction process; S4: After the collection is completed, use the image dataset quickly collected by the single drone output in step S1 to construct a modeling task, and quickly generate a basic three-dimensional model based on step S3; S5: In the basic three-dimensional model obtained in step S4, use the selected unfamiliar key target facilities to identify feature points, identify the areas of interest for important ground features, and generate corresponding point, line, and surface vector constraint information; S6: Based on the vector constraint information and aerial photography tasks identified in the basic three-dimensional model obtained in step S5, use the MTSP algorithm to plan the adaptive flight paths of multiple aerial drones; The specific process of planning the adaptive flight path of the MTSP algorithm includes: combining the oblique photography trajectory and the front view trajectory point set into a point set S, inputting the number of drones k, and the calculation process: ① Convert the high point set of longitude and latitude into xyz coordinates; ② Use K-means clustering to divide the point set into categories and limit the number of points in each category; ③ Model each category of point set as a TSP problem and solve the shortest path using an improved greedy algorithm; ④ Store the shortest path of each category of point set as an ordered point set, that is, the path assigned to each drone; ⑤ Obtain the multi-drone path and download the executable JSON file; S7: Based on the adaptive flight path obtained in step S6, multiple drones cooperate to complete the collection of oblique photography measurement data in the target area; S8: Use the oblique photography measurement data collected by multiple drones in cooperation to perform cluster processing of the modeling tasks in step S3 to complete the construction of a fine three-dimensional model of the target area.

2. The rapid modeling method for an unfamiliar environment based on multi-UAV collaboration according to claim 1, wherein: In step S1, when the single drone conducts a scanning shot, the requirement for image overlap is 60% - 65%.

3. A rapid modeling system for an unfamiliar environment based on multi-UAV collaboration using the method according to claim 1, characterized in that: It includes at least a flight path planning module, a position information collection module, an image rapid stitching module, and a three-dimensional modeling processing module, The flight path planning module is used to implement the flight path planning for drone aerial photography and the flight path planning for multiple drones to cooperate in aerial photography; The position information collection module is used to implement the data collection of the real-time shooting position of the drone; The image rapid stitching module is used to implement the real-time stitching of the aerial images transmitted back in real time; The three-dimensional modeling processing module uses oblique photogrammetry data to construct a three-dimensional model; the three-dimensional model includes a basic three-dimensional model and a refined three-dimensional model; The drone, according to the aerial photography route planned by the route planning module, collects data during the aerial photography route of the drone through the position information collection module, splices the aerial photography images through the image rapid splicing module, and then constructs a three-dimensional model through the three-dimensional modeling processing module to achieve rapid modeling of an unfamiliar environment.

Images