Multi-machine collaborative 3D modeling system and method for search and rescue in wild mountain forest environments

Abstract

The present invention discloses a multi-machine collaborative 3D modeling system and method for search and rescue in wild mountain forest environments, wherein a ground station control module is used to communicate with a UAV, send a preliminarily generated rough flight trajectory to the UAV, control the flight of the UAV and receive status information sent back by the UAV; a real-time aerial image stitching module performs real-time stitching based on feature-matched images; a depth estimation module obtains a global depth map of the stitched image based on a deep-learning monocular image depth estimation model; a multi-machine collaborative path planning module obtains a high-altitude area in a global scene as a key area, generates an oblique photography trajectory for the key area, and then uses an improved k-clustering algorithm to divide all trajectory points into categories with balanced task amounts, so that the UAV can pass through all trajectory points of each type of point set, and establishes a 3D reconstruction model of the area to be searched based on the image sent back by the UAV.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent control technology, and in particular relates to a multi-machine collaborative three-dimensional modeling system and method for search and rescue in wild mountain forest environments. Background Art

[0002] Since field search and rescue is a race against time and emphasizes efficiency, and the terrain is changeable and the trees are dense, it is difficult for manual search and rescue to be carried out smoothly and quickly, so field search and rescue is a difficult task. In the prior art, drones are usually used to help implement field search and rescue work. In field search and rescue missions, drones need to quickly model the search and rescue area to find trapped people, but it is difficult to efficiently complete flight missions in complex scenes such as forests by relying solely on manual flight control and manual determination of key areas. Therefore, drones need to have the ability to autonomously determine key areas and model them. In addition, in field search and rescue missions, drones need to autonomously determine the search and rescue targets. When manually operating the flight control, it is difficult to judge the search area through the limited photos taken by the drone's perspective. Therefore, autonomous three-dimensional modeling of drones for complex field environments is a problem with great application value.

[0003] Currently, there are many ways to implement drone control platforms, but most of them lack good support for collaborative tasks. For example, they control multiple drones in a leader-follower mode instead of in a more cooperative way. At the same time, the existing 3D modeling requires manual determination of the key areas to be photographed, which is neither intelligent nor efficient. To achieve rapid field search and rescue by drones, support from drone control platforms and 3D modeling is required.

[0004] In terms of drone control platforms, most of them only support the master-slave mode, and do not support programmed control of collaborative tasks. In addition, each open source control platform such as GCSs only supports a specific set of drones or drone flight controllers. The existing ground control system lacks an energy simulation module, which informs the energy feasibility of the flight before takeoff, thereby avoiding mission interruption and subsequent dangers caused by low batteries. In terms of three-dimensional modeling, most of the systems currently used on the market are orthographic photography or orthographic photography + manually determined key area oblique photography. The orthographic modeling accuracy is not high and cannot meet the needs of field search and rescue. Manually determining the key area oblique photography will reduce the modeling efficiency. Therefore, a better solution is hybrid modeling, that is, combining orthographic photography with oblique photography, in which deep learning is used to autonomously determine the key areas for oblique photography, which can well meet the needs of high efficiency and high precision in field search and rescue. Summary of the invention

[0005] The present invention aims at the problem that 3D modeling in the prior art is difficult to meet the needs of search and rescue tasks in wild mountain forest environments, and provides a multi-machine collaborative 3D modeling system and method for search and rescue in wild mountain forest environments, which at least includes a ground station control module, an aerial image real-time stitching module, a depth estimation module and a multi-machine collaborative path planning module; the ground station control module is used to communicate with the drone, and for the search and rescue area, the initially generated single-machine rough flight trajectory route is sent to the drone, the drone is controlled to fly and the status information sent back by the drone is received; the aerial image real-time stitching module receives the images sent back by the drone in the ground station control module, and performs real-time stitching based on the images with feature matching; the depth estimation module receives the aerial image real-time stitching module, and the multi-machine collaborative path planning module is used to communicate with the drone, and the multi-machine collaborative path planning module is used to communicate with the drone, and the multi-machine collaborative path planning module is used to communicate with the drone, and the multi-machine collaborative path planning module is used to communicate with the ground station control module, and sends the preliminarily generated single-machine rough flight trajectory route to the drone for the search and rescue area, controls the flight of the drone and receives the status information sent back by the drone; the aerial image real-time stitching module receives the images sent back by the drone in the ground station control module, and performs real-time stitching based on the images with feature matching; the depth estimation module receives the aerial image real-time stitching module, and the multi-machine collaborative path planning module is used to communicate with the drone, and sends the preliminarily generated single-machine rough flight trajectory route to the drone, controls the flight of the drone and receives the status information sent back by the drone; the aerial image real-time stitching module receives the aerial image real-time stitching module, and performs real-time stitching based on the images with feature matching; the depth estimation module receives the aerial image real-time stitching module, and the multi-machine collaborative path planning module is used to communicate with the drone, and sends the preliminarily generated single-machine rough flight trajectory route to the drone, controls the flight The global stitching picture output by the real-time image stitching module is obtained, and the global depth map of the stitching image is obtained based on the monocular image depth estimation model of deep learning; the multi-machine collaborative path planning module obtains the area with higher altitude in the global scene as the key area according to the global depth map output by the depth estimation module, generates oblique photography trajectories for the key areas, and then uses the improved k-clustering algorithm to divide all trajectory points into categories with balanced task volume, so that the drone can pass through all trajectory points of each type of point set, and establish a three-dimensional reconstruction model of the area to be searched according to the image sent back by the drone, so as to obtain more detailed landforms in the wild mountain forest environment and formulate detailed search and rescue routes, which is of great significance to the field of intelligent control technology.

[0006] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a multi-machine collaborative 3D modeling system for search and rescue in wild mountain forest environments, comprising at least: a ground station control module, a real-time splicing module for aerial images, a depth estimation module and a multi-machine collaborative path planning module;

[0007] The ground station control module is used to communicate with the drone, send the initially generated single-machine rough flight trajectory route to the drone for the search and rescue area, control the drone flight and receive the status information sent back by the drone;

[0008] The aerial image real-time stitching module receives the images sent back by the drone in the ground station control module, and performs real-time stitching based on the feature matching images;

[0009] The depth estimation module receives the global stitching picture output by the aerial image real-time stitching module, and obtains the global depth map of the stitching image based on the monocular image depth estimation model of deep learning;

[0010] The multi-machine collaborative path planning module obtains the areas with higher altitudes in the global scene as key areas based on the global depth map output by the depth estimation module, generates oblique photography trajectories for the key areas, and then uses the improved k-clustering algorithm to divide all trajectory points into categories with balanced task volumes, so that the drones can pass through all trajectory points of each type of point set, and establish a three-dimensional reconstruction model of the area to be searched based on the images sent back by the drones.

[0011] In order to achieve the above object, the present invention also adopts a technical solution: a multi-machine collaborative 3D modeling method for search and rescue in a wild mountain forest environment, comprising the following steps:

[0012] S1: The ground station control module generates a single-aircraft rough flight trajectory route based on the selected search and rescue area and sends it to the UAV, and controls the UAV to execute according to the rough flight trajectory;

[0013] S2: The UAV executes and obtains images in the rough flight trajectory, and uses a UAV aerial image real-time stitching module based on feature matching to stitch the UAV aerial images received in the ground control station module in real time to obtain a global stitching picture, and the step further includes:

[0014] S21: The ground station control module defines the path points of each segment of the UAV's flight trajectory, obtains the final waypoint point set, and sends the waypoint point set to the UAV in sequence for execution;

[0015] S22: According to the pictures I received in sequence t (t=1,2,...,n), get the GPS latitude and longitude information of the image from the EXIF ​​header t (t=1,2,...,n), for any image I received at time t t , calculate the distance dis between it and the previously received image based on the latitude and longitude information i (i=1,2,...,t-1), and find the three closest images I j ,I k ,I l ;

[0016] S23: Calculation I t ,I j ,I k ,I l SIFT feature points;

[0017] S24: For reference image I t and observation map I j ,I k ,I l Perform key point matching, wherein the matching method is as follows: when the feature point of a pair of matching images is greater than a set threshold, an edge is established between the two images, and this process is repeated, and then the maximum connected component algorithm is used to solve the maximum connected component of the entire matching graph to obtain the best matching solution;

[0018] S25: Using the RANSAC algorithm to remove the mismatched points in step S24;

[0019] S26: By optimizing the objective function, we can reduce the cumulative error that may occur in the splicing of multiple images over a large area. Assume that a total of N groups of SIFT feature matches can be found in M ​​UAV remote sensing images. Then the transformation matrix of the i-th image for the selected reference image is H i , its specific form is as follows:

[0020]

[0021] After writing it in vector form, the corresponding column vector is X i ,definition The objective function E(X) that needs to be optimized is:

[0022]

[0023] Among them, e i =X m p i,m -X n p i,n ,r i =X ref p i,ref -p i,ref ,p i,m is the coordinate of the feature point in image m in the i-th matching group, and ref is the number of the reference image;

[0024] S27: After the optimization process in S26, the global optimal transformation matrix H of each image can be obtained i , according to the transformation matrix H of each image i , map all images to the final fusion image imaging space;

[0025] S28: Record each image I acquired in real time during the rough flight stage i The longitude and latitude information of (x,y)(x∈[0,m),y∈[0,n)) is GPS i =(long i ,lat i ), because the camera of the drone is pointing vertically downward during the rough flight phase, it can be assumed that in the original image I i middle, The latitude and longitude coordinates of the GPS i =(long i ,lat i ), using the optimized transformation matrix H i You can get it in the splicing diagram The latitude and longitude coordinates of the GPS i =(long i ,lat i ), for a series of pixel coordinate positions and longitude and latitude coordinate values ​​(xi ,y i ,GPS i ) Use the least squares method to fit the plane to obtain the longitude plane equation long(x,y) and the latitude plane equation lat(x,y);

[0026] S3: A monocular image depth estimation model based on deep learning, taking the global stitched image obtained in step S2 as input, to obtain a depth map of the stitched image; the depth estimation model is: a supervised depth estimation model with an Encoder-Decoder structure;

[0027] S4: According to the depth map obtained in step S3, the area with higher altitude in the global scene is obtained as the key area, and the oblique photography trajectory is generated for the key area. Then, the improved k-clustering algorithm is used to divide all the trajectory points into categories with balanced task volume, so that the drone can pass through all the trajectory points of each type of point set, and a three-dimensional reconstruction model of the area to be searched is established based on the image sent back by the drone.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] (1) In the design of the ground station system, most of the existing technologies only support the control of a single drone, and need to be targeted at a specific model of drone. In the search and rescue scenario, high efficiency is required, and a single machine cannot meet the demand. In addition, the search and rescue team often updates the model of a drone due to a failure, resulting in inconsistencies in the models of all drones. The present invention shields this hardware difference by installing a Raspberry Pi 3 motherboard on each drone and running the UAV Agent program on it. Its main function is to parse the control commands sent by the ground station control system into an API received by a specific drone flight control. The drone's status information is parsed into a format that the ground station can receive and transmitted back to the ground station via a character stream using Wi-Fi or OLSR connection.

[0030] (2) In terms of multi-machine collaboration, most existing technologies require the use of a leader-follower mode to control multiple drones. This mode requires maintaining a certain formation and selecting a leader drone during the multi-machine collaboration process, which limits the ability of multiple drones to perform large-scale collaborative modeling. The present invention uses centralized control through the "Sky Painter" ground station system to monitor and control the status of multiple drones at the same time, achieving more flexible multi-machine collaboration and greatly improving modeling efficiency.

[0031] (3) In modeling route planning, existing technologies mostly generate a "fence-like" reciprocating trajectory with a fixed altitude. The present invention obtains the terrain information of the area to be modeled through deep learning technology, automatically calibrates key areas according to the terrain characteristics, generates oblique photography trajectories for areas with undulating terrain and generates orthographic trajectories for areas with flat terrain, thus achieving a trade-off between flight time and modeling accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a system architecture diagram of the "Sky Painter" platform used by the ground station control module in Example 1 of the present invention;

[0033] Figure 2 This is the logical architecture diagram of the flight monitor;

[0034] Figure 3 This is a desktop GUI interface diagram of the "Air Painter" system used by the ground station control module in Example 1 of the present invention;

[0035] Figure 4 It is a mosaic result and a projection result diagram of the shooting point after the image is fused by Laplace in Example 2 of the present invention;

[0036] Figure 5 is the longitude and latitude plane equation obtained by least squares fitting in Example 2 of the present invention;

[0037] Figure 6 This is a schematic diagram of multi-machine collaborative hybrid modeling in Example 2 of the present invention;

[0038] Figure 7 is a demonstration diagram of generating a trajectory according to a selected area in Embodiment 2 of the present invention;

[0039] Figure 8 It is a diagram showing the kmeasn clustering of trajectory points and the TSP problem solving result of the clustering results in Example 2 of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0041] Example 1

[0042] A multi-machine collaborative 3D modeling system for search and rescue in wild mountain forest environments, including at least: a ground station control module, a real-time aerial image stitching module, a depth estimation module, and a multi-machine collaborative path planning module;

[0043] The ground station control module is used to communicate with the drone, send the initially generated single-machine rough flight trajectory route to the drone for the search and rescue area, control the drone flight and receive the status information sent back by the drone; the ground station control module used in this embodiment is the "Sky Painter" system, and its system architecture is as follows Figure 1 As shown, it mainly includes two modules: UAV agent and flight monitor:

[0044] The UAV agent is responsible for handling three important types of information, including status, device control, and exceptions. UAV APIs are provided for communication between the flight controller and the monitor. Before transmission, the UAV agent periodically parses the flight status (from the flight controller) into the format required by the exception monitor and the message sender, which further converts it into a character stream for transmission. The SDK between the flight controller and the UAV agent program is used to access and update the flight status information, which will be output to the flight monitor on the ground.

[0045] The control messages from the GCS flow to the UAV agent's message listener in the format of a character stream. The UAV agent's command processor converts each command into a format executable by the flight control module.

[0046] The flight status information is periodically sent to the abnormal monitoring module of the UAV agent for inspection. If any emergency occurs, the abnormal monitor either sends a high priority command to the flight controller or reports to the flight monitor through the information transmitter.

[0047] The main task of the flight monitor is to communicate with each drone and further provide a series of necessary services for their cooperation. Its architecture is as follows Figure 2 In addition, the flight monitor also provides interfaces for interacting with upper-layer applications and end users through API and GUI respectively.

[0048] Based on the information received from each drone, the service controller in the flight monitor can generate control messages to implement cooperative services for connection maintenance, path planning, synchronization, deviation avoidance, and collision avoidance. By configuring network services, the "Sky Painter" system is able to connect all drones and flight monitors, and all connected drones are managed by the agent manager to achieve connection maintenance; set the Boolean type synchronization value to indicate the status of each drone to achieve synchronization; by comparing the real-time trajectory with the planned results, deviations can be captured and further compensated to achieve deviation avoidance; by checking the velocity vector of any two aircraft and calculating the distance between them, collisions can be avoided.

[0049] Upper-level user applications can choose to use swarm APIs to implement API interactions. Swarm APIs are mainly implemented by the message center, which is responsible for bidirectional processing and transmission of information and messages. Among them, the status API that provides drone status information is integrated into the information parser, and the control API that provides control functions is integrated into the message distributor. In order to meet the specific needs of user interaction with different tasks, GUI APIs are also provided to help developers quickly create visual interfaces. The GUI interface is as follows: Figure 3 shown.

[0050] Upper-layer user applications can choose to interact with the flight monitor via mission files in CoUAS, which are ordered lists of actions;

[0051] The Sky Painter system enables UAVs and flight monitors to communicate with each other through high-performance Wi-Fi connections to the data plane and control plane. A TCP connection is established for data and control transmission over the Wi-Fi network. Each data packet consists of a header and a body. A field is specified in the data packet header to distinguish different types of data packets in the Sky Painter system. The data packet body contains important information, such as actions in subtasks, the center location and radius of geofences, etc. In order to support more functions in the future, new types of data packets can be defined and added;

[0052] To support cross-UAV collaboration and smooth mission execution, different UAVs periodically synchronize with each other to cope with asynchronous situations (e.g., position deviations, low connection quality, or different transmission delays). To this end, the large task sent to each UAV must be divided into a series of small subtasks, called steps. Each step contains at most one action with synchronization set to true (synchronization required), so that the UAVs will proceed to the next step in synchronization at the end of each previous step. After its step is completed, the UAVagent sends a synchronization message to the flight monitor and waits until it receives a confirmation message. After collecting all the synchronization messages, the flight monitor sends a confirmation to all UAVs to proceed to the next step. By carefully dividing the tasks, the "Sky Painter" system minimizes waiting time.

[0053] The aerial image real-time stitching module receives the images sent back by the drone in the ground station control module, and performs real-time stitching based on the images with feature matching; in the aerial image real-time stitching module, Laplace fusion is used to establish the Laplace pyramid LP(x, y, n) and Gaussian pyramid G(x, y, n+1) of the image, and the Gaussian pyramid G of the mask image is calculated. mask(x, y, n+1); stitch the Laplacian pyramid of the image, stitch the part of the image that passes through the mask on each layer to get the Laplacian pyramid of the result, calculate the top layer of the image pyramid of the result, stitch the top layer of the Gaussian pyramid of each image through the top layer of the corresponding mask pyramid, start from the top layer of the image pyramid of the result, add each layer of the Laplacian pyramid of the result and then upsample to get the fusion result.

[0054] The depth estimation module receives the global stitching picture output by the real-time stitching module of aerial images, and obtains the global depth map of the stitching image based on the deep learning monocular image depth estimation model; the multi-machine collaborative path planning module obtains the area with higher altitude in the global scene as the key area according to the global depth map output by the depth estimation module, generates oblique photography trajectories for the key areas, and then uses the improved k-clustering algorithm to divide all trajectory points into categories with balanced task volume, so that the drone can pass through all trajectory points of each type of point set, and establish a three-dimensional reconstruction model of the area to be searched according to the image sent back by the drone.

[0055] Embodiment 2:

[0056] The multi-machine collaborative 3D modeling method for outdoor forest environment search and rescue, the ground station control module in this embodiment also adopts the "Sky Painter" system, including the following steps:

[0057] S1: The ground station control module generates a single-aircraft rough flight trajectory route based on the selected search and rescue area and sends it to the UAV, and controls the UAV to execute according to the rough flight trajectory;

[0058] S2: The UAV executes and obtains images in the rough flight trajectory, and uses the UAV aerial image real-time stitching module based on feature matching to stitch the UAV aerial images received in the ground control station module in real time to obtain a global stitching picture.

[0059] According to the longitude and latitude coordinates of the vertices of the polygonal area to be searched and rescued given in the "Sky Painter" system, a rough flight route for a single aircraft is initially generated, and the drone image real-time stitching module based on feature matching is used to stitch the drone aerial images received by the "Sky Painter" system in real time. The specific implementation steps are as follows:

[0060] First, find the minimum and maximum longitude and latitude of the longitude and latitude coordinates of the vertices of the polygonal area, thereby determining the minimum circumscribed regular rectangle of the entire area to be modeled, which is used as the area for the first rough flight.

[0061] According to the rough flight route height h, flight speed v, route overlap rate α, lateral overlap rate β determined in the "Sky Painter" system, and the internal parameters of the drone's onboard camera - focal length f and sensor long frame length l and short frame length w, the time interval △t for the drone to take photos and the route spacing △s of the fence route are determined as follows:

[0062]

[0063]

[0064] Based on the fence-like trajectory generated by the above calculation, the path point interval △τ is used to define the path points of each segmented trajectory to obtain the final waypoint point set. The "Air Painter" system sends this waypoint point set to the drone for execution in sequence.

[0065] According to the pictures I received in order t (t=1,2,...,n), get the latitude and longitude information GPS of the image from the EXIF ​​header t (t=1,2,...,n), for any image I received at time t t , calculate the distance dis between it and the previously received image based on the latitude and longitude information i (i=1,2,...,t-1), and find the three closest images I j ,I k ,I l .

[0066] Computation I t ,I j ,I k ,I l SIFT feature points of each image are first obtained, and the grayscale image I(x, y) is generated according to the Gaussian function G(x, y, σ) Where σ is the fuzzy coefficient of the Gaussian function. Based on the Gaussian pyramid G(x, y, σ), the differential pyramid D(x, y, σ) is constructed. The initial extreme value detection is performed in the differential pyramid, and the Taylor expansion is fitted near the detected extreme value points to locate the sub-pixel. From the spatial position of the feature point (x, y, σ) back to the pyramid, a small area is sampled near the feature point to determine the main direction θ of the feature point and generate a descriptor by gradient histogram statistics, and the descriptor is normalized to form the final description vector.

[0067] For reference figure I t and observation map I j ,I k ,I l Perform key point matching and define the key point descriptor R in the template image t =(rt1 ,r t2 ,...,r t128 ) and the key point descriptor S in the observation graph μ =(s μ1 ,s μ2 ,...,s μ128 ), μ∈{j,k,l}, calculate the similarity measure between any two descriptors μ∈{j,k,l}, only d(R i ,S μ ) can only satisfy the match when the distance between the closest point and the second closest point is less than a certain threshold. In order to further reduce the mismatch, a matching graph is constructed based on the image feature matching information. The specific method is to establish an edge between the two images when the feature points of a pair of matching images are greater than the set threshold, and then use the maximum connected component algorithm to solve the maximum connected component of the entire matching graph to obtain the best matching solution.

[0068] The RANSAC algorithm is used to eliminate the mismatched points in the above steps. First, four non-collinear sample data are randomly selected, and the transformation matrix H is calculated based on these four matching points. The projection error of all data in the data set under the current transformation matrix H is calculated. If the error is less than the set threshold, the consensus set that satisfies the current transformation matrix can be added, and the number of elements in the consensus set is returned. According to the number of elements in the current consensus set, it is determined whether it is the optimal consensus set. If so, the optimal consensus set is updated, and the number of iterations k is updated at the same time. If the number of iterations is greater than k, exit, otherwise continue to iterate.

[0069] Assume that a total of N sets of SIFT feature matches can be found in M ​​UAV remote sensing images. The transformation matrix of the i-th image for the selected reference image is H i , the corresponding column vector is X i ,definition Then by optimizing the objective function:

[0070]

[0071] To reduce the cumulative error that may occur when stitching multiple images over a large area. i =X m p i,m -X n p i,n , r i =X ref p i,ref -p i,ref , p i,m are the coordinates of the feature points in the i-th group of matching image m, and ref is the number of the randomly determined reference image.

[0072] According to the transformation matrix H of each picture i After that, all images can be mapped to the final fusion image imaging space. In order to reduce the blur phenomenon at the splicing point, the present invention adopts Laplace fusion. Since Laplace fusion requires multi-scale fusion and consumes a lot of memory, in order to solve this problem, all images are pre-processed before image fusion. i If the overlap ratio of the mapped image with other mapped images exceeds the specified threshold thresh, the image is removed. For the overlapping parts of overlapping images with an overlap ratio less than thresh, only the pixel values ​​of the images corresponding to the maximum value from the minimum value of the image boundary are retained.

[0073] After preprocessing, the Laplacian pyramid LP(x,y,n) and Gaussian pyramid G(x,y,n+1) of the image are established, and the Gaussian pyramid G of the mask image is calculated. mask (x, y, n+1); stitch the Laplacian pyramid of the image: stitch the part of the image that passes through the mask on each layer to get the Laplacian pyramid of the result; calculate the top layer of the image pyramid of the result: stitch the top layer of the Gaussian pyramid of each image through the top layer of the corresponding mask pyramid, starting from the top layer of the image pyramid of the result, add each layer of the Laplacian pyramid of the result and then upsample, and finally get the fusion result, such as Figure 4 shown.

[0074] Remember each picture I i The longitude and latitude information of (x,y)(x∈[0,m),y∈[0,n)) is GPS i =(long i ,lat i ), in the rough flight stage, the camera value of the drone is downward, so it can be assumed that in the original image I i middle, The latitude and longitude coordinates of the GPS i =(long i ,lat i ), using the optimized transformation matrix H i In the fusion diagram, we can get The latitude and longitude coordinates of the GPS i =(long i ,lat i ), for a series of pixel coordinate positions and longitude and latitude coordinate values ​​(x i ,y i ,GPS i ) Use the least squares method to fit the plane to obtain the longitude plane equation long(x,y) and the latitude plane equation lat(x,y), as shown Figure 5 shown.

[0075] S3: A monocular image depth estimation model based on deep learning takes the global stitched image obtained in step S2 as input to obtain a depth map of the stitched image. The specific implementation steps are as follows:

[0076] Step S31: Take the spliced ​​image obtained in step S2 as input, perform a common 7*7 convolution operation, adjust the number of image channels to 64 channels, and perform batch normalization and relu activation;

[0077] Step S32: perform the maximum pooling operation on the obtained feature map featuremap1 and pass it into the residual block to obtain a feature map featuremap2 with a channel number of 256; pass the obtained feature map featuremap2 into the residual block to obtain a feature map featuremap3 with a channel number of 512; pass the obtained feature map featuremap3 into the residual block to obtain a feature map featuremap4 with a channel number of 1024; pass the obtained feature map featuremap4 into the residual block to obtain a feature map featuremap5 with a channel number of 2048; input featuremap5, perform upsampling to restore it to the size of featuremap4, called featuremap5', then perform feature fusion of featuremap4 and featuremap5' to generate a feature map called featuremap45, and then perform convolution operation to output the estimated disparity map disparity1, and then use featuremap45 as input, and then repeat the above operation to generate depth maps of different scales disparity2, disparity3 and disparity4 as outputs of the depth estimation network;

[0078] Step S33: construct the encoding and decoding structure of the pose estimation network; input two pictures, perform a normal 7*7 convolution operation, and adjust the number of picture channels to 64 channels, perform batch normalization and relu activation; perform the maximum pooling operation on the obtained feature map featuremap1 and pass it into the residual block to obtain a feature map featuremap2 with 64 channels; pass the obtained feature map featuremap2 into the residual block to obtain a feature map featuremap3 with 128 channels; pass the obtained feature map featuremap3 into the residual block to obtain a feature map featuremap4 with 256 channels; pass the obtained feature map featuremap4 into the residual block to obtain a feature map featuremap with 512 channels eaturemap5; input featuremap5, use 1*1 convolution to change the number of channels to 256, and then use relu function to activate to get feature map featuremap6; use 3*3*256 convolution operation to extract features from featuremap6 and output featuremap7; use 3*3*256 convolution operation to extract features from featuremap7 and output featuremap8; use 1*1 convolution to change the number of feature map channels to 6 for featuremap8 and output featuremap9; average the second and third dimensions of featuremap9, then transform the dimension to generate a vector with a shape of [4,6], which is the relative camera pose change between adjacent frames;

[0079] Step S34: train and tune the neural network model established in step S31 and step S32; shuffle the data set to generate training samples and test samples; obtain pre-training weights; use the adam optimizer, and the learning rate automatically decreases during the training process; calculate the training loss and validation loss after each epoch; compare the validation loss of each epoch, and save the model parameters with the smallest validation loss;

[0080] Step S35: Input the test image into the network to obtain the depth estimation result, calculate the various losses and accuracy in the depth estimation evaluation index, and evaluate the network performance.

[0081] S4: According to the depth map obtained in step S3, the area with higher altitude in the global scene is obtained as the key area, and the oblique photography trajectory is generated for the key area. The global orthographic trajectory generated in step 2 is combined to realize the multi-machine trajectory allocation for the second round of precision flight. The schematic diagram is shown as follows: Figure 6 As shown, the steps specifically include:

[0082] Step S41: According to the depth map obtained in step S3, the minimum bounding rectangle of the key area is obtained, and the pixel coordinates of its four vertices on the stitched image are (x1, y1), (x2, y2), (x3, y3), and (x4, y4), respectively. The longitude and latitude plane equations long(x, y) and lat(x, y) obtained in step S2 are used to obtain the longitude and latitude of the four vertices of the key area. Multiple groups of expansion rectangles of the key area are obtained through the flight interval flyspace, and the coverage trajectory (directly above the area) for each edge of the modeling area rectangle is generated, such as Figure 7 As shown. Apply the path point interval actspace to define the path points of each segmented trajectory to obtain the final point set;

[0083] Step S42: The oblique photography trajectory and the frontal view trajectory point set are combined into a point set S. The number of drones is k. The point set is classified using the k-clustering algorithm. The spatial distance between the trajectory points is used as the similarity measurement index. The clustering goal is to make the cluster sum of squares As small as possible, where K is the total number of categories, N k is the number of track points of each type, u k is the center point of each category. In order to ensure that the tasks of each drone are as balanced as possible, the clustering target is updated to The clustering process is as follows Figure 8 As shown;

[0084] Step S43: Model each type of point set as a TSP problem, so that each drone can pass through all trajectory points of a certain type of point set to ensure that the photos taken can cover the entire scene, and use the improved greedy algorithm to solve the shortest path of each type, store it as a sequential point set, and save it as a taskfile that can be read by CoUAS in step 1;

[0085] Step S44: The UAV flies according to the assigned route and obtains image information, and establishes a three-dimensional model of the area to be searched and rescued based on the aerial images taken by the UAV.

[0086] After planning the multi-machine coordinated flight path in step S4, the drones transmit images back to the "Sky Painter" system platform in real time. The "Sky Painter" system platform connects with the 3D reconstruction software through an interface and starts real-time modeling. After the drone mission is completed, a 3D model of the entire area can be obtained. The 3D model can provide a more detailed picture of the terrain and topography of the entire area, and the rescue team can formulate a detailed search and rescue route based on this.

[0087] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A multi-machine collaborative 3D modeling method for search and rescue in wild mountain forest environments, characterized in that: The steps include: S1: The ground station control module generates a single-aircraft rough flight trajectory route based on the selected search and rescue area and sends it to the UAV, and controls the UAV to execute according to the rough flight trajectory; S2: The UAV executes and obtains images in the rough flight trajectory, and uses a UAV aerial image real-time stitching module based on feature matching to stitch the UAV aerial images received in the ground station control module in real time to obtain a global stitching picture, and the step further includes: S21: The ground station control module defines the path points of each segment of the UAV's flight trajectory, obtains the final waypoint point set, and sends the waypoint point set to the UAV in sequence for execution; S22: According to the pictures I received in sequence t ,t=1,2,...n, get the GPS latitude and longitude information of the image from the EXIF ​​header t , t=1,2,...n, for any image I received at time t t , calculate the distance dis between it and the previously received image based on the latitude and longitude information i , i = 1, 2, ... t-1, and find the three closest pictures I j ,I k ,I l ; S23: Calculation I t ,I j ,I k ,I l SIFT feature points; S24: For reference image I t and observation map I j ,I k ,I l Perform key point matching, wherein the matching method is as follows: when the feature point of a pair of matching images is greater than a set threshold, an edge is established between the two images, and this process is repeated, and then the maximum connected component algorithm is used to solve the maximum connected component of the entire matching graph to obtain the best matching solution; S25: Using the RANSAC algorithm to remove the mismatched points in step S24; S26: By optimizing the objective function, we can reduce the cumulative error caused by stitching multiple images over a large area. Suppose that N groups of SIFT feature matches are found in M ​​UAV remote sensing images. Then the transformation matrix of the i-th image for the selected reference image is H i , its specific form is as follows: After writing it in vector form, the corresponding column vector is X i ,definition The objective function E(X) that needs to be optimized is: Among them, e i =X m p i,m -X n p i,n ,r i =X ref p i,ref -p i,ref ,p i,m is the coordinate of the feature point in image m in the i-th matching group, and ref is the number of the reference image; S27: Obtain the global optimal transformation matrix H of each image after the optimization process in S26 i , according to the transformation matrix H of each image i , map all images to the final fusion image imaging space; S28: Record each image I acquired in real time during the rough flight stage i The longitude and latitude information of (x,y),x∈[0,m),y∈[0,n) is GPS i =(long i ,lat i ), because the camera of the drone is pointing vertically downward during the rough flight stage, so it is assumed that in the original image I i middle, The latitude and longitude coordinates of the GPS i =(long i ,lat i ), using the optimized transformation matrix H i In the mosaic diagram The latitude and longitude coordinates of the GPS i =(long i ,lat i ), for a series of pixel coordinate positions and longitude and latitude coordinate values ​​(x i ,y i ,GPS i ) Use the least squares method to fit the plane to obtain the longitude plane equation long(x,y) and the latitude plane equation lat(x,y); S3: A monocular image depth estimation model based on deep learning, taking the global stitched image obtained in step S2 as input, to obtain a depth map of the stitched image; the depth estimation model is: a supervised depth estimation model with an Encoder-Decoder structure; S4: According to the depth map obtained in step S3, the area with large height value in the global scene is obtained as the key area, and the oblique photography trajectory is generated for the key area. Then, the improved k-clustering algorithm is used to divide all the trajectory points into categories with balanced task volume, so that the drone can pass through all the trajectory points of each type of point set, and a three-dimensional reconstruction model of the area to be searched is established based on the image sent back by the drone.

2. The multi-machine collaborative 3D modeling method for outdoor forest environment search and rescue as claimed in claim 1, characterized in that: In step S1, the minimum circumscribed regular rectangle of the entire search and rescue area is determined according to the minimum and maximum longitude and latitude of the vertices of the search and rescue area, that is, the single-aircraft rough flight trajectory route area is preliminarily generated.

3. The multi-machine collaborative 3D modeling method for outdoor forest environment search and rescue as claimed in claim 2, characterized in that: The step S4 further comprises: S41: According to the depth map obtained in step S3, the minimum circumscribed rectangle of the key area is obtained, and the pixel coordinates of its four vertices on the stitched image are (x1, y1), (x2, y2), (x3, y3), and (x4, y4), respectively; according to the longitude plane equation long(x, y) and the latitude plane equation lat(x, y) obtained in step S28, the longitude and latitude of the four vertices of the key area are obtained, and multiple groups of outer expansion rectangles of the key area are obtained through the route interval, and the coverage trajectory for each side of the modeling area rectangle is generated, and the path point interval is used to delineate the path points of each segmented trajectory to obtain the final point set; S42: The oblique photography trajectory and the frontal view trajectory point set are combined into a point set S. The number of drones is k. The point set is divided into categories using the k-clustering algorithm. The spatial distance between the trajectory points is used as the similarity measurement indicator. The clustering goal is to make the cluster sum of squares minimum, where K is the total number of categories, N k is the number of trajectory points in each category, u k is the center point of each class. In order to ensure the balance of tasks for each drone, the clustering target is updated to Minimum; S43: Model each type of point set as a TSP problem, so that each drone can pass through all the trajectory points of a certain type of point set, use the improved greedy algorithm to solve the shortest path of each type, store it as a sequential point set, and assign the flight trajectory of the drone; S44: The UAV flies according to the assigned route and obtains image information, and establishes a three-dimensional model of the area to be searched and rescued based on the aerial images taken by the UAV.

4. A multi-machine collaborative 3D modeling system for search and rescue in wild mountain forest environments, the system is used to implement the method according to claim 1, characterized in that: At least include: ground station control module, aerial image real-time stitching module, depth estimation module and multi-machine collaborative path planning module; The ground station control module is used to communicate with the drone, send the initially generated single-machine rough flight trajectory route to the drone for the search and rescue area, control the drone flight and receive the status information sent back by the drone; The aerial image real-time stitching module receives the images sent back by the drone in the ground station control module, and performs real-time stitching based on the feature matching images; The depth estimation module receives the global stitching picture output by the aerial image real-time stitching module, and obtains the global depth map of the stitching image based on the monocular image depth estimation model of deep learning; The multi-machine collaborative path planning module obtains the area with large height value in the global scene as the key area based on the global depth map output by the depth estimation module, generates oblique photography trajectory for the key area, and then uses the improved k-clustering algorithm to divide all trajectory points into categories with balanced task volume, so that the drone can pass through all trajectory points of each type of point set, and establish a three-dimensional reconstruction model of the area to be searched based on the image sent back by the drone.

5. The multi-machine collaborative 3D modeling system for outdoor forest environment search and rescue as claimed in claim 4, characterized in that: The ground station control module at least includes a flight monitor, a flight control board and a UAV agent, and the UAV agent is responsible for processing three types of status information: drone status, equipment control and abnormality, and provides UAV APIs for communication between the flight control board and the monitor; The flight monitor is used to communicate with each drone and interact with upper-layer applications and end users through an API and a GUI, respectively.

6. The multi-machine collaborative 3D modeling system for outdoor forest environment search and rescue as claimed in claim 4, characterized in that: The large task sent to each UAV in the ground station control module must be divided into a series of small subtasks, called steps. Each step contains at most one action with synchronization set to true. The UAV will proceed to the next step synchronously at the end of each previous step; after its step is completed, the UAV agent sends a synchronization message to the flight monitor and waits until it receives a confirmation message. After collecting all the synchronization messages, the flight monitor sends confirmation to all UAVs to proceed to the next step.

7. The multi-machine collaborative 3D modeling system for outdoor forest environment search and rescue as claimed in claim 5 or 6, characterized in that: In the aerial image real-time stitching module, Laplace fusion is used to establish the Laplace pyramid LP(x, y, n) and Gaussian pyramid G(x, y, n+1) of the image, and the Gaussian pyramid G of the mask image is obtained. mask (x, y, n+1); stitch the Laplacian pyramids of the images, stitch the parts of the images that pass through the mask on each layer of the Laplacian pyramid to obtain a mixed Laplacian pyramid, calculate the top layer of the mixed Laplacian pyramid, stitch the parts of the top layer of the Gaussian pyramid of each image that pass through the top layer mask to obtain a fusion starting image; starting from the fusion starting image, upsample it and add it to the top layer of the mixed Laplacian pyramid, continue to upsample it and add it to the corresponding layer of the Laplacian pyramid, and finally obtain the fusion result.

Images