Real-time three-dimensional reconstruction method and device based on flight of unmanned aerial vehicle

By combining the circular flight strategy of UAVs with the OpenSFM/OpenMVS framework, 3D reconstruction results are generated in real time, solving the problem of insufficient real-time performance of 3D reconstruction in existing technologies and achieving fast and accurate 3D reconstruction.

CN118982632APending Publication Date: 2024-11-19AEROSPACE INFORMATION RES INST CAS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411010713.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Existing technologies for 3D reconstruction have poor real-time performance, failing to achieve fast and real-time 3D reconstruction while ensuring accuracy, especially in large-scale scene reconstruction tasks where timeliness is insufficient.

Method used

A real-time 3D reconstruction method based on UAV flight is adopted. The UAV path is planned by a circular flight strategy, and image processing is performed using the OpenSFM and OpenMVS frameworks to obtain dense point clouds. The 3D reconstruction results are generated in real time through Poisson reconstruction and texture mapping.

Benefits of technology

It enables real-time generation of 3D reconstruction results during UAV flight, improving the real-time performance and efficiency of 3D reconstruction, avoiding model overlap, and enhancing the speed and accuracy of image processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982632A_ABST
    Figure CN118982632A_ABST
Patent Text Reader

Abstract

The invention provides a real-time three-dimensional reconstruction method and device based on unmanned aerial vehicle flight, and relates to the technical field of three-dimensional reconstruction, and the method comprises the steps: carrying out the unmanned aerial vehicle flight path planning of a to-be-reconstructed region based on a circular flight strategy; based on a group of return images corresponding to each circular flight path, obtaining dense point clouds corresponding to each group of return images; cutting the dense point cloud corresponding to each group of returned images based on the effective area, and performing Poisson reconstruction and texture mapping to obtain a three-dimensional reconstruction result corresponding to each group of returned images; and splicing the three-dimensional reconstruction results corresponding to all groups of return images to obtain a three-dimensional reconstruction result corresponding to the to-be-reconstructed region. According to the invention, the reconstruction of the flash is realized, and the real-time performance of the three-dimensional reconstruction is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to a real-time three-dimensional reconstruction method and device based on unmanned aerial vehicle flight. Background Art

[0002] In recent years, with the development of drones, large-scale 3D reconstruction using drones as a collection method has become one of the trends in the surveying and mapping field. Real-time 3D reconstruction focuses more on the timeliness of the reconstructed scene to achieve fast 3D scene modeling, while also ensuring the accuracy of the modeling. Faced with the task of reconstructing large-scale scenes, how to achieve fast and real-time 3D reconstruction while ensuring accuracy is very challenging.

[0003] Most 3D reconstruction strategies are to transfer all images obtained in a collection task to the local computer and obtain the 3D model of the area to be measured through offline reconstruction. However, offline reconstruction not only requires a large number of cluster devices, but also consumes a huge amount of modeling time, and the timeliness of modeling is poor. When facing emergency scenarios, it is impossible to provide fast 3D scene information.

[0004] Therefore, a real-time 3D reconstruction strategy is urgently needed to achieve the goal of “reconstruction while flying”. Summary of the invention

[0005] The present invention provides a real-time 3D reconstruction method and device based on UAV flight, which are used to solve the defect of poor real-time performance of 3D reconstruction in the prior art, realize "reconstruction while flying", and improve the real-time performance of 3D reconstruction.

[0006] The present invention provides a real-time three-dimensional reconstruction method based on unmanned aerial vehicle flight, comprising: The UAV flight path is planned for the reconstruction area based on a circular flight strategy; the circular flight strategy is that each flight path of the UAV is a circular flight path of equal size, the centers of all circular flight paths are on a straight line, and the subsequent circular flight path passes through the center of the previous circular flight path; Based on a set of returned images corresponding to each circular flight path, a dense point cloud corresponding to each set of returned images is obtained; The dense point cloud corresponding to each set of returned images is cropped based on the effective area, and Poisson reconstruction and texture mapping are performed to obtain the three-dimensional reconstruction result corresponding to each set of returned images; the effective area is a rectangular area with the center of the circular flight path as the center of mass and the radius of the circular flight path as the side length; The three-dimensional reconstruction results corresponding to all groups of returned images are spliced ​​together to obtain the three-dimensional reconstruction result corresponding to the area to be reconstructed.

[0007] In some embodiments, the step of obtaining a dense point cloud corresponding to each set of returned images based on a set of returned images corresponding to each circular flight path includes: Based on the OpenSFM framework, the camera pose of each returned image in each group of returned images is estimated to obtain the camera pose information corresponding to each returned image in each group of returned images; Based on the OpenMVS framework and the camera posture information corresponding to each set of returned images, dense point cloud reconstruction is performed to obtain a dense point cloud corresponding to each set of returned images.

[0008] In some embodiments, the camera pose estimation is performed on each returned image in each group of returned images based on the OpenSFM framework to obtain the camera pose information corresponding to each returned image in each group of returned images, including: For each set of returned images, SIFT feature extraction algorithm is used to extract feature points from each returned image; Match the feature points extracted from different returned images to obtain feature point pairs; Perform camera pose estimation based on the feature point pairs to obtain initial camera pose information corresponding to each returned image; Taking minimizing the reprojection error of the three-dimensional point cloud as the optimization goal, the bundle adjustment algorithm is used to optimize the initial camera posture information corresponding to each returned image to obtain the camera posture information corresponding to each returned image.

[0009] In some embodiments, the dense point cloud reconstruction based on the OpenMVS framework and the camera posture information corresponding to each returned image in each set of returned images to obtain a dense point cloud corresponding to each set of returned images includes: Determine the potential neighboring frames corresponding to each returned image; Determine the depth map corresponding to each returned image based on the camera pose information corresponding to each returned image and the potential neighboring frames corresponding to each returned image; Optimize the depth map corresponding to each returned image; Based on the confidence level, the optimized depth map corresponding to each returned image is fused to obtain a fused depth map; The fused depth map is converted into a three-dimensional point cloud and fused to obtain a dense point cloud corresponding to each set of returned images.

[0010] In some embodiments, determining the potential neighboring frames corresponding to each returned image includes: For each returned image, the following steps are performed: Taking the current returned image as a reference frame, and determining a candidate neighboring frame according to the current returned image; For each of the candidate neighborhood frames, an evaluation score of each candidate neighborhood frame is determined according to a minimum coverage area of ​​a common viewpoint in the reference frame and the candidate neighborhood frame, a weight score of an angle between the common viewpoint and the candidate neighborhood frame, and a weight score of a resolution similarity between the reference frame and the candidate neighborhood frame; the common viewpoint is a feature point that belongs to both a feature point set of the reference frame and a feature point set of the candidate neighborhood frame; Based on the evaluation score of each candidate neighborhood frame, a potential neighborhood frame corresponding to the current returned image is determined.

[0011] In some embodiments, determining a depth map corresponding to each returned image based on the camera posture information corresponding to each returned image and the potential neighboring frames corresponding to each returned image includes: For each returned image, the following steps are performed: The current returned image and a corresponding potential neighboring frame are regarded as an image pair; Determine the induced unitarity corresponding to each image pair based on the camera pose information corresponding to the current returned image and the camera pose information corresponding to the potential neighboring frame in each image pair; For each pixel in the current returned image, the following steps are performed: According to the current pixel, a window including the current pixel is set; Determine, according to the induced unitary property corresponding to each image pair, a corresponding pixel point in a potential neighborhood frame for each pixel point within the window under each image pair; Determine a normalized cross-correlation coefficient corresponding to each pixel point in the window under each image pair according to the induced unitary property corresponding to each image pair and the corresponding pixel point in the potential neighborhood frame of each pixel point in the window under each image pair; Determining a maximum normalized cross-correlation coefficient according to the normalized cross-correlation coefficient corresponding to each pixel point in the window under each image pair; Based on the maximum normalized cross-correlation coefficient, a depth value of the current pixel is determined.

[0012] In some embodiments, the optimizing the depth map corresponding to each returned image includes: For each pixel in each returned image, the following steps are performed: According to the depth value of the current pixel and the camera posture information corresponding to the current returned image, the current pixel is back-projected into the three-dimensional space to obtain the three-dimensional space point corresponding to the current pixel; Projecting the three-dimensional space point to each potential neighboring frame corresponding to the current returned image, and determining a depth value corresponding to the three-dimensional space point in each potential neighboring frame; Determine the number of target potential neighborhood frames corresponding to the current pixel point according to a depth difference between a depth value corresponding to the three-dimensional space point in the potential neighborhood frame and a depth value of a camera corresponding to the three-dimensional space point relative to the potential neighborhood frame; The current pixel is optimized according to the number of target potential neighborhood frames corresponding to the current pixel.

[0013] The present invention also provides a real-time three-dimensional reconstruction device based on unmanned aerial vehicle flight, comprising: A path planning module, used for planning the flight path of the UAV for the reconstruction area based on a circular flight strategy; the circular flight strategy is that each flight path of the UAV is a circular flight path of equal size, the centers of all circular flight paths are on a straight line, and the subsequent circular flight path passes through the center of the previous circular flight path; A processing module, configured to obtain a dense point cloud corresponding to each set of returned images based on a set of returned images corresponding to each circular flight path; A cropping module is used to crop the dense point cloud corresponding to each set of returned images based on the effective area, and perform Poisson reconstruction and texture mapping to obtain the three-dimensional reconstruction result corresponding to each set of returned images; the effective area is a rectangular area with the center of the circular flight path as the center of mass and the radius of the circular flight path as the side length; The stitching module is used to stitch the three-dimensional reconstruction results corresponding to all groups of returned images to obtain the three-dimensional reconstruction result corresponding to the area to be reconstructed.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a real-time three-dimensional reconstruction method based on drone flight as described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the real-time three-dimensional reconstruction method based on drone flight as described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned real-time three-dimensional reconstruction methods based on drone flight.

[0017] The real-time 3D reconstruction method and device based on UAV flight provided by the present invention plans the UAV flight path into multiple circular flight paths through a circular flight strategy, processes a group of returned images corresponding to a circular flight path each time, and obtains a dense point cloud corresponding to a group of returned images, avoids processing all returned images at one time, and improves image processing efficiency, crops the dense point cloud corresponding to each group of returned images based on the effective area, and performs Poisson reconstruction and texture mapping to obtain the 3D reconstruction results corresponding to each group of returned images, avoiding model overlap, and finally splices the 3D reconstruction results corresponding to all groups of returned images to obtain the 3D reconstruction results corresponding to the area to be reconstructed, thereby realizing "reconstruction while flying" and improving the real-time performance of 3D reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 This is one of the flow charts of the real-time three-dimensional reconstruction method based on UAV flight provided by the present invention; Figure 2 It is a schematic diagram of the planning of the flight path of the UAV provided by the present invention; Figure 3 It is a schematic diagram of the effective area provided by the present invention; Figure 4 This is the second flow chart of the real-time 3D reconstruction method based on UAV flight provided by the present invention; Figure 5 It is a structural schematic diagram of a real-time three-dimensional reconstruction device based on unmanned aerial vehicle flight provided by the present invention.

[0020] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Figure 1This is one of the flow charts of the real-time 3D reconstruction method based on UAV flight provided by the present invention, such as Figure 1 As shown, the present invention provides a real-time 3D reconstruction method based on UAV flight, comprising the following steps: Step 110, planning the UAV flight path for the area to be reconstructed based on the circular flight strategy; the circular flight strategy is that each flight path of the UAV is a circular flight path of equal size, the centers of all circular flight paths are on a straight line, and the subsequent circular flight path passes through the center of the previous circular flight path.

[0023] Specifically, current real-time reconstruction methods for three-dimensional scenes are mostly based on the Simultaneous Localization and Mapping (SLAM) framework, but the SLAM framework has the problem of being unable to close the loop from beginning to end.

[0024] In order to avoid the problem of being unable to close the loop at the beginning and end, a circular flight strategy is used to plan the UAV flight path in the reconstruction area, so that the image data obtained by the UAV can be closed in data collection.

[0025] The circular flight strategy is that each flight path of the drone is a circular flight path of equal size, the centers of all circular flight paths are on a straight line, and the next circular flight path passes through the center of the previous circular flight path. That is, if the coordinates of the center of the previous circular flight path are (x, y, z), then the center of the next circular flight path is (x+r, y, z), where r is the radius of the circular flight path.

[0026] The radius of the circular flight path and the number of the circular flight paths are determined according to the size of the area to be reconstructed. Figure 2 is a schematic diagram of the planning of the UAV flight path provided by the present invention, such as Figure 2 As shown, there are five circular flight paths of equal size, the centers of the five circular flight paths are in a straight line, and the latter circular flight path passes through the center of the former circular flight path.

[0027] Taking the first two circular flight paths as an example, the specific flight process can be divided into two cases. When the take-off point is the intersection of the two circular flight paths, the intersection is used as the take-off point. After flying the entire circle according to the first circular flight path, the drone returns to the intersection and performs the flight mission directly at the intersection according to the second circular flight path. When the take-off point is not the intersection of the two circular flight paths, the take-off point of the first circular flight path is point ①, and the starting point on the second circular flight path is point ②. After flying the entire circle according to the first circular flight path, the drone returns to point ①. Then the drone flies directly from point ① to point ②, and then performs the flight mission according to the second circular flight path.

[0028] Step 120 , based on a set of returned images corresponding to each circular flight path, obtain a dense point cloud corresponding to each set of returned images.

[0029] Specifically, each flight of the drone is guaranteed to fly at least one entire circular flight path. During the flight, the shooting direction of the drone is always the center of the circular flight path. On each circular flight path, a fixed shooting time interval is set to obtain a set of return images corresponding to each circular flight path. Each set of return images is processed to obtain a dense point cloud corresponding to each set of return images.

[0030] Step 130, based on the effective area, the dense point cloud corresponding to each group of returned images is cropped, and Poisson reconstruction and texture mapping are performed to obtain the three-dimensional reconstruction results corresponding to each group of returned images; the effective area is a rectangular area with the center of the circular flight path as the center of mass and the radius of the circular flight path as the side length.

[0031] Specifically, since there will be overlaps between different groups of returned images, the dense point clouds corresponding to different groups of returned images will also overlap, resulting in overlaps between the 3D reconstruction results corresponding to different groups of returned images, and the final 3D reconstruction results will be confusing.

[0032] In order to solve the overlapping problem, the central rectangular area of ​​the circular flight path is taken as the effective area. Figure 3 It is a schematic diagram of the effective area provided by the present invention, such as Figure 3 As shown, the effective area is a rectangular area with the center of the circular flight path as the center of mass and the radius of the circular flight path as the side length. The acquired dense point cloud is cropped based on the effective area to ensure that the dense point clouds corresponding to different groups of returned images can be seamlessly connected without overlapping.

[0033] After cropping the dense point cloud corresponding to each set of returned images, Poisson reconstruction and texture mapping are performed on the cropped dense point cloud to obtain the three-dimensional reconstruction results corresponding to each set of returned images.

[0034] Step 140 , stitching the 3D reconstruction results corresponding to all groups of returned images to obtain a 3D reconstruction result corresponding to the area to be reconstructed.

[0035] Specifically, after obtaining the three-dimensional reconstruction results corresponding to all groups of returned images, the three-dimensional reconstruction results corresponding to all groups of returned images are spliced ​​to obtain the three-dimensional reconstruction results corresponding to the area to be reconstructed.

[0036] The real-time 3D reconstruction method based on UAV flight provided by the present invention plans the UAV flight path into multiple circular flight paths through a circular flight strategy, processes a group of returned images corresponding to a circular flight path each time, and obtains a dense point cloud corresponding to a group of returned images, avoids processing all returned images at one time, and improves image processing efficiency, crops the dense point cloud corresponding to each group of returned images based on the effective area, and performs Poisson reconstruction and texture mapping to obtain the 3D reconstruction result corresponding to each group of returned images, avoiding model overlap, and finally splices the 3D reconstruction results corresponding to all groups of returned images to obtain the 3D reconstruction result corresponding to the area to be reconstructed, thereby realizing "reconstruction while flying" and improving the real-time performance of 3D reconstruction.

[0037] In some embodiments, Figure 4 This is the second flow chart of the real-time 3D reconstruction method based on UAV flight provided by the present invention. Figure 4 As shown, based on a set of returned images corresponding to each circular flight path, a dense point cloud corresponding to each set of returned images is obtained, including: Step 410 , based on the OpenSFM framework, the camera pose of each returned image in each group of returned images is estimated to obtain the camera pose information corresponding to each returned image in each group of returned images.

[0038] Specifically, the OpenSFM framework is an open source software framework that can reconstruct three-dimensional structures from a series of two-dimensional images based on Structure from Motion (SfM).

[0039] Based on the OpenSFM framework, the camera pose is estimated for each return image in each group of return images to obtain the camera pose information corresponding to each return image in each group of return images. Based on the camera pose information corresponding to each return image in each group of return images, the sparse point cloud corresponding to each group of return images is obtained.

[0040] Step 420 , based on the OpenMVS framework and the camera posture information corresponding to each of the returned images in each set of returned images, dense point cloud reconstruction is performed to obtain a dense point cloud corresponding to each set of returned images.

[0041] Specifically, the OpenMVS framework is a mature open source framework for 3D reconstruction, based on Multi-View Stereo (MVS), with comprehensive reconstruction effects and performance.

[0042] Based on the OpenMVS framework, dense point cloud reconstruction is performed on the basis of the camera posture information corresponding to each set of returned images, aiming to reconstruct high-quality dense 3D point clouds from a set of images with overlapping views, thereby obtaining a dense point cloud corresponding to each set of returned images.

[0043] In some embodiments, the camera pose estimation is performed on each returned image in each group of returned images based on the OpenSFM framework to obtain the camera pose information corresponding to each returned image in each group of returned images, including: For each set of returned images, SIFT feature extraction algorithm is used to extract feature points from each returned image; Match the feature points extracted from different returned images to obtain feature point pairs; The camera pose is estimated based on the feature point pairs to obtain the initial camera pose information corresponding to each returned image; Taking minimizing the reprojection error of the three-dimensional point cloud as the optimization goal, the bundle adjustment algorithm is used to optimize the initial camera posture information corresponding to each returned image to obtain the camera posture information corresponding to each returned image.

[0044] Specifically, a Scale-Invariant Feature Transform (SIFT) feature extraction algorithm is used to extract feature points of each returned image in each group of returned images, so as to identify unique feature points in each returned image and obtain feature points of each returned image.

[0045] The feature points extracted from different returned images are matched to obtain feature point pairs to identify the same scene features observed between different returned images. The matching process can be: for each feature point , extract the feature descriptor of each feature point , use the distance metric (such as Euclidean distance) between feature descriptors to find the best matching feature point pairs.

[0046] The expression of the feature descriptor is as follows:

[0047] in, Representing feature points The feature descriptor of represents the Gaussian blur function, where Indicates the scale, and the Gaussian blur function is used to Smooth the image to improve the scale invariance of feature points. Representing feature points The pixel value at .

[0048] After obtaining the feature point pairs, each feature point pair contains two feature points under different views. The camera pose information (i.e., camera position and orientation) corresponding to each returned image is estimated by solving the multi-view geometry problem. This can be achieved by solving the PnP (Perspective-n-Point) problem. Its goal is to find the optimal camera pose information (rotation matrix R and translation vector t) corresponding to each returned image so that the projection error from the three-dimensional point to the two-dimensional image point is minimized.

[0049] After obtaining the camera posture information corresponding to each returned image, the feature points of each returned image can be remapped into three-dimensional space according to the camera posture to obtain the initial three-dimensional point cloud corresponding to each returned image.

[0050] The relationship between the two-dimensional projection point and the three-dimensional space point is as follows:

[0051] in, Represents the two-dimensional projection point corresponding to the three-dimensional space point, represents a point in three-dimensional space, Represents the camera's internal parameter matrix, including focal length and principal point coordinates, represents the rotation matrix of the camera, Represents the camera translation vector.

[0052] With the optimization goal of minimizing the reprojection error of the 3D point cloud, the bundle adjustment algorithm is used to optimize the initial 3D point cloud corresponding to each return image and the initial camera posture information corresponding to each return image, and the sparse point cloud corresponding to each return image and the camera posture information corresponding to each return image are obtained. The sparse point cloud corresponding to each return image in each group of return images is merged to obtain the sparse point cloud corresponding to each group of return images.

[0053] The expression of the reprojection error of the 3D point cloud is as follows:

[0054] in, Indicates The parameters of the camera, Indicates The coordinates of the three-dimensional points, It is The camera observed The image coordinates of the three-dimensional points, and It is the image coordinates calculated based on the camera parameters and the 3D point coordinates.

[0055] In some embodiments, dense point cloud reconstruction is performed based on the OpenMVS framework and the camera pose information corresponding to each of the returned images in each set of returned images to obtain a dense point cloud corresponding to each set of returned images, including: Determine the potential neighboring frames corresponding to each returned image; Determine the depth map corresponding to each returned image based on the camera pose information corresponding to each returned image and the potential neighboring frames corresponding to each returned image; Optimize the depth map corresponding to each returned image; Based on the confidence level, the optimized depth map is fused to obtain a fused depth map; The fused depth map is converted into a three-dimensional point cloud and fused to obtain a dense point cloud corresponding to each set of returned images.

[0056] In some embodiments, determining a potential neighboring frame corresponding to each returned image includes: For each returned image, the following steps are performed: The current returned image is used as a reference frame, and candidate neighboring frames are determined based on the current returned image; For each candidate neighborhood frame, the evaluation score of each candidate neighborhood frame is determined according to the minimum coverage area of ​​the common viewpoint in the reference frame and the candidate neighborhood frame, the weight score of the angle between the common viewpoint and the reference frame and the candidate neighborhood frame, and the weight score of the resolution similarity between the reference frame and the candidate neighborhood frame; the common viewpoint is a feature point that belongs to the feature point set of the reference frame and the feature point set of the candidate neighborhood frame; Based on the evaluation score of each candidate neighborhood frame, the potential neighborhood frame corresponding to the current returned image is determined.

[0057] Specifically, the current returned image is used as the reference frame M to determine the candidate neighborhood frame V according to factors such as the scene content and scale of the current returned image.

[0058] The feature points of the feature point set of the reference frame M and the feature point set of the candidate neighborhood frame V are taken as the common viewpoint f. For each candidate neighborhood frame, the evaluation score of each candidate neighborhood frame is calculated based on the minimum coverage area of ​​the common viewpoint f in the reference frame M and the candidate neighborhood frame V, the weight score of the angle between the common viewpoint f in the reference frame M and the candidate neighborhood frame V, and the weight score of the resolution similarity between the reference frame M and the candidate neighborhood frame V.

[0059] The expression of the evaluation score is as follows:

[0060] in, represents the minimum coverage area of ​​the common viewpoint f in the reference frame M and the candidate neighborhood frame V, represents the feature point set of the candidate neighborhood frame V, represents the feature point set of the reference frame M, represents the weight score of the angle between the common viewpoint f in the reference frame M and the candidate neighborhood frame V, represents all candidate neighborhood frames, The weight score representing the resolution similarity between the reference frame M and the candidate neighborhood frame V.

[0061] To encourage a good disparity range within a neighborhood, the weight function is defined as The product of all view pairs in , the weight function The expression is as follows:

[0062] in, represents the weight score of the angle between the common viewpoint f in the reference frame M and the candidate neighborhood frame V, Represents the weight score of the angle between two frames, and represents different candidate neighborhood frames, represents the angle between the common viewpoint f in the reference frame R and the candidate neighborhood frame V, Indicates the maximum angle threshold set.

[0063] In order to estimate the 3D sampling rate of the candidate neighborhood frame V near the common viewpoint f, a sphere centered at the common viewpoint f and projected in the candidate neighborhood frame V with a diameter equal to the pixel spacing in the candidate neighborhood frame V is calculated. , is calculated in a similar way for the reference frame M , then confirm and Ratio . ratio The expression is as follows:

[0064] in, represents the ratio, represents the volume of a sphere centered at the common viewpoint f, whose projection diameter in the candidate neighborhood frame V is equal to the pixel spacing in the reference frame M, It represents the volume of a sphere centered at the common viewpoint f, whose projection diameter in the candidate neighborhood frame V is equal to the pixel spacing in the candidate neighborhood frame V.

[0065] Define a weight function based on the ratio r , which is used to measure the similarity in resolution between the reference frame M and the candidate neighborhood frame V at the common viewpoint f. Weight function The expression is as follows:

[0066] The evaluation scores of all candidate neighborhood frames are sorted from large to small. However, directly selecting the candidate neighborhood frame with the largest evaluation score will fall into a local optimal solution and may not cover the entire scene. To avoid this, a Markov random field is used to select the best potential neighborhood frame for each returned image.

[0067] In some embodiments, based on the camera posture information corresponding to each returned image and the potential neighboring frames corresponding to each returned image, determining the depth map corresponding to each returned image includes: For each returned image, the following steps are performed: The current returned image and a corresponding potential neighboring frame are regarded as an image pair; Determine the induced unitarity corresponding to each image pair based on the camera pose information corresponding to the current returned image and the camera pose information corresponding to the potential neighboring frame in each image pair; For each pixel in the current returned image, the following steps are performed: According to the current pixel, a window including the current pixel is set; Determine the corresponding pixel point in the potential neighborhood frame for each pixel point in the window under each image pair according to the induced unitary property corresponding to each image pair; Determine a normalized cross-correlation coefficient corresponding to each pixel in the window under each image pair according to the induced unitary property corresponding to each image pair and the corresponding pixel in the potential neighborhood frame of each pixel in the window under each image pair; Determining a maximum normalized cross-correlation coefficient according to the normalized cross-correlation coefficient corresponding to each pixel point in the window under each image pair; Based on the maximum normalized cross-correlation coefficient, the depth value of the current pixel is determined.

[0068] Specifically, for each returned image, the following steps are performed: A returned image corresponds to multiple potential neighboring frames. The current returned image and any corresponding potential neighboring frame are taken as an image pair. The current returned image corresponding to the camera And the current returned image Corresponding potential neighbor frames Take an image pair as an example.

[0069] First, return the current image Each pixel point p in is assigned to a random 3D plane. Assume that the coordinates of pixel point p are:

[0070] in, Indicates the horizontal coordinate value. Indicates the vertical coordinate value.

[0071] 3D Point (Sparse point cloud) must be located in the line of sight of pixel p, within the depth range Choose a random depth within , then Calculate in spherical coordinate system for:

[0072] in, represents a sparse point cloud, Indicates the depth value, Indicates the current returned image The corresponding camera internal parameter matrix, Representing images The pixels in .

[0073] On Camera The normal vectors of randomly assigned planes in the spherical coordinate system of :

[0074] in, is a random angle in the range [0˚,360˚], is a random angle in the range [0˚,60˚]. These range settings are based on a simple assumption: when a 3D point The normal of the neighborhood area in the corresponding image and When the angle between the z-axis of the coordinate system is lower than a certain threshold, the image block is in the current returned image. is visible.

[0075] Current returned image Corresponding camera parameters , potential neighbor frame Corresponding camera parameters , K is the camera intrinsic parameter, R is the rotation matrix, and Q is the camera center.

[0076] Given the projection matrices of two cameras ( is the 3×3 identity matrix, is a unit vector, indicating that the world coordinate system is consistent with the camera P) and , and by Defined plane, where Representative Points The normal line of is the normal vector, then the homography induced by the plane is:

[0077] The camera parameters of the image pair are and ,flat On Camera Therefore, the projection matrix and plane parameters can be converted to standard form (place the world origin at ):

[0078] After applying the above transformation, the obtained camera and , and the induced homography is:

[0079] According to the pixel point p, set a window (such as a square window) B containing the pixel point p. For each pixel point q in the window B, use homography mapping Find its potential neighboring frames Then, through the pixel q and The normalized cross-correlation (NCC) is used to calculate the aggregate matching cost m(p) of pixel p.

[0080] The expressions of the normalized cross-correlation coefficient NCC and the aggregate matching cost m(p) are as follows:

[0081] in, represents the normalized cross-correlation coefficient, represents the average pixel value in area B, Represents the window set around pixel p, represents any pixel in window B, Represents pixel The corresponding homography, m(p) represents the aggregate matching cost of pixel p.

[0082] The disparity can be obtained based on the camera parameters and the identity matrix. By selecting the disparity corresponding to the maximum NCC value (that is, the minimum matching cost), a disparity value can be estimated for each pixel and converted into the corresponding depth value:

[0083] in, represents the focal length of the camera, represents the distance of the baseline between the two cameras, represents parallax, Indicates the depth value.

[0084] In some embodiments, optimizing the depth map corresponding to each returned image includes: For each pixel in each returned image, the following steps are performed: According to the depth value of the current pixel and the camera posture information corresponding to the current returned image, the current pixel is back-projected into the three-dimensional space to obtain the three-dimensional space point corresponding to the current pixel; Project the three-dimensional space point to each potential neighboring frame corresponding to the current returned image, and determine the depth value corresponding to the three-dimensional space point in each potential neighboring frame; Determine the number of target potential neighborhood frames corresponding to the current pixel point according to a depth difference between a depth value corresponding to the three-dimensional space point in the potential neighborhood frame and a depth value of a camera corresponding to the three-dimensional space point relative to the potential neighborhood frame; The current pixel is optimized according to the number of target potential neighborhood frames corresponding to the current pixel.

[0085] Specifically, since the original depth maps may not be completely consistent in common areas due to depth errors, refinement is required to ensure consistency between adjacent views.

[0086] For the current returned image The current pixel point p in the image is back-projected into the three-dimensional space according to the depth value of the current pixel point and the camera posture information corresponding to the current returned image, and the three-dimensional space point corresponding to the current pixel point is obtained. The back-projection expression is as follows:

[0087] In the formula, represents a point in three-dimensional space, Indicates the depth value, Indicates the current returned image The corresponding camera rotation matrix, represents the camera's translation vector, Indicates the current returned image The corresponding camera parameters are: Indicates the current pixel.

[0088] The three-dimensional space point The three-dimensional space points are projected into each potential neighboring frame corresponding to the current returned image to determine the depth value corresponding to each potential neighboring frame.

[0089] Determine the depth difference between the depth value corresponding to the 3D space point in the potential neighboring frame and the depth value of the camera corresponding to the 3D space point relative to the potential neighboring frame. Compare the depth difference with the threshold. If the depth difference is less than the threshold, it indicates that the 3D space point X is in the current returned image. and potential neighbor frames If they are consistent, the potential neighboring frames As the target potential neighborhood frame. If the number of target potential neighborhood frames is at least L (for example, L is 2), the current pixel point p is considered to be a reliable scene point and its pixel in the depth map is retained, otherwise its pixel in the depth map is deleted.

[0090]

[0091] in, Represents a point in three-dimensional space Relative to potential neighbor frames The corresponding camera depth, Indicates the potential neighbor frame The potential neighbor frame is used at the projection of the 3D space point X The depth value calculated from the depth map, represents the mth potential neighboring frame corresponding to the current returned image, Indicates the threshold value.

[0092] After the above refinement process, most errors can be removed, so that a relatively clean depth map can be obtained for each returned image, that is, an optimized depth map corresponding to each returned image is obtained.

[0093] In some embodiments, the optimized depth map corresponding to each returned image is fused based on the confidence level to obtain the fused depth map in the following specific process: After optimization, all depth maps can be fused to represent the scene. However, directly fused depth maps may contain a lot of redundancy because different depth maps may have common coverage areas of the scene, especially for neighboring images. To remove these redundancies, the depth maps are further reduced by confidence-based depth map fusion.

[0094] Determine a reference view and render all depth maps to the reference view. Select the depth estimate with the highest confidence for each pixel as the initial estimate. If the depth estimate produced by another depth map is The pixel in the current depth map The initial depth estimate The difference is at the threshold , which is a strong indication that both viewpoints have correctly reconstructed the same surface. These close observations are averaged into a single depth estimate, with each observation weighted by its confidence value according to the following formula:

[0095] in, represents the confidence of the reference view, Indicates the pixel point in the current depth map after the kth iteration The corresponding confidence level, Indicates the pixel point in the current depth map after the k+1th iteration The corresponding confidence level, Indicates that another depth map is at pixel point The depth estimate produced is Indicates the pixel point in the current depth map after the kth iteration The corresponding depth estimate, Indicates the pixel point in the current depth map after the k+1th iteration The corresponding depth estimate.

[0096] The total confidence of each depth estimate must be above a threshold , otherwise it will be considered an outlier and will not be processed further. The remaining pixels are checked with the current pixel through the visibility constraint. If the combined depth estimate of a point is outside the confidence region for the 3D point If occlusion occurs, the confidence level will be reduced accordingly:

[0097] in, Indicates the pixel point in the current depth map after the k+1th iteration The corresponding confidence level, Indicates the pixel point in the current depth map after the kth iteration The corresponding confidence level, Indicates that another depth map is at pixel point The confidence generated.

[0098]

[0099] in, Indicates the pixel point in the current depth map after the k+1th iteration The corresponding confidence level, Indicates the pixel point in the current depth map after the kth iteration The corresponding confidence level, Represents pixel The corresponding 3D point, express In View I y The image coordinates in Indicate point The corresponding confidence level.

[0100] After the visibility constraint, if the confidence is positive, it will be kept. If the confidence is negative, the depth estimate will be discarded as an outlier.

[0101] After discarding outliers, there will be many holes in the fused depth map, assuming that any small missing parts of the depth map are most likely close to their neighbors. To fill these gaps, we use a Find all inliers within the window. If there are enough inliers to make a good estimate, the median of the inliers is assigned as the depth of the pixel. If there are only a few neighboring inliers, the depth map is left blank. Finally, use a smaller window A median filter is used to smooth out the inliers.

[0102] In some embodiments, the specific process of converting the fused depth map into a three-dimensional point cloud and fusing it to obtain a dense point cloud corresponding to each set of returned images is as follows: The fused depth maps are converted into 3D point clouds and fused into a global dense point cloud. Consecutive fused depth maps partially overlap each other, and each fused depth map is compared with the previous fused depth map during the generation process. If a new depth estimate violates the free space of the previous fused depth map, the new depth estimate will be rejected. If the difference between a new depth estimate and the previous fused depth map is within a threshold Within , the two estimates are merged into a single vertex, which is generated only once in the output.

[0103] For each pixel in the image, its coordinates in three-dimensional space are calculated using the following formula based on its depth value and the internal and external parameters of the camera:

[0104] Among them, X represents a point in three-dimensional space, K represents the camera intrinsic parameter matrix, R represents the rotation matrix of the camera, t represents the translation vector of the camera, p is the coordinate of the pixel point in the image, and d is the depth value of the pixel point p.

[0105] The real-time three-dimensional reconstruction device based on drone flight provided by the present invention is described below. The real-time three-dimensional reconstruction device based on drone flight described below and the real-time three-dimensional reconstruction method based on drone flight described above can be referenced to each other.

[0106] Figure 5is a schematic diagram of the structure of a real-time 3D reconstruction device based on UAV flight provided by the present invention, such as Figure 5 As shown, the present invention provides a real-time three-dimensional reconstruction device based on unmanned aerial vehicle flight, comprising: A path planning module 510 is used to plan the flight path of the UAV for the reconstruction area based on a circular flight strategy; the circular flight strategy is that each flight path of the UAV is a circular flight path of equal size, the centers of all circular flight paths are on a straight line, and the subsequent circular flight path passes through the center of the previous circular flight path; A processing module 520, configured to obtain a dense point cloud corresponding to each set of returned images based on a set of returned images corresponding to each circular flight path; The cropping module 530 is used to crop the dense point cloud corresponding to each set of returned images based on the effective area, and perform Poisson reconstruction and texture mapping to obtain the three-dimensional reconstruction result corresponding to each set of returned images; the effective area is a rectangular area with the center of the circular flight path as the center of mass and the radius of the circular flight path as the side length; The stitching module 540 is used to stitch the 3D reconstruction results corresponding to all groups of returned images to obtain the 3D reconstruction result corresponding to the area to be reconstructed.

[0107] In some embodiments, the processing module 520 includes: The first processing submodule is used to estimate the camera posture of each returned image in each group of returned images based on the OpenSFM framework to obtain the camera posture information corresponding to each returned image in each group of returned images; The second processing submodule is used to perform dense point cloud reconstruction based on the OpenMVS framework and the camera posture information corresponding to each returned image in each group of returned images, so as to obtain a dense point cloud corresponding to each group of returned images.

[0108] In some embodiments, the first processing submodule is specifically used to: For each set of returned images, SIFT feature extraction algorithm is used to extract feature points from each returned image; Match the feature points extracted from different returned images to obtain feature point pairs; Perform camera pose estimation based on the feature point pairs to obtain initial camera pose information corresponding to each returned image; Taking minimizing the reprojection error of the three-dimensional point cloud as the optimization goal, the bundle adjustment algorithm is used to optimize the initial camera posture information corresponding to each returned image to obtain the camera posture information corresponding to each returned image.

[0109] In some embodiments, the second processing submodule includes: A first determining unit, used to determine a potential neighboring frame corresponding to each returned image; A second determination unit is used to determine a depth map corresponding to each returned image based on camera posture information corresponding to each returned image and a potential neighboring frame corresponding to each returned image; An optimization unit, used to optimize the depth map corresponding to each returned image; A fusion unit, used to fuse the optimized depth map corresponding to each returned image based on the confidence level to obtain a fused depth map; The conversion unit is used to convert the fused depth map into a three-dimensional point cloud and fuse it to obtain a dense point cloud corresponding to each group of returned images.

[0110] In some embodiments, the first determining unit is specifically configured to: For each returned image, the following steps are performed: Taking the current returned image as a reference frame, and determining a candidate neighboring frame according to the current returned image; For each of the candidate neighborhood frames, an evaluation score of each candidate neighborhood frame is determined according to a minimum coverage area of ​​a common viewpoint in the reference frame and the candidate neighborhood frame, a weight score of an angle between the common viewpoint and the candidate neighborhood frame, and a weight score of a resolution similarity between the reference frame and the candidate neighborhood frame; the common viewpoint is a feature point that belongs to both a feature point set of the reference frame and a feature point set of the candidate neighborhood frame; Based on the evaluation score of each candidate neighborhood frame, a potential neighborhood frame corresponding to the current returned image is determined.

[0111] In some embodiments, the second determining unit is specifically configured to: For each returned image, the following steps are performed: The current returned image and a corresponding potential neighboring frame are regarded as an image pair; Determine the induced unitarity corresponding to each image pair based on the camera pose information corresponding to the current returned image and the camera pose information corresponding to the potential neighboring frame in each image pair; For each pixel in the current returned image, the following steps are performed: According to the current pixel, a window including the current pixel is set; Determine, according to the induced unitary property corresponding to each image pair, a corresponding pixel point in a potential neighborhood frame for each pixel point within the window under each image pair; Determine a normalized cross-correlation coefficient corresponding to each pixel point in the window under each image pair according to the induced unitary property corresponding to each image pair and the corresponding pixel point in the potential neighborhood frame of each pixel point in the window under each image pair; Determining a maximum normalized cross-correlation coefficient according to the normalized cross-correlation coefficient corresponding to each pixel point in the window under each image pair; Based on the maximum normalized cross-correlation coefficient, a depth value of the current pixel is determined.

[0112] In some embodiments, the optimization unit is specifically configured to: For each pixel in each returned image, the following steps are performed: According to the depth value of the current pixel and the camera posture information corresponding to the current returned image, the current pixel is back-projected into the three-dimensional space to obtain the three-dimensional space point corresponding to the current pixel; Projecting the three-dimensional space point to each potential neighboring frame corresponding to the current returned image, and determining a depth value corresponding to the three-dimensional space point in each potential neighboring frame; Determine the number of target potential neighborhood frames corresponding to the current pixel point according to a depth difference between a depth value corresponding to the three-dimensional space point in the potential neighborhood frame and a depth value of a camera corresponding to the three-dimensional space point relative to the potential neighborhood frame; The current pixel is optimized according to the number of target potential neighborhood frames corresponding to the current pixel.

[0113] Figure 6 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communications interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute a real-time three-dimensional reconstruction method based on the flight of a drone, and the method includes: The UAV flight path is planned for the area to be reconstructed based on the circular flight strategy; the circular flight strategy is that each flight path of the UAV is a circular flight path of equal size, the centers of all circular flight paths are on a straight line, and the latter circular flight path passes through the center of the former circular flight path; based on a group of returned images corresponding to each circular flight path, a dense point cloud corresponding to each group of returned images is obtained; based on the effective area, the dense point cloud corresponding to each group of returned images is cropped, and Poisson reconstruction and texture mapping are performed to obtain the three-dimensional reconstruction result corresponding to each group of returned images; the effective area is a rectangular area with the center of the circular flight path as the center of mass and the radius of the circular flight path as the side length; the three-dimensional reconstruction results corresponding to all groups of returned images are spliced ​​to obtain the three-dimensional reconstruction result corresponding to the area to be reconstructed.

[0114] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0115] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the real-time three-dimensional reconstruction method based on drone flight provided by the above methods, the method comprising: The UAV flight path is planned for the area to be reconstructed based on the circular flight strategy; the circular flight strategy is that each flight path of the UAV is a circular flight path of equal size, the centers of all circular flight paths are on a straight line, and the latter circular flight path passes through the center of the former circular flight path; based on a group of returned images corresponding to each circular flight path, a dense point cloud corresponding to each group of returned images is obtained; based on the effective area, the dense point cloud corresponding to each group of returned images is cropped, and Poisson reconstruction and texture mapping are performed to obtain the three-dimensional reconstruction result corresponding to each group of returned images; the effective area is a rectangular area with the center of the circular flight path as the center of mass and the radius of the circular flight path as the side length; the three-dimensional reconstruction results corresponding to all groups of returned images are spliced ​​to obtain the three-dimensional reconstruction result corresponding to the area to be reconstructed.

[0116] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the real-time three-dimensional reconstruction method based on drone flight provided by the above methods, the method comprising: The UAV flight path is planned for the area to be reconstructed based on the circular flight strategy; the circular flight strategy is that each flight path of the UAV is a circular flight path of equal size, the centers of all circular flight paths are on a straight line, and the latter circular flight path passes through the center of the former circular flight path; based on a group of returned images corresponding to each circular flight path, a dense point cloud corresponding to each group of returned images is obtained; based on the effective area, the dense point cloud corresponding to each group of returned images is cropped, and Poisson reconstruction and texture mapping are performed to obtain the three-dimensional reconstruction result corresponding to each group of returned images; the effective area is a rectangular area with the center of the circular flight path as the center of mass and the radius of the circular flight path as the side length; the three-dimensional reconstruction results corresponding to all groups of returned images are spliced ​​to obtain the three-dimensional reconstruction result corresponding to the area to be reconstructed.

[0117] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0118] Through the description of the above implementation methods, technicians in this field can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some of the technical features therein with equivalents. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time 3D reconstruction method based on UAV flight, It is characterized in that include: The UAV flight path is planned for the reconstruction area based on a circular flight strategy; the circular flight strategy is that each flight path of the UAV is a circular flight path of equal size, the centers of all circular flight paths are on a straight line, and the subsequent circular flight path passes through the center of the previous circular flight path; Based on a set of returned images corresponding to each circular flight path, a dense point cloud corresponding to each set of returned images is obtained; The dense point cloud corresponding to each set of returned images is cropped based on the effective area, and Poisson reconstruction and texture mapping are performed to obtain the three-dimensional reconstruction result corresponding to each set of returned images; the effective area is a rectangular area with the center of the circular flight path as the center of mass and the radius of the circular flight path as the side length; The three-dimensional reconstruction results corresponding to all groups of returned images are spliced ​​together to obtain the three-dimensional reconstruction result corresponding to the area to be reconstructed.

2. The real-time 3D reconstruction method based on UAV flight according to claim 1, It is characterized in that The method of obtaining a dense point cloud corresponding to each set of returned images based on a set of returned images corresponding to each circular flight path includes: Based on the OpenSFM framework, the camera pose of each returned image in each group of returned images is estimated to obtain the camera pose information corresponding to each returned image in each group of returned images; Based on the OpenMVS framework and the camera posture information corresponding to each set of returned images, dense point cloud reconstruction is performed to obtain a dense point cloud corresponding to each set of returned images.

3. The real-time 3D reconstruction method based on UAV flight according to claim 2, It is characterized in that The camera posture estimation is performed on each returned image in each group of returned images based on the OpenSFM framework to obtain the camera posture information corresponding to each returned image in each group of returned images, including: For each set of returned images, SIFT feature extraction algorithm is used to extract feature points from each returned image; Match the feature points extracted from different returned images to obtain feature point pairs; Perform camera pose estimation based on the feature point pairs to obtain initial camera pose information corresponding to each returned image; Taking minimizing the reprojection error of the three-dimensional point cloud as the optimization goal, the bundle adjustment algorithm is used to optimize the initial camera posture information corresponding to each returned image to obtain the camera posture information corresponding to each returned image.

4. The real-time 3D reconstruction method based on UAV flight according to claim 2, It is characterized in that The dense point cloud reconstruction is performed based on the OpenMVS framework and the camera posture information corresponding to each returned image in each group of returned images to obtain a dense point cloud corresponding to each group of returned images, including: Determine the potential neighboring frames corresponding to each returned image; Determine the depth map corresponding to each returned image based on the camera pose information corresponding to each returned image and the potential neighboring frames corresponding to each returned image; Optimize the depth map corresponding to each returned image; Based on the confidence level, the optimized depth map corresponding to each returned image is fused to obtain a fused depth map; The fused depth map is converted into a three-dimensional point cloud and fused to obtain a dense point cloud corresponding to each set of returned images.

5. The real-time 3D reconstruction method based on UAV flight according to claim 4, It is characterized in that The determining of the potential neighboring frames corresponding to each returned image includes: For each returned image, the following steps are performed: Taking the current returned image as a reference frame, and determining a candidate neighboring frame according to the current returned image; For each of the candidate neighborhood frames, an evaluation score of each candidate neighborhood frame is determined according to a minimum coverage area of ​​a common viewpoint in the reference frame and the candidate neighborhood frame, a weight score of an angle between the common viewpoint and the candidate neighborhood frame, and a weight score of a resolution similarity between the reference frame and the candidate neighborhood frame; the common viewpoint is a feature point that belongs to both a feature point set of the reference frame and a feature point set of the candidate neighborhood frame; Based on the evaluation score of each candidate neighborhood frame, a potential neighborhood frame corresponding to the current returned image is determined.

6. The real-time 3D reconstruction method based on UAV flight according to claim 4, It is characterized in that The method of determining a depth map corresponding to each returned image based on the camera posture information corresponding to each returned image and the potential neighboring frames corresponding to each returned image includes: For each returned image, the following steps are performed: The current returned image and a corresponding potential neighboring frame are regarded as an image pair; Determine the induced unitarity corresponding to each image pair based on the camera pose information corresponding to the current returned image and the camera pose information corresponding to the potential neighboring frame in each image pair; For each pixel in the current returned image, the following steps are performed: According to the current pixel, a window including the current pixel is set; Determine, according to the induced unitary property corresponding to each image pair, a corresponding pixel point in a potential neighborhood frame for each pixel point within the window under each image pair; Determine a normalized cross-correlation coefficient corresponding to each pixel point in the window under each image pair according to the induced unitary property corresponding to each image pair and the corresponding pixel point in the potential neighborhood frame of each pixel point in the window under each image pair; Determining a maximum normalized cross-correlation coefficient according to the normalized cross-correlation coefficient corresponding to each pixel point in the window under each image pair; Based on the maximum normalized cross-correlation coefficient, a depth value of the current pixel is determined.

7. The real-time 3D reconstruction method based on UAV flight according to claim 4, It is characterized in that The optimizing the depth map corresponding to each returned image includes: For each pixel in each returned image, the following steps are performed: According to the depth value of the current pixel and the camera posture information corresponding to the current returned image, the current pixel is back-projected into the three-dimensional space to obtain the three-dimensional space point corresponding to the current pixel; Projecting the three-dimensional space point to each potential neighboring frame corresponding to the current returned image, and determining a depth value corresponding to the three-dimensional space point in each potential neighboring frame; Determine the number of target potential neighborhood frames corresponding to the current pixel point according to a depth difference between a depth value corresponding to the three-dimensional space point in the potential neighborhood frame and a depth value of a camera corresponding to the three-dimensional space point relative to the potential neighborhood frame; The current pixel is optimized according to the number of target potential neighborhood frames corresponding to the current pixel.

8. A real-time 3D reconstruction device based on UAV flight, It is characterized in that include: A path planning module, used for planning the flight path of the UAV for the reconstruction area based on a circular flight strategy; the circular flight strategy is that each flight path of the UAV is a circular flight path of equal size, the centers of all circular flight paths are on a straight line, and the subsequent circular flight path passes through the center of the previous circular flight path; A processing module, configured to obtain a dense point cloud corresponding to each set of returned images based on a set of returned images corresponding to each circular flight path; A cropping module is used to crop the dense point cloud corresponding to each set of returned images based on the effective area, and perform Poisson reconstruction and texture mapping to obtain the three-dimensional reconstruction result corresponding to each set of returned images; the effective area is a rectangular area with the center of the circular flight path as the center of mass and the radius of the circular flight path as the side length; The stitching module is used to stitch the three-dimensional reconstruction results corresponding to all groups of returned images to obtain the three-dimensional reconstruction result corresponding to the area to be reconstructed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the real-time three-dimensional reconstruction method based on drone flight as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the real-time three-dimensional reconstruction method based on drone flight as described in any one of claims 1 to 7 is implemented.