End-to-end 3D reconstruction method and device based on drone images

Abstract

The present invention discloses an end-to-end 3D reconstruction method and device based on drone images, the method comprising: obtaining drone aerial images; screening key images of the aerial images based on a parallax analysis strategy to obtain a key image set; extracting features of the key images based on a scale-invariant feature conversion operator and a parallel processing strategy to obtain feature points of each key image, and performing pairwise matching calculations on the feature points between the key images based on a fast approximate nearest neighbor algorithm to obtain matching feature point pairs; estimating the camera pose based on the matching feature point pairs and a normalized RANSAC algorithm, and calculating the 3D coordinates of the feature points according to the camera pose to obtain a sparse point cloud; performing depth map calculation on the sparse point cloud based on a depth map strategy to obtain a dense point cloud; and constructing a triangular mesh on the dense point cloud based on a voxel reconstruction algorithm to generate a scene 3D model. The present invention improves the efficiency and accuracy of 3D reconstruction.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to an end-to-end three-dimensional reconstruction method and device based on drone images. Background Art

[0002] In recent years, with the development of the low-altitude digital economy, images collected by drones equipped with imaging sensors have gradually become the main source of three-dimensional reconstruction data for large outdoor scenes.

[0003] In the related art, the image-based end-to-end 3D reconstruction method includes: the first is a scene 3D reconstruction method based on image voxels, which is specifically implemented by dividing the 3D object into a number of small cubic units, and then reconstructing it using the position and color information of the image voxels. However, the memory usage resources of this method will increase cubically with the increase of image resolution. Therefore, it is only suitable for 3D reconstruction tasks of small scenes or single objects. The second is a scene 3D reconstruction method based on patches. Although this method can be applied to the reconstruction task of large scenes, the topological relationship between patches will have problems such as overlap and missing, resulting in poor integrity of the reconstructed 3D model. The third is to use the colmap algorithm for end-to-end scene 3D reconstruction, but this method has low reconstruction efficiency.

[0004] Therefore, no effective solution has been proposed for the problems of low reconstruction efficiency and poor visual effect of the reconstructed scene when performing 3D reconstruction of high-resolution large scenes in existing end-to-end 3D reconstruction methods. Summary of the invention

[0005] The embodiments of the present invention provide an end-to-end 3D reconstruction method and device based on drone images, aiming to solve the problems of low reconstruction efficiency and poor visual effect of the reconstructed scene when performing 3D reconstruction of high-resolution large scenes in the prior art methods.

[0006] In a first aspect, an embodiment of the present invention provides an end-to-end 3D reconstruction method based on drone images, the method comprising:

[0007] Acquire drone aerial images;

[0008] Based on the disparity analysis strategy, key images are screened for the drone aerial images to obtain a key image set for three-dimensional reconstruction;

[0009] Based on a scale-invariant feature conversion operator and a parallel processing strategy, feature extraction is performed on each key image in the key image set to obtain feature points of each key image, and based on a fast approximate nearest neighbor algorithm, pairwise matching calculations are performed on the feature points between the key images to obtain matching feature point pairs between each pair of key images;

[0010] The camera pose is estimated based on the matching feature point pairs between each pair of key images and the normalized RANSAC algorithm, and the three-dimensional coordinates of each feature point in the key image are calculated according to the camera pose and the matching feature point pairs under the camera pose to obtain a sparse point cloud;

[0011] A neighborhood image of the current key image is selected based on the sparse point cloud and the camera pose, and a depth value of each pixel in the current key image is calculated by a region growing algorithm based on the neighborhood image. After a depth map of the current key image is obtained based on the depth value, the depth maps of all key images in the key image set are fused to obtain a dense point cloud;

[0012] A triangular mesh is constructed for the dense point cloud based on a voxel reconstruction algorithm to generate a three-dimensional model of the scene corresponding to the drone aerial image.

[0013] In a second aspect, an embodiment of the present invention provides an end-to-end 3D reconstruction device based on drone images, comprising:

[0014] An acquisition unit, used for acquiring drone aerial images;

[0015] An image processing unit, used for performing key image screening on the drone aerial image based on a parallax analysis strategy to obtain a key image set for three-dimensional reconstruction;

[0016] The sparse point cloud construction unit is used to extract features from each key image in the key image set based on a scale-invariant feature conversion operator and a parallel processing strategy to obtain feature points of each key image, and to perform pairwise matching calculations on the feature points between the key images based on a fast approximate nearest neighbor algorithm to obtain matching feature point pairs between each pair of key images.

[0017] The camera pose is estimated based on the matching feature point pairs between each pair of key images and the normalized RANSAC algorithm, and the three-dimensional coordinates of each feature point in the key image are calculated according to the camera pose and the matching feature point pairs under the camera pose to obtain a sparse point cloud;

[0018] A dense point cloud construction unit is used to select a neighborhood image of the current key image based on the sparse point cloud and the camera pose, and calculate the depth value of each pixel in the current key image by a region growing algorithm based on the neighborhood image, and after obtaining the depth map of the current key image based on the depth value, fuse the depth maps of all key images in the key image set to obtain a dense point cloud;

[0019] The three-dimensional model generating unit is used to construct a triangular mesh for the dense point cloud based on a voxel reconstruction algorithm to generate a three-dimensional model of the scene corresponding to the drone aerial image.

[0020] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0021] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the method described in the first aspect above.

[0022] The embodiments of the present invention provide an end-to-end three-dimensional reconstruction method and device based on drone images. Acquire UAV aerial images; screen key images of UAV aerial images based on parallax analysis strategy to obtain a key image set for three-dimensional reconstruction; extract features of each key image in the key image set based on a scale-invariant feature conversion operator and a parallel processing strategy to obtain feature points of each key image, and perform pairwise matching calculations on the feature points between each key image based on a fast approximate nearest neighbor algorithm to obtain matching feature point pairs between each pair of key images; estimate the camera pose based on the matching feature point pairs between each pair of key images and the normalized RANSAC algorithm, and calculate the three-dimensional coordinates of each feature point in the key image based on the camera pose and the matching feature point pairs under the camera pose to obtain a sparse point cloud; select the neighborhood image of the current key image based on the sparse point cloud and the camera pose, and calculate the depth value of each pixel in the current key image based on the neighborhood image through a region growing algorithm, and after obtaining the depth map of the current key image based on the depth value, fuse the depth maps of all key images in the key image set to obtain a dense point cloud; construct a triangular mesh for the dense point cloud based on a voxel reconstruction algorithm to generate a three-dimensional model of the scene corresponding to the UAV aerial image.

[0023] In the above method, firstly, the key views are screened out from the drone aerial images by the parallax analysis strategy, and a more robust image can be obtained for subsequent feature point extraction and matching; then, in order to improve the computational efficiency, the present invention adopts the CPU multi-core parallel and GPU accelerated working mode to perform SIFT feature extraction and feature matching on the key views, and then estimates the camera pose through the matching results and the normalized RANSAC algorithm, and obtains the sparse point cloud based on the camera pose by the bundle adjustment calculation; then, the present invention performs depth map calculation on the sparse point cloud based on the depth map strategy to obtain a dense point cloud, thereby obtaining a complete and visually good scene outline; finally, in order to improve the reconstruction accuracy, the present invention adopts the region growing voxel reconstruction algorithm to generate a triangular mesh to obtain a three-dimensional model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0025] Figure 1 A schematic diagram of a process flow of an end-to-end 3D reconstruction method based on drone images provided by an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a sub-process of an end-to-end 3D reconstruction method based on drone images provided by an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of another sub-process of the end-to-end 3D reconstruction method based on drone images provided by an embodiment of the present invention;

[0028] Figure 4 A schematic block diagram of an end-to-end 3D reconstruction device based on drone images provided by an embodiment of the present invention;

[0029] Figure 5 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0032] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0033] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0034] The embodiment of the present invention provides an end-to-end 3D reconstruction method based on drone images. Figure 1 A schematic diagram of a process flow of an end-to-end 3D reconstruction method based on drone images provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes steps S110 to S160.

[0035] S110: Acquire drone aerial images.

[0036] In this embodiment, aerial images of the scene to be reconstructed are acquired by using a drone. It should be noted that drone aerial images have the characteristics of high overlap and super resolution, and are very suitable for three-dimensional reconstruction of large scenes.

[0037] S120, screening key images of the drone aerial images based on the parallax analysis strategy to obtain a key image set for three-dimensional reconstruction.

[0038] Redundant images will cause a lot of time to be wasted in subsequent image processing, feature extraction and feature point matching, and the redundant feature points will also cause mismatching and other problems. Therefore, in this embodiment, the key images of the drone aerial images are screened based on the parallax analysis strategy to obtain the key image set for 3D reconstruction.

[0039] In one embodiment, if Figure 2 As shown, step S120 includes sub-steps S121, S122, S123, S124 and S125.

[0040] S121. Extract feature points from the drone aerial image based on a feature extraction strategy, and use a starting drone aerial image in the drone aerial image as a reference image.

[0041] Specifically, feature points are extracted from each drone aerial image acquired in step S110 using a commonly used image feature extraction method to obtain s evenly distributed feature points, and the starting drone aerial image in the drone aerial images is used as a reference image.

[0042] S122. Perform feature tracking on the feature points extracted from the starting UAV aerial image based on the LK optical flow method to obtain a tracking image containing a preset number of successfully tracked feature points from the remaining UAV aerial images, and calculate the average disparity value between the tracking image and the reference image based on the disparity between each successfully tracked feature point in the tracking image and the corresponding feature point in the reference image.

[0043] Specifically, the LK optical flow method is used to perform feature tracking on the s feature points extracted from the starting drone aerial image, so as to obtain a tracking image from the remaining drone aerial images in which the feature point tracking of the starting drone aerial image is successful and the number of successful tracking is a preset number m. Then, the disparity between each successfully tracked feature point in the tracking image and each corresponding feature point in the reference image is calculated to obtain each disparity value between each successfully tracked feature point in the reference image and the tracking image, and an average calculation is performed based on each calculated disparity value to obtain an average disparity value dm between the tracking image and the reference image.

[0044] S123: Compare the average disparity value between the tracking image and the reference image with a preset threshold to obtain a comparison result.

[0045] Specifically, the average disparity value dm of m feature points in the tracking image that successfully track the reference image is used as a judgment basis, and the average disparity value dm between the tracking image and the reference image is compared with a preset threshold dT to obtain a comparison result.

[0046] S124. When the comparison result shows that the average disparity value between the tracking image and the reference image is greater than or equal to a preset threshold, the tracking image is set as a key image, and the key image is used as a current reference image to obtain the next key image from the remaining drone aerial images.

[0047] S125. When the comparison result shows that the average disparity value between the tracking image and the reference image is less than the preset threshold, continue to perform feature tracking on the feature points extracted from the starting UAV aerial image until a key image is obtained.

[0048] S130, extracting features from each key image in the key image set based on a scale-invariant feature conversion operator and a parallel processing strategy to obtain feature points of each key image, and performing pairwise matching calculations on the feature points between the key images based on a fast approximate nearest neighbor algorithm to obtain matching feature point pairs between each pair of key images.

[0049] Since the scene images collected by drones are large in scale and rich in content, it takes a long time to extract features from drone aerial images. To solve this problem, this embodiment uses CPU multi-core parallelism and GPU acceleration to extract features of the scale-invariant feature conversion operator on the key views obtained above, thereby improving computational efficiency.

[0050] Specifically, based on the scale-invariant feature transformation operator and the parallel processing strategy, feature extraction is performed on each key image in the key image set to obtain feature points of each key image, including: obtaining key images from the key image set; upsampling, downsampling and color space conversion are performed on the key images in the constructed scale space by means of CPU multi-core parallel processing to obtain the converted key images; then, key point detection is performed on the converted key images in the constructed Gaussian difference pyramid by means of GPU accelerated processing to obtain local extreme points as key points, and feature descriptors are calculated based on the local extreme points to obtain feature points of the key images.

[0051] After extracting the feature points of all key images through the above steps, the feature points between the key images are matched pairwise based on the fast approximate nearest neighbor algorithm to obtain matching feature point pairs between each pair of key images. It should be noted that GPU is used for acceleration when matching feature point pairs to further reduce the time complexity of the calculation process and improve calculation efficiency.

[0052] S140, estimating the camera pose based on the matching feature point pairs between each pair of key images and the normalized RANSAC algorithm, and calculating the three-dimensional coordinates of each feature point in the key image according to the camera pose and the matching feature point pairs under the camera pose to obtain a sparse point cloud.

[0053] Preferably, in this embodiment, Figure 3 As shown, the camera pose is estimated based on the matching feature point pairs between each pair of key images and the normalized RANSAC algorithm, including sub-steps S141-S144.

[0054] S141. Obtain matching feature point pairs between the current key image pairs.

[0055] S142. Randomly select a preset number of matching feature point pairs from the matching feature point pairs, and normalize the preset number of matching feature point pairs based on the normalized RANSAC algorithm to obtain a basic matrix so that the center of gravity of the same image coincides with the origin of the image coordinate system and the mean square of the distance from each image point in the image to the origin is a preset threshold.

[0056] Specifically, 8 pairs of matching feature points are randomly selected from the matching feature point pairs between the current key image pairs. The 8 pairs of matching feature points are normalized based on the normalized RANSAC algorithm to obtain a basic matrix so that the center of gravity of the same image coincides with the origin of the image coordinate system and the mean square of the distance from each image point in the image to the origin is a preset threshold, such as 2.

[0057] S143, verifying and calculating the basic matrix through the remaining matching feature point pairs in the matching feature point pairs, recording the number of matching feature point pairs that satisfy the basic matrix, and obtaining the number of point pairs.

[0058] Specifically, the remaining matching feature point pairs are obtained from all matching feature point pairs between the current key image pair, and then the basic matrix obtained by these remaining matching feature point pairs is verified and calculated, and the number of matching feature point pairs that satisfy the basic matrix is ​​recorded to obtain the number of point pairs N.

[0059] S144. Obtain the basic matrix corresponding to the maximum number of point pairs, and decompose the basic matrix to obtain the camera pose.

[0060] Specifically, steps S142-S143 are repeated to perform calculations based on randomly selected different matching feature point pairs, a basic matrix corresponding to the maximum value of the point pair data N is selected, and the basic matrix is ​​decomposed to obtain the camera pose.

[0061] Next, the three-dimensional coordinates of each feature point in the key image are calculated based on the camera pose obtained above and the matching feature point pairs under the camera pose to obtain a sparse point cloud. It should be noted that when solving the three-dimensional coordinates of the feature points, this embodiment mainly uses a nonlinear method that minimizes the reprojection error to solve, thereby avoiding the noise and error sensitivity of the linear direct solution method and effectively dealing with the shortcomings of nonlinear distortion.

[0062] S150. Select a neighborhood image of the current key image based on the sparse point cloud and the camera pose, and calculate the depth value of each pixel in the current key image through a region growing algorithm based on the neighborhood image. After obtaining the depth map of the current key image based on the depth value, fuse the depth maps of all the key images in the key image set to obtain a dense point cloud.

[0063] In this embodiment, firstly, the neighborhood images of the current key image are selected based on the sparse point cloud and the camera pose, so as to construct a global image set. Then, the depth value of each pixel in the current key image is calculated by the region growing algorithm based on the obtained neighborhood images, and the depth map of the current key image is obtained based on the calculated depth value.

[0064] Similarly, the depth maps of all key images in the key image set are obtained, and these depth maps are fused to obtain a dense point cloud.

[0065] S160, constructing a triangular mesh of the dense point cloud based on a voxel reconstruction algorithm to generate a three-dimensional model of the scene corresponding to the drone aerial image.

[0066] In this embodiment, a dense point cloud is triangulated based on a voxel reconstruction algorithm, including: performing denoising and filtering on the dense point cloud based on a data preprocessing strategy to obtain a preprocessed dense point cloud; to reduce the influence of noise points and outliers. Then, the preprocessed dense point cloud is mapped to a three-dimensional voxel according to a preset voxel resolution, and a seed point is randomly selected from the preprocessed dense point cloud as a growth starting point, and a triangulated mesh is constructed for the area where the growth starting point is located based on the voxel reconstruction algorithm until a complete area is formed.

[0067] In one embodiment, a triangular mesh is constructed for the area where the growth starting point is located based on a voxel reconstruction algorithm until a complete area is formed, and then the constructed triangular mesh is screened based on a screening strategy to remove redundant triangular meshes to obtain a screened triangular mesh; then, a topological structure optimization process is performed on the screened triangular mesh based on an optimization strategy to obtain an optimized triangular mesh.

[0068] The voxel reconstruction algorithm using region growing in this implementation can effectively parallelize the processing during three-dimensional reconstruction and can retain more local details in the reconstructed scene.

[0069] In one embodiment, after generating a scene 3D model corresponding to the drone aerial image, in order to enhance the visual effect of the 3D model, the surface of the 3D model may be texture mapped using multiple registered texture images. Specifically, a number of key texture images with feature point matching are obtained; the number of key texture images with feature point matching are spliced ​​and fused to obtain a fused texture image; the scene 3D model is texture mapped based on the fused texture image to obtain a scene 3D model with high-quality texture.

[0070] In the end-to-end 3D reconstruction method based on drone images provided in an embodiment of the present invention, drone aerial images are obtained; key images of the drone aerial images are screened based on a parallax analysis strategy to obtain a key image set for 3D reconstruction; feature extraction is performed on each key image in the key image set based on a scale-invariant feature conversion operator and a parallel processing strategy to obtain feature points of each key image, and pairwise matching calculations are performed on the feature points between each key image based on a fast approximate nearest neighbor algorithm to obtain matching feature point pairs between each pair of key images; matching feature point pairs between each pair of key images and normalized RANSAC are used to obtain the matching feature point pairs between each pair of key images; The algorithm estimates the camera pose and calculates the three-dimensional coordinates of each feature point in the key image based on the camera pose and the matching feature point pairs under the camera pose to obtain a sparse point cloud; based on the sparse point cloud and camera pose, the neighborhood image of the current key image is selected, and the depth value of each pixel in the current key image is calculated by the region growing algorithm based on the neighborhood image. After the depth map of the current key image is obtained based on the depth value, the depth maps of all key images in the key image set are fused to obtain a dense point cloud; based on the voxel reconstruction algorithm, a triangular mesh is constructed for the dense point cloud to generate a three-dimensional model of the scene corresponding to the drone aerial image. In the above method, firstly, the key views are screened out from the drone aerial images by the parallax analysis strategy, and a more robust image can be obtained for subsequent feature point extraction and matching; then, in order to improve the computational efficiency, the present invention adopts the CPU multi-core parallel and GPU accelerated working mode to perform SIFT feature extraction and feature matching on the key views, and then estimates the camera pose through the matching results and the normalized RANSAC algorithm, and obtains the sparse point cloud based on the camera pose by the bundle adjustment calculation; then, the present invention performs depth map calculation on the sparse point cloud based on the depth map strategy to obtain a dense point cloud, thereby obtaining a complete and visually good scene outline; finally, in order to improve the reconstruction accuracy, the present invention adopts the region growing voxel reconstruction algorithm to generate a triangular mesh to obtain a three-dimensional model.

[0071] In order to verify the effectiveness of the present invention, the openMVG and colmap open source algorithms, which have advantages in both computational efficiency and reconstruction accuracy, were selected for comparative analysis with the present invention. In terms of reconstruction efficiency, 5 sets of public drone aerial image data sets of different scales were selected for ablation experiments, and the two evaluation indicators of feature extraction time and feature matching time were analyzed; in terms of reconstruction accuracy, 2 sets of public data sets with real ground data were selected for experiments, and the deviation between the camera position estimated by the algorithm and the actual camera position was used as the evaluation indicators for analysis.

[0072] Table 1 is a time comparison table of different algorithms for image feature extraction and feature matching in different scenes. As shown in Table 1, the time used for feature extraction and feature matching in the five scenes by the present invention is significantly less than that of the colmap and openMVG algorithms. And as the scene data set increases, the rate advantage of the present invention for feature extraction and feature matching will further increase, which shows that the method of selecting key views by disparity analysis in the present invention can filter redundant images in large scenes, thereby significantly improving efficiency, especially in the task of 3D reconstruction of large scenes with high resolution, which has a better reconstruction efficiency advantage;

[0073] Table 1 Comparison of the time required for image feature extraction and feature matching by different algorithms in different scenarios

[0074]

[0075] Table 2 is a comparison table of the deviations between the camera position estimated by different algorithms and the actual position under different data sets. As shown in Table 2, the camera position deviation estimated by the scheme of the present invention is significantly smaller than that of the colmap and openMVG algorithms, which indicates that the accuracy of estimating the camera pose by the normalized RANSAC algorithm proposed in the present invention is significantly higher than that of the colmap and openMVG algorithms.

[0076] Table 2 is a comparison of the deviation between the camera position estimated by different algorithms and the actual position under different data sets.

[0077]

[0078] In addition, in order to further verify that the present invention can perform end-to-end fast 3D model reconstruction in high-resolution large-scene 3D reconstruction tasks, the present invention scheme is compared and analyzed with the existing end-to-end 3D reconstruction system under the same experimental conditions and data sets. The specific experimental data are 129 and 159 images obtained by DJI UAV's 360-degree aerial photography of a church and the Jinzifeng Protection Station of Baishanzu National Park. Table 3 is a comparison of the time used in each stage by the present invention and the colmap, openMVG+openMVS end-to-end 3D reconstruction system, as shown in Table 3:

[0079] Table 3 Timeline of the three different 3D reconstruction algorithms used in each stage

[0080]

[0081] According to the experimental results, the time used by the present invention in the sparse point cloud construction stage, the dense point cloud construction stage and the overall operation is significantly less than that of the colmap and openMVG+openMVS algorithms. This shows that when the present invention performs end-to-end three-dimensional scene reconstruction, the efficiency of reconstructing the three-dimensional model is significantly improved, and as the scene data set increases, the efficiency advantage gradually increases.

[0082] The embodiment of the present invention also provides an end-to-end 3D reconstruction device based on drone images. Figure 4 A schematic block diagram of an end-to-end 3D reconstruction device based on drone images provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the end-to-end 3D reconstruction device 100 based on drone images includes an acquisition unit 110, an image processing unit 120, a sparse point cloud construction unit 130, a dense point cloud construction unit 140 and a 3D model generation unit 150.

[0083] The acquisition unit 110 is used to acquire the drone aerial image.

[0084] In this embodiment, aerial images of the scene to be reconstructed are acquired by using a drone. It should be noted that drone aerial images have the characteristics of high overlap and super resolution, and are very suitable for three-dimensional reconstruction of large scenes.

[0085] The image processing unit 120 is used to perform key image screening on the drone aerial image based on a parallax analysis strategy to obtain a key image set for three-dimensional reconstruction.

[0086] Redundant images will cause a lot of time to be wasted in subsequent image processing, feature extraction and feature point matching, and the redundant feature points will also cause mismatching and other problems. Therefore, in this embodiment, the key images of the drone aerial images are screened based on the parallax analysis strategy to obtain the key image set for 3D reconstruction.

[0087] In one embodiment, the image processing unit 120 further includes the following.

[0088] Feature points are extracted from the UAV aerial images based on the feature extraction strategy, and the starting UAV aerial image in the UAV aerial images is used as a reference image.

[0089] Specifically, feature points are extracted from each drone aerial image acquired in step S110 using a commonly used image feature extraction method to obtain s evenly distributed feature points, and the starting drone aerial image in the drone aerial images is used as a reference image.

[0090] Based on the LK optical flow method, feature tracking is performed on the feature points extracted from the starting UAV aerial image to obtain a tracking image containing a preset number of successfully tracked feature points from the remaining UAV aerial images, and the average disparity value between the tracking image and the reference image is calculated based on the disparity between each successfully tracked feature point in the tracking image and the corresponding feature point in the reference image.

[0091] Specifically, the LK optical flow method is used to perform feature tracking on the s feature points extracted from the starting drone aerial image, so as to obtain a tracking image from the remaining drone aerial images in which the feature point tracking of the starting drone aerial image is successful and the number of successful tracking is a preset number m. Then, the disparity between each successfully tracked feature point in the tracking image and each corresponding feature point in the reference image is calculated to obtain each disparity value between each successfully tracked feature point in the reference image and the tracking image, and an average calculation is performed based on each calculated disparity value to obtain an average disparity value dm between the tracking image and the reference image.

[0092] The average disparity value between the tracking image and the reference image is compared with a preset threshold to obtain a comparison result.

[0093] Specifically, the average disparity value dm of m feature points in the tracking image that successfully track the reference image is used as a judgment basis, and the average disparity value dm between the tracking image and the reference image is compared with a preset threshold dT to obtain a comparison result.

[0094] When the comparison result is that the average disparity value between the tracking image and the reference image is greater than or equal to the preset threshold, the tracking image is set as the key image, and the key image is used as the current reference image to obtain the next key image from the remaining drone aerial images.

[0095] When the comparison result shows that the average disparity value between the tracking image and the reference image is less than the preset threshold, feature tracking of the feature points extracted from the starting UAV aerial image continues until the key image is obtained.

[0096] The sparse point cloud construction unit 130 is used to extract features from each key image in the key image set based on a scale-invariant feature conversion operator and a parallel processing strategy to obtain feature points of each key image, and perform pairwise matching calculations on the feature points between each key image based on a fast approximate nearest neighbor algorithm to obtain matching feature point pairs between each pair of key images; estimate the camera pose based on the matching feature point pairs between each pair of key images and a normalized RANSAC algorithm, and calculate the three-dimensional coordinates of each feature point in the key image based on the camera pose and the matching feature point pairs under the camera pose to obtain a sparse point cloud.

[0097] Since the scene images collected by drones are large in scale and rich in content, it takes a long time to extract features from drone aerial images. To solve this problem, this embodiment uses CPU multi-core parallelism and GPU acceleration to extract features of the scale-invariant feature conversion operator on the key views obtained above, thereby improving computational efficiency.

[0098] Specifically, based on the scale-invariant feature transformation operator and the parallel processing strategy, feature extraction is performed on each key image in the key image set to obtain feature points of each key image, including: obtaining key images from the key image set; upsampling, downsampling and color space conversion are performed on the key images in the constructed scale space by means of CPU multi-core parallel processing to obtain the converted key images; then, key point detection is performed on the converted key images in the constructed Gaussian difference pyramid by means of GPU accelerated processing to obtain local extreme points as key points, and feature descriptors are calculated based on the local extreme points to obtain feature points of the key images.

[0099] After extracting the feature points of all key images through the above steps, the feature points between the key images are matched pairwise based on the fast approximate nearest neighbor algorithm to obtain matching feature point pairs between each pair of key images. It should be noted that GPU is used for acceleration when matching feature point pairs to further reduce the time complexity of the calculation process and improve calculation efficiency.

[0100] Next, the camera pose is estimated based on the matching feature point pairs between each pair of key images and the normalized RANSAC algorithm, and the three-dimensional coordinates of each feature point in the key image are calculated according to the camera pose and the matching feature point pairs under the camera pose to obtain a sparse point cloud.

[0101] Specifically, the camera pose is estimated based on the matching feature point pairs between each pair of key images and the normalized RANSAC algorithm, including: first, obtaining the matching feature point pairs between the current key image pair. Then, a preset number of matching feature point pairs are randomly selected from the matching feature point pairs, and the preset number of matching feature point pairs are normalized based on the normalized RANSAC algorithm to obtain a basic matrix, so that the centroid of the same image coincides with the origin of the image coordinate system and the mean square of the distance from each image point in the image to the origin is a preset threshold. Specifically, 8 pairs of matching feature point pairs are randomly selected from the matching feature point pairs between the current key image pair. The 8 pairs of matching feature point pairs are normalized based on the normalized RANSAC algorithm to obtain a basic matrix, so that the centroid of the same image coincides with the origin of the image coordinate system and the mean square of the distance from each image point in the image to the origin is a preset threshold, such as 2. Then, the basic matrix is ​​verified and calculated by the remaining matching feature point pairs in the matching feature point pairs, and the number of matching feature point pairs that meet the basic matrix is ​​recorded to obtain the number of point pairs. Specifically, the remaining matching feature point pairs are obtained from all matching feature point pairs between the current key image pair, and then the basic matrix obtained by these remaining matching feature point pairs is verified and calculated, and the number of matching feature point pairs that satisfy the basic matrix is ​​recorded to obtain the number of point pairs N. Finally, the basic matrix corresponding to the maximum number of point pairs is obtained, and the basic matrix is ​​decomposed to obtain the camera pose. Specifically, the above steps are repeated, and calculations are performed based on different randomly selected matching feature point pairs, and the basic matrix corresponding to the maximum value of the point pair data N is selected, and the basic matrix is ​​decomposed to obtain the camera pose.

[0102] Next, the three-dimensional coordinates of each feature point in the key image are calculated based on the camera pose obtained above and the matching feature point pairs under the camera pose to obtain a sparse point cloud. It should be noted that when solving the three-dimensional coordinates of the feature points, this embodiment mainly uses a nonlinear method that minimizes the reprojection error to solve, thereby avoiding the noise and error sensitivity of the linear direct solution method and effectively dealing with the shortcomings of nonlinear distortion.

[0103] The dense point cloud construction unit 140 is used to select the neighborhood image of the current key image based on the sparse point cloud and the camera pose, and calculate the depth value of each pixel in the current key image based on the neighborhood image through the region growing algorithm. After obtaining the depth map of the current key image based on the depth value, the depth maps of all the key images in the key image set are fused to obtain a dense point cloud.

[0104] In this embodiment, firstly, the neighborhood images of the current key image are selected based on the sparse point cloud and the camera pose, so as to construct a global image set. Then, the depth value of each pixel in the current key image is calculated by the region growing algorithm based on the obtained neighborhood images, and the depth map of the current key image is obtained based on the calculated depth value.

[0105] Similarly, the depth maps of all key images in the key image set are obtained, and these depth maps are fused to obtain a dense point cloud.

[0106] The three-dimensional model generating unit 150 is used to construct a triangular mesh for the dense point cloud based on a voxel reconstruction algorithm to generate a three-dimensional model of the scene corresponding to the drone aerial image.

[0107] In this embodiment, a dense point cloud is triangulated based on a voxel reconstruction algorithm, including: performing denoising and filtering on the dense point cloud based on a data preprocessing strategy to obtain a preprocessed dense point cloud; to reduce the influence of noise points and outliers. Then, the preprocessed dense point cloud is mapped to a three-dimensional voxel according to a preset voxel resolution, and a seed point is randomly selected from the preprocessed dense point cloud as a growth starting point, and a triangulated mesh is constructed for the area where the growth starting point is located based on the voxel reconstruction algorithm until a complete area is formed.

[0108] In one embodiment, a triangular mesh is constructed for the area where the growth starting point is located based on a voxel reconstruction algorithm until a complete area is formed, and then the constructed triangular mesh is screened based on a screening strategy to remove redundant triangular meshes to obtain a screened triangular mesh; then, a topological structure optimization process is performed on the screened triangular mesh based on an optimization strategy to obtain an optimized triangular mesh.

[0109] The voxel reconstruction algorithm using region growing in this implementation can effectively parallelize the processing during three-dimensional reconstruction and can retain more local details in the reconstructed scene.

[0110] In one embodiment, after generating a scene 3D model corresponding to the drone aerial image, in order to enhance the visual effect of the 3D model, the surface of the 3D model may be texture mapped using multiple registered texture images. Specifically, a number of key texture images with feature point matching are obtained; the number of key texture images with feature point matching are spliced ​​and fused to obtain a fused texture image; the scene 3D model is texture mapped based on the fused texture image to obtain a scene 3D model with high-quality texture.

[0111] The end-to-end 3D reconstruction device based on drone images provided in an embodiment of the present invention is applied to the above-mentioned end-to-end 3D reconstruction method based on drone images, obtaining drone aerial images; performing key image screening on the drone aerial images based on a parallax analysis strategy to obtain a key image set for 3D reconstruction; performing feature extraction on each key image in the key image set based on a scale-invariant feature conversion operator and a parallel processing strategy to obtain feature points of each key image, and performing pairwise matching calculations on the feature points between each key image based on a fast approximate nearest neighbor algorithm to obtain matching feature point pairs between each pair of key images; and performing matching feature point pair extraction based on the matching feature point pairs between each pair of key images. The camera pose is estimated by the normalized RANSAC algorithm, and the three-dimensional coordinates of each feature point in the key image are calculated according to the camera pose and the matching feature point pairs under the camera pose to obtain a sparse point cloud; the neighborhood image of the current key image is selected based on the sparse point cloud and the camera pose, and the depth value of each pixel in the current key image is calculated by the region growing algorithm based on the neighborhood image. After the depth map of the current key image is obtained based on the depth value, the depth maps of all the key images in the key image set are fused to obtain a dense point cloud; a triangular mesh is constructed for the dense point cloud based on the voxel reconstruction algorithm to generate a three-dimensional model of the scene corresponding to the drone aerial image. In the above method, firstly, the key views are screened out from the drone aerial images by the parallax analysis strategy, and a more robust image can be obtained for subsequent feature point extraction and matching; then, in order to improve the computational efficiency, the present invention adopts the CPU multi-core parallel and GPU accelerated working mode to perform SIFT feature extraction and feature matching on the key views, and then estimates the camera pose through the matching results and the normalized RANSAC algorithm, and obtains the sparse point cloud based on the camera pose by the bundle adjustment calculation; then, the present invention performs depth map calculation on the sparse point cloud based on the depth map strategy to obtain a dense point cloud, thereby obtaining a complete and visually good scene outline; finally, in order to improve the reconstruction accuracy, the present invention adopts the region growing voxel reconstruction algorithm to generate a triangular mesh to obtain a three-dimensional model.

[0112] The above-mentioned end-to-end 3D reconstruction method based on drone images can be implemented in the form of a computer program. The computer program can be used in Figure 5 Runs on the computer device shown.

[0113] See also Figure 5 , Figure 5 is a schematic block diagram of a computer device provided by an embodiment of the present invention. The computer device may be used to perform an end-to-end 3D reconstruction method based on drone images.

[0114] See also Figure 5The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a storage medium 503 and an internal memory 504 .

[0115] The storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 may execute an end-to-end 3D reconstruction method based on drone images, wherein the storage medium 503 may be a volatile storage medium or a non-volatile storage medium.

[0116] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0117] The internal memory 504 provides an environment for the operation of the computer program 5032 in the storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute an end-to-end three-dimensional reconstruction method based on drone images.

[0118] The network interface 505 is used for network communication, such as providing data information transmission, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device 500 to which the solution of the present invention is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0119] The processor 502 is used to run a computer program 5032 stored in the memory to implement the corresponding functions in the above-mentioned end-to-end three-dimensional reconstruction method based on drone images.

[0120] Those skilled in the art will understand that Figure 5 The embodiments of the computer device shown in the figure do not constitute a limitation on the specific composition of the computer device. In other embodiments, the computer device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, in some embodiments, the computer device may only include a memory and a processor. In such embodiments, the structure and function of the memory and the processor are the same as those of the embodiment of the present invention. Figure 5 The embodiments shown are consistent and will not be described again here.

[0121] It should be understood that in the embodiment of the present invention, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0122] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps included in the above-mentioned end-to-end 3D reconstruction method based on drone images are implemented.

[0123] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the above-described equipment, devices and units can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. Those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0124] In the several embodiments provided by the present invention, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. Units with the same function may also be combined into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or units, or may be an electrical, mechanical or other form of connection.

[0125] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0126] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0127] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a computer-readable storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned computer-readable storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a magnetic disk or an optical disk.

[0128] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An end-to-end 3D reconstruction method based on drone images, characterized in that: The method comprises: Acquire drone aerial images; Based on the disparity analysis strategy, key images are screened for the drone aerial images to obtain a key image set for three-dimensional reconstruction; The method of performing key image screening on the drone aerial image based on the parallax analysis strategy to obtain a key view set for three-dimensional reconstruction includes: Extracting feature points from the drone aerial image based on a feature extraction strategy, and using a starting drone aerial image in the drone aerial image as a reference image; Based on the LK optical flow method, feature tracking is performed on the feature points extracted from the starting UAV aerial image to obtain a tracking image containing a preset number of successfully tracked feature points from the remaining UAV aerial images, and an average disparity value between the tracking image and the reference image is calculated based on the disparity between each successfully tracked feature point in the tracking image and each corresponding feature point in the reference image; Comparing the average disparity value between the tracking image and the reference image with a preset threshold to obtain a comparison result; When the comparison result is that the average disparity value between the tracking image and the reference image is greater than or equal to the preset threshold, setting the tracking image as a key image, and using the key image as a current reference image to obtain a next key image from the remaining drone aerial images; When the comparison result is that the average disparity value between the tracking image and the reference image is less than the preset threshold, continue to perform feature tracking on the feature points extracted from the starting drone aerial image until a key image is obtained; Based on a scale-invariant feature conversion operator and a parallel processing strategy, feature extraction is performed on each key image in the key image set to obtain feature points of each key image, and based on a fast approximate nearest neighbor algorithm, pairwise matching calculations are performed on the feature points between the key images to obtain matching feature point pairs between each pair of key images; The feature extraction of each key image in the key image set based on the scale-invariant feature conversion operator and the parallel processing strategy to obtain feature points of each key image includes: Acquire a key image from the key image set; The key image is upsampled, downsampled and converted in color space in the constructed scale space by CPU multi-core parallel processing to obtain a converted key image; Performing key point detection on the converted key image in the constructed Gaussian difference pyramid by GPU acceleration processing, obtaining local extreme points as key points, and performing feature descriptor calculation based on the key points to obtain feature points of the key image; The camera pose is estimated based on the matching feature point pairs between each pair of key images and the normalized RANSAC algorithm, and the three-dimensional coordinates of each feature point in the key image are calculated according to the camera pose and the matching feature point pairs under the camera pose to obtain a sparse point cloud; A neighborhood image of the current key image is selected based on the sparse point cloud and the camera pose, and a depth value of each pixel in the current key image is calculated by a region growing algorithm based on the neighborhood image. After a depth map of the current key image is obtained based on the depth value, the depth maps of all key images in the key image set are fused to obtain a dense point cloud; A triangular mesh is constructed for the dense point cloud based on a voxel reconstruction algorithm to generate a three-dimensional model of the scene corresponding to the drone aerial image.

2. The method according to claim 1, characterized in that The camera pose is estimated based on the matching feature point pairs between each pair of key images and the normalized RANSAC algorithm, including: Get the matching feature point pairs between the current key image pairs; A preset number of matching feature point pairs are randomly selected from the matching feature point pairs, and the preset number of matching feature point pairs are normalized based on a normalized RANSAC algorithm to obtain a basic matrix, so that the center of gravity of the same image coincides with the origin of the image coordinate system and the mean square of the distance from each image point in the image to the origin is a preset threshold; Verifying and calculating the basic matrix through the remaining matching feature point pairs in the matching feature point pairs, recording the number of matching feature point pairs that satisfy the basic matrix, and obtaining the number of point pairs; The basic matrix corresponding to the maximum number of point pairs is obtained, and the basic matrix is ​​decomposed to obtain the camera pose.

3. The method according to claim 1, characterized in that The constructing a triangular mesh of the dense point cloud based on a voxel reconstruction algorithm includes: Performing denoising and filtering processing on the dense point cloud based on a data preprocessing strategy to obtain a preprocessed dense point cloud; The preprocessed dense point cloud is mapped into three-dimensional voxels according to a preset voxel resolution, and a seed point is randomly selected from the preprocessed dense point cloud as a growth starting point, and a triangular mesh is constructed for the area where the growth starting point is located based on a voxel reconstruction algorithm until a complete area is formed.

4. The method according to claim 3, characterized in that After constructing a triangular mesh for the region where the growth starting point is located based on the voxel reconstruction algorithm until a complete region is formed, the method includes: The constructed triangular mesh is screened based on the screening strategy to remove redundant triangular meshes and obtain a screened triangular mesh; The selected triangular mesh is subjected to topological structure optimization processing based on an optimization strategy to obtain an optimized triangular mesh.

5. The method according to claim 1, characterized in that After generating the three-dimensional model of the scene corresponding to the drone aerial image, the method includes: Obtaining several key texture images with matched feature points; Splicing and fusing the plurality of key texture images whose feature points have been matched to obtain a fused texture image; Texture mapping is performed on the scene three-dimensional model based on the fused texture image to obtain a scene three-dimensional model with high-quality texture.

6. An end-to-end 3D reconstruction device based on drone images, characterized in that: The device comprises: An acquisition unit, used for acquiring drone aerial images; An image processing unit, used for performing key image screening on the drone aerial image based on a parallax analysis strategy to obtain a key image set for three-dimensional reconstruction; The method of performing key image screening on the drone aerial image based on the parallax analysis strategy to obtain a key view set for three-dimensional reconstruction includes: Extracting feature points from the drone aerial image based on a feature extraction strategy, and using a starting drone aerial image in the drone aerial image as a reference image; Based on the LK optical flow method, feature tracking is performed on the feature points extracted from the starting UAV aerial image to obtain a tracking image containing a preset number of successfully tracked feature points from the remaining UAV aerial images, and an average disparity value between the tracking image and the reference image is calculated based on the disparity between each successfully tracked feature point in the tracking image and each corresponding feature point in the reference image; Comparing the average disparity value between the tracking image and the reference image with a preset threshold to obtain a comparison result; When the comparison result is that the average disparity value between the tracking image and the reference image is greater than or equal to the preset threshold, setting the tracking image as a key image, and using the key image as a current reference image to obtain a next key image from the remaining drone aerial images; When the comparison result is that the average disparity value between the tracking image and the reference image is less than the preset threshold, continue to perform feature tracking on the feature points extracted from the starting drone aerial image until a key image is obtained; A sparse point cloud construction unit, configured to extract features from each key image in the key image set based on a scale-invariant feature conversion operator and a parallel processing strategy to obtain feature points of each key image, and to perform pairwise matching calculations on the feature points between the key images based on a fast approximate nearest neighbor algorithm to obtain matching feature point pairs between each pair of key images, to estimate a camera pose based on the matching feature point pairs between each pair of key images and a normalized RANSAC algorithm, and to calculate three-dimensional coordinates of each feature point in the key image based on the camera pose and the matching feature point pairs under the camera pose to obtain a sparse point cloud; The feature extraction of each key image in the key image set based on the scale-invariant feature conversion operator and the parallel processing strategy to obtain feature points of each key image includes: Acquire a key image from the key image set; The key image is upsampled, downsampled and converted in color space in the constructed scale space by CPU multi-core parallel processing to obtain a converted key image; Performing key point detection on the converted key image in the constructed Gaussian difference pyramid by GPU acceleration processing, obtaining local extreme points as key points, and performing feature descriptor calculation based on the key points to obtain feature points of the key image; A dense point cloud construction unit is used to select a neighborhood image of the current key image based on the sparse point cloud and the camera pose, and calculate the depth value of each pixel in the current key image by a region growing algorithm based on the neighborhood image, and after obtaining the depth map of the current key image based on the depth value, fuse the depth maps of all key images in the key image set to obtain a dense point cloud; The three-dimensional model generating unit is used to construct a triangular mesh for the dense point cloud based on a voxel reconstruction algorithm to generate a three-dimensional model of the scene corresponding to the drone aerial image.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images