Unmanned aerial vehicle image rapid splicing and three-dimensional reconstruction method based on deep learning

Abstract

The invention provides an unmanned aerial vehicle image rapid splicing and three-dimensional reconstruction method based on deep learning, and relates to the technical field of image splicing and three-dimensional reconstruction, and the method comprises the following steps: S01, dividing operation sub-regions for a plurality of images collected by an unmanned aerial vehicle based on the spatial frequency and pixel activeness of the images, a scale space is constructed in parallel by adopting a multi-thread collaborative SURF optimization algorithm, deep semantic features of the image are extracted by utilizing a deep neural network, hierarchical fusion is performed on the deep semantic features and shallow geometric features obtained by the SURF algorithm, and a composite feature vector is generated; and S02, based on the composite feature vector, carrying out rapid feature retrieval through an adaptive KD tree. According to the method, through a parallel multi-thread cooperation SURF algorithm, KD tree accelerated matching and deep learning fusion feature, image enhancement and depth estimation optimization are combined, high-precision rapid splicing and high-quality three-dimensional reconstruction of unmanned aerial vehicle images are realized, the efficiency and the model trueness are remarkably improved, and the method is adaptive to operation of common computing equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of image stitching and three-dimensional reconstruction, and in particular to a method for rapid stitching and three-dimensional reconstruction of drone images based on deep learning. Background Art

[0002] In recent years, with the continuous improvement of UAV hardware performance and the rapid development of aerial survey technology, photogrammetry based on UAV platforms has become an important means of obtaining high-resolution surface data. However, in actual engineering applications, UAV image processing still faces many technical bottlenecks, especially in the two key links of image stitching and three-dimensional reconstruction.

[0003] In terms of image stitching, traditional feature point-based methods show obvious limitations when processing large-scale aerial survey data. On the one hand, their computational complexity grows exponentially with the amount of data, resulting in low processing efficiency. On the other hand, in weak texture areas or repeated texture scenes, traditional algorithms find it difficult to extract a sufficient number of stable feature points, which seriously affects the stitching accuracy.

[0004] For example, Chinese patent, publication number CN116977591A, discloses a method for three-dimensional reconstruction;

[0005] And, Chinese patent, publication number CN108428255B, discloses a real-time 3D reconstruction method based on drones

[0006] In the existing technologies including the above two patents, especially in the three-dimensional reconstruction link, since drone aerial surveys are limited by factors such as flight altitude and lighting conditions, the acquired images often have problems such as uneven exposure and motion blur, which directly affect the accuracy of depth estimation. In addition, when dealing with complex terrain, existing reconstruction algorithms often find it difficult to balance the relationship between detail preservation and noise suppression. Excessive smoothing will lead to the loss of terrain features, while excessive sensitivity will introduce a large amount of noise. Especially in high-frequency details such as building edges and vegetation areas, the three-dimensional models reconstructed by existing methods often have defects such as edge jaggedness and surface fragmentation, which are difficult to meet the needs of high-precision modeling. Summary of the Invention

[0007] In response to the above technical problems, the technical solution adopted by the present invention is a method for rapid stitching and 3D reconstruction of drone images based on deep learning, which includes the following steps:

[0008] S01: For multiple images collected by drones, we divide the computational sub-regions based on the spatial frequency and pixel activity of the images, and construct the scale space in parallel using a multi-threaded collaborative SURF optimization algorithm. We then use a deep neural network to extract the deep semantic features of the images and perform hierarchical fusion with the shallow geometric features obtained by the SURF algorithm to generate a composite feature vector.

[0009] S02. Based on the composite feature vector, a fast feature retrieval is performed through an adaptive KD tree, initial matching point pairs are screened in combination with a dynamic threshold, and an improved RANSAC algorithm including a local structural similarity verification mechanism is used to purify the matching results to obtain a set of matching point pairs with high confidence;

[0010] S03. Based on the set of high-confidence matching point pairs, an iterative least squares algorithm is used to calculate the transformation matrix between the images. After geometric correction and resampling of the images, pixel fusion weights are dynamically assigned through fuzzy logic reasoning based on texture entropy and brightness gradient information of the overlapping areas. The overlapping areas of the multiple images are fused to output a seamless panoramic image.

[0011] S04. A deep denoising and enhancement network based on a residual attention module performs noise suppression and detail enhancement on the panoramic image, and uses block parallel computing and dynamic load balancing technology to accelerate processing, thereby building a binocular stereo vision deep learning model.

[0012] S05. Calibrate the camera intrinsic parameter matrix K based on a multi-objective joint optimization algorithm int , while fusing high-precision measurement data with the Kalman filter algorithm to optimize the external parameter matrix, combined with the depth estimation results of the binocular stereo vision deep learning model, through the external parameter matrix [R|t] ext Generate point cloud;

[0013] S06. For the point cloud, using an improved DBSCAN algorithm based on density peak search to remove outlier noise points, and performing density compensation on sparse areas using a variable coefficient radial basis function interpolation algorithm to generate a uniformly distributed high-density point cloud;

[0014] S07. Based on the high-density point cloud, a Poisson surface reconstruction algorithm is used to optimize mesh generation in combination with local point cloud density, curvature, and normal vector direction consistency. At the same time, illumination-invariant texture is extracted from the original image, brightness differences are eliminated through adaptive illumination correction, and bilinear interpolation technology is used to map the texture to a three-dimensional mesh surface, and a three-dimensional model is output.

[0015] Preferably, the multi-threaded collaborative SURF optimization algorithm in step S01 constructs a scale space in parallel, and dynamically allocates thread resources according to the sub-region entropy value and gradient mean. The scale space is constructed using the formula:

[0016]

[0017] in, ω(x,y) and β(x,y) are weight and adjustment coefficient respectively, is the Laplace operator, is the convolution operation, σ is the scale parameter, and (x, y) represents the coordinate position in the image.

[0018] Preferably, the deep neural network in step S01 adopts a nested multi-scale feature pyramid structure, using the formula:

[0019]

[0020] in, is the corresponding feature vector component, α1, α2, α3 are fusion weight coefficients, n is the vector dimension, ⊙ is the element-by-element multiplication operation, is the shallow feature of the i-th sample, is the deep feature of the i-th sample.

[0021] As a preference, the adaptive KD tree in step S02 adjusts the splitting strategy according to the feature vector distribution density and dimension correlation, the improved RANSAC algorithm introduces a local structure similarity verification mechanism, and the matching point pair screening adopts the improved Mahalanobis distance formula

[0022]

[0023] Among them, Σ is the covariance matrix, γ is the angle influence coefficient, angle(p i ,p j ) is the angle between the feature point direction vectors, p i and p j Indicates two feature points.

[0024] Preferably, in the process of fusing the overlapping areas of the multiple images in step S03, the multiple images are fused according to the multi-dimensional information of the overlapping areas and the pixels of the overlapping areas by fuzzy logic inference weights, and the formula is:

[0025]

[0026] Among them, I1(P) and I2(P) are pixel values, w1(P), w2(P), and w3(P) are weights, and ⊕ is a custom fusion operation.

[0027] Preferably, the binocular stereo vision deep learning model in step S04 adopts a space-channel dual-dimensional attention mechanism, and adopts a scheduling strategy based on task priority and resource utilization in block parallel;

[0028] The formula of the spatial-channel dual-dimensional attention mechanism is:

[0029]

[0030] Among them, N(x,y) represents the neighborhood centered on the pixel point (x,y), Sspatial ((x,y),(m,n)) is the spatial attention function.

[0031] As an example, the external parameter matrix [R|t] in step S05 is ext When generating point clouds, the camera intrinsic parameter matrix K int Using multi-objective joint optimization self-calibration algorithm, the external parameter matrix [R|t] ext By integrating high-precision measurement data and the Kalman filter algorithm, the variable coefficient radial basis function interpolation formula is used in the sparse area of ​​the point cloud:

[0032]

[0033] Among them, P0 represents the target point to be interpolated, P i Points with known attribute values ​​and location information, k represents the number of known points involved in the interpolation calculation, w i is the i-th known point P i The weight, λ i are the parameters of the interpolation function.

[0034] Preferably, the calculation of optimizing mesh generation by using the Poisson surface reconstruction algorithm in step S07 in combination with local point cloud density, curvature and normal vector direction consistency is as follows:

[0035]

[0036] The grid topology adopts a hierarchical region growing strategy, where i is an index variable used to traverse the m points involved in the calculation, and m represents the total number of points involved in the normal vector optimization calculation. i ,y i ,z i ) represents the coordinate position of the i-th point in three-dimensional space, represents the original normal vector of the i-th point, ρ represents the density function, cos(θ i ) where θ i Represents the original normal vector of the i-th point and other reference directions.

[0037] Preferably, during the texture mapping in step S07, a method based on illumination clustering is used to correct the texture, and a technique such as bilinear interpolation is combined to map the texture, using the formula:

[0038] Q enhanced =λ1·PSNR+λ2·SSIM+λ3·ENT

[0039] Evaluate the denoising and enhancement effect, where λ1, λ2, and λ3 are three weight coefficients, ENT is information entropy, PSNR is an indicator to measure the degree of image distortion, and SSIM is a measure of the structural similarity between two images.

[0040] The present invention has at least the following beneficial effects:

[0041] 1. Feature point extraction and detection are performed through a parallel multi-threaded collaborative SURF optimization algorithm, which significantly shortens the time for constructing the scale space and feature point detection. At the same time, the fast nearest neighbor search algorithm based on the KD tree and the parallel block depth estimation strategy significantly improve the speed of feature point matching and depth estimation, thereby greatly improving the efficiency of the entire image stitching process, which can meet the needs of fast stitching of large-area and multi-strip drone images.

[0042] 2. The deep features extracted by deep learning are integrated with the shallow features of the multi-threaded collaborative SURF optimization algorithm to construct a new feature vector. This increases the ground object information contained in the feature vector, making feature point matching more accurate. The matching results are purified by combining the RANSAC algorithm to effectively remove mismatched point pairs, improve the accuracy of image stitching, reduce stitching errors, and make the stitched image more accurately reflect the actual scene.

[0043] 3. The image is pre-processed based on a deep learning denoising network and image enhancement algorithm to remove noise and enhance image quality, providing better data for depth estimation. The use of a deep learning model for depth estimation can more accurately obtain the depth information of each pixel in the image, and the generated point cloud data is more precise. After steps such as point cloud optimization, mesh generation, and texture mapping, the final 3D model has a smoother surface and more realistic texture, which can better restore the actual scene and improve the quality of 3D reconstruction.

[0044] 4. A parallel processing strategy is adopted in each processing link, such as parallel construction of scale space, parallel detection of feature points, parallel depth estimation, etc., to fully utilize the computing resources of multi-core processors. Without increasing hardware costs, the execution efficiency of the algorithm is improved, the consumption of computing resources is reduced, and the method can run quickly on computers with ordinary configurations. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1This is a flowchart of a method for rapid stitching and 3D reconstruction of drone images based on deep learning provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0048] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0049] Example 1

[0050] This embodiment provides a method for rapid stitching and 3D reconstruction of drone images based on deep learning, which includes the following steps: Figure 1 As shown:

[0051] S01: For multiple images collected by drones, we divide the computational sub-regions based on the spatial frequency and pixel activity of the images, and construct the scale space in parallel using a multi-threaded collaborative SURF optimization algorithm. We then use a deep neural network to extract the deep semantic features of the images and perform hierarchical fusion with the shallow geometric features obtained by the SURF algorithm to generate a composite feature vector.

[0052] The multi-threaded collaborative SURF optimization algorithm in the above steps constructs the scale space in parallel, dynamically allocating thread resources based on the sub-region entropy value and gradient mean. The scale space construction adopts the formula:

[0053]

[0054] in, ω(x,y) and β(x,y) are weight and adjustment coefficient respectively, is the Laplace operator, is the convolution operation, σ is the scale parameter, and (x, y) represents the coordinate position in the image.

[0055] Furthermore, the deep neural network in the step adopts a nested multi-scale feature pyramid structure, using the formula:

[0056]

[0057] in, is the corresponding feature vector component, α1, α2, α3 are fusion weight coefficients, n is the vector dimension, ⊙ is the element-by-element multiplication operation, is the shallow feature of the i-th sample, is the deep feature of the i-th sample.

[0058] Specifically, after acquiring the drone image, the multi-threaded collaborative SURF optimization algorithm is used to divide the operation sub-areas according to the image spatial frequency and pixel activity. Each thread constructs the scale space in parallel, and the threads dynamically allocate resources based on the sub-area entropy value and gradient mean to ensure that the scale space is constructed efficiently and accurately. After the construction is completed, each sub-area synchronously detects feature points. By calculating the determinant value of the Hessian matrix, it is identified as a potential feature point when it is greater than the preset threshold, and the information is summarized in the feature point summary module.

[0059] Intercept the 32×32 pixel image blocks around the feature points detected by the multi-threaded collaborative SURF optimization algorithm, input the CNN network with a nested multi-scale feature pyramid structure, and output the deep feature F deep , and combine it with the multi-threaded collaborative SURF optimization algorithm shallow feature F surf F = [F surf ,F deep ] and normalize the scale.

[0060] S02. Based on the composite feature vector, fast feature retrieval is performed through an adaptive KD tree, initial matching point pairs are screened using a dynamic threshold, and an improved RANSAC algorithm with a local structural similarity verification mechanism is used to refine the matching results to obtain a set of matching point pairs with high confidence.

[0061] The adaptive KD tree in the above steps adjusts the splitting strategy according to the feature vector distribution density and dimension correlation. The improved RANSAC algorithm introduces a local structure similarity verification mechanism, and the matching point pair screening adopts the improved Mahalanobis distance formula.

[0062]

[0063] Among them, Σ is the covariance matrix, γ is the angle influence coefficient, angle(p i ,p j ) is the angle between the feature point direction vectors, p i and p j Indicates two feature points.

[0064] Specifically, a KD tree is constructed using F. The splitting strategy is dynamically adjusted based on the density of feature vectors and dimensional correlation, and the tree structure is regularly optimized. In the KD tree, matching candidates are initially screened using Euclidean distance, followed by a secondary screening using the modified Mahalanobis distance. The screened matching point pairs are then fed into the RANSAC algorithm, which randomly samples the image transformation model multiple times. The model with the most inliers is selected, and mismatched point pairs are removed. Based on the matching point pairs purified by RANSAC, an iterative least squares fitting optimization algorithm is used, using the affine transformation matrix as an example to solve the transformation matrix. Based on the transformation matrix, bilinear interpolation is used to resample the image.

[0065] S03. Based on the set of high-confidence matching point pairs, the transformation matrix between the images is calculated using the iterative least squares algorithm. After geometric correction and resampling of the images, pixel fusion weights are dynamically assigned through fuzzy logic reasoning based on the texture entropy and brightness gradient information of the overlapping areas. The overlapping areas of the multiple images are fused to output a seamless panoramic image.

[0066] In the above steps, when fusing the overlapping areas of multiple images, the multiple images are combined according to the multi-dimensional information of the overlapping areas and the pixels of the overlapping areas are fused through fuzzy logic inference weights. The formula is:

[0067]

[0068] Among them, I1(P) and I2(P) are pixel values, w1(P), w2(P), and w3(P) are weights, and ⊕ is a custom fusion operation.

[0069] In the above technology, for overlapping areas, the fusion weight is calculated by fuzzy logic reasoning according to the texture entropy value, brightness gradient, etc., and the pixels are fused to generate a spliced ​​image. At the same time, the image is input into the DnCNN denoising network designed based on the new residual attention module. The denoising parameters are dynamically adjusted according to the noise intensity and texture complexity. Histogram equalization is used on the denoised image. The grayscale histogram is first counted, and then the grayscale distribution is adjusted to the 0,255 range. Adaptive histogram equalization is used to retain local details.

[0070] S04: A deep denoising and enhancement network based on the residual attention module suppresses noise and enhances details in panoramic images. Block-based parallel computing and dynamic load balancing technology are used to accelerate processing and build a binocular stereo vision deep learning model.

[0071] In the above steps, the binocular stereo vision deep learning model adopts a spatial-channel dual-dimensional attention mechanism, and the block parallelization adopts a scheduling strategy based on task priority and resource utilization;

[0072] The formula of the spatial-channel dual-dimensional attention mechanism is:

[0073]

[0074] Among them, N(x,y) represents the neighborhood centered on the pixel point (x,y), S spatial ((x,y),(m,n)) is the spatial attention function.

[0075] S05. Calibrate the camera intrinsic parameter matrix K based on a multi-objective joint optimization algorithm int At the same time, the high-precision measurement data is integrated with the Kalman filter algorithm to optimize the external parameter matrix, and the depth estimation results of the binocular stereo vision deep learning model are combined to obtain the external parameter matrix [R|t] ext Generate point cloud;

[0076] In the above steps, the external parameter matrix [R|t] ext When generating point clouds, the camera intrinsic parameter matrix K int Using multi-objective joint optimization self-calibration algorithm, the external parameter matrix [R|t] ext By integrating high-precision measurement data and the Kalman filter algorithm, the variable coefficient radial basis function interpolation formula is used in the sparse area of ​​the point cloud:

[0077]

[0078] Among them, P0 represents the target point to be interpolated, P i Points with known attribute values ​​and location information, k represents the number of known points involved in the interpolation calculation, w i is the i-th known point P i The weight, λ i are the parameters of the interpolation function.

[0079] S06. For the point cloud, the improved DBSCAN algorithm based on density peak search is used to remove outlier noise points, and the variable coefficient radial basis function interpolation algorithm is used to compensate the density of sparse areas to generate a uniformly distributed high-density point cloud;

[0080] S07. Based on high-density point clouds, the Poisson surface reconstruction algorithm is used to optimize mesh generation by combining local point cloud density, curvature, and normal vector direction consistency. At the same time, illumination-invariant textures are extracted from the original images, and brightness differences are eliminated through adaptive illumination correction. The textures are mapped to the 3D mesh surface using bilinear interpolation technology to output a 3D model.

[0081] In the above steps, the Poisson surface reconstruction algorithm is used in combination with the local point cloud density, curvature and normal vector direction consistency to optimize the calculation of mesh generation as follows:

[0082]

[0083] The grid topology adopts a hierarchical region growing strategy, where i is an index variable used to traverse the m points involved in the calculation, and m represents the total number of points involved in the normal vector optimization calculation. i ,y i ,z i ) represents the coordinate position of the i-th point in three-dimensional space, represents the original normal vector of the i-th point, ρ represents the density function, cos(θ i ) where θ i Represents the original normal vector of the i-th point and other reference directions.

[0084] Furthermore, in the above texture mapping, the texture is corrected using a method based on illumination clustering, and the texture is mapped using techniques such as bilinear interpolation, using the formula:

[0085] Q enhanced =λ1·PSNR+λ2·SSIM+λ3·ENT

[0086] Evaluate the denoising and enhancement effect, where λ1, λ2, and λ3 are three weight coefficients, ENT is information entropy, PSNR is an indicator to measure the degree of image distortion, and SSIM is a measure of the structural similarity between two images.

[0087] Specifically, a convolutional neural network depth estimation model with a spatial-channel dual-dimensional attention mechanism is constructed. The preprocessed and calibrated binocular image is input. For large-scale images, it is divided into 512×512 pixel sub-blocks, and each sub-block is input into the model in parallel. Data transmission is dynamically scheduled according to task priority and resource utilization, and sub-blocks with rich textures and large depth changes are prioritized. The attention weight is calculated, and the depth value is output in combination with the image features. Based on the depth value, the camera intrinsic parameter matrix calibrated by multi-objective joint optimization and the extrinsic parameter matrix fused with high-precision measurement data are combined to generate the original point cloud. The camera parameters are calibrated regularly, and the DBSCAN algorithm based on density peak search is used to determine the density threshold and neighborhood radius. The core points are searched to construct clusters, outliers are eliminated, and parameters are adaptively adjusted according to the point cloud distribution. For sparse point cloud areas, an interpolation method based on variable coefficient radial basis function is used for filling optimization. The local density and curvature of the point cloud are monitored in real time, and the interpolation coefficients are dynamically adjusted.

[0088] A Poisson surface reconstruction algorithm based on the optimization calculation of the local density, curvature and normal vector direction consistency of the point cloud is used to optimize the normal vector calculation and construct an implicit function to generate a triangular mesh. A hierarchical region growing strategy is adopted to extract the mesh vertex texture information from the original image. The illumination distribution is analyzed through illumination clustering, the region is divided for correction and normalization of the texture, and the texture is mapped using bilinear interpolation and edge smoothing technology to generate a realistic three-dimensional model.

[0089] In summary, the first embodiment of the present invention uses a parallelized multi-threaded collaborative SURF optimization algorithm to extract and detect feature points, which greatly shortens the time in the scale space construction and feature point detection stages. At the same time, the fast nearest neighbor search algorithm based on the KD tree and the parallelized block depth estimation strategy significantly improve the speed of feature point matching and depth estimation, thereby greatly improving the efficiency of the entire image stitching process and meeting the needs of fast stitching of large-area, multi-strip drone images. Secondly, the deep-level features extracted by deep learning are integrated with the shallow-level features of the multi-threaded collaborative SURF optimization algorithm to construct a new feature vector, which increases the ground object information contained in the feature vector and makes the feature point matching more accurate. The matching results are purified by combining the RANSAC algorithm, effectively removing mismatched point pairs, improving the accuracy of image stitching, reducing stitching errors, and making the stitched image more accurately reflect the actual scene. Furthermore, a deep learning-based denoising network and image enhancement algorithm preprocess the image to remove noise and enhance image quality, providing better data for depth estimation. Using a deep learning model for depth estimation can more accurately obtain the depth information of each pixel in the image, and the generated point cloud data is more precise. After steps such as point cloud optimization, mesh generation, and texture mapping, the resulting 3D model has a smoother surface and more realistic texture, which can better restore the actual scene and improve the quality of 3D reconstruction. Finally, a parallel processing strategy is adopted in each processing step, such as parallel construction of scale space, parallel detection of feature points, and parallel depth estimation. This fully utilizes the computing resources of multi-core processors. Without increasing hardware costs, this improves the algorithm's execution efficiency and reduces computing resource consumption, enabling the method to run quickly on computers with ordinary configurations.

[0090] Example 2

[0091] Based on the above embodiment 1, when the multi-threaded collaborative SURF optimization algorithm constructs the scale space in step S01, the formula is used:

[0092]

[0093] Among them, ω1 and ω2 are dynamic weight coefficients, which are adjusted in real time according to the sub-region entropy value and gradient mean;

[0094] Specifically, the weights are adjusted in real time through the entropy value (reflecting texture complexity) and gradient mean (reflecting edge strength) of the sub-region, and computing resources are allocated preferentially to high-complexity areas (such as building edges and areas with dense vegetation), avoiding redundant calculations in low-activity areas (such as the sky and water surface), thereby improving the overall processing speed.

[0095] Fusion of Gaussian kernel G(x,y,σ) and Laplace operator Stable feature point extraction is possible even in low-light or blurred images, improving the matching success rate. This provides more accurate shallow-level geometric features (such as edges and corners) for the composite feature vector in the subsequent step S02, complementing the deep semantic features (such as object outlines and semantic segmentation) extracted by the deep neural network, significantly improving the robustness of feature matching.

[0096] Furthermore, the weight coefficient ω of the variable coefficient radial basis function interpolation in step S06 is i Through the adaptive adjustment of local point cloud density, the formula is:

[0097]

[0098] Among them, ρ(x i ) is the local density function of the point cloud.

[0099] Specifically, in sparse point cloud areas (such as tree gaps and building facade depressions), according to the local density ρ(x i ) dynamically assigns weights, reducing the weights of high-density areas to avoid overfitting and increasing the weights of low-density areas to compensate for missing data, thereby reducing reconstruction errors in sparse areas. Density-adaptive interpolation fills point cloud holes caused by blind spots or occlusions in the drone's field of view, ensuring a continuous and smooth 3D mesh surface. This also provides a uniformly distributed high-density point cloud for Poisson surface reconstruction in step S07. Combined with local density and curvature to optimize normal vector calculation, the ability to restore mesh details (such as brick wall texture and vegetation morphology) is significantly improved.

[0100] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the present profession can make some changes or modifications to equivalent embodiments of the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A method for rapid stitching and 3D reconstruction of drone images based on deep learning, characterized by: The method comprises the following steps: S01: For multiple images collected by drones, we divide the computational sub-regions based on the spatial frequency and pixel activity of the images, and construct the scale space in parallel using a multi-threaded collaborative SURF optimization algorithm. We then use a deep neural network to extract the deep semantic features of the images and perform hierarchical fusion with the shallow geometric features obtained by the SURF algorithm to generate a composite feature vector. S02. Based on the composite feature vector, a fast feature retrieval is performed through an adaptive KD tree, initial matching point pairs are screened in combination with a dynamic threshold, and an improved RANSAC algorithm including a local structural similarity verification mechanism is used to purify the matching results to obtain a set of matching point pairs with high confidence; S03. Based on the set of high-confidence matching point pairs, an iterative least squares algorithm is used to calculate the transformation matrix between the images. After geometric correction and resampling of the images, pixel fusion weights are dynamically assigned through fuzzy logic reasoning based on texture entropy and brightness gradient information of the overlapping areas. The overlapping areas of the multiple images are fused to output a seamless panoramic image. S04. A deep denoising and enhancement network based on a residual attention module performs noise suppression and detail enhancement on the panoramic image, and uses block parallel computing and dynamic load balancing technology to accelerate processing, thereby building a binocular stereo vision deep learning model. S05. Calibrate the camera intrinsic parameter matrix K based on a multi-objective joint optimization algorithm int , while fusing high-precision measurement data with the Kalman filter algorithm to optimize the external parameter matrix, combined with the depth estimation results of the binocular stereo vision deep learning model, through the external parameter matrix [R|t] ext Generate point cloud; S06. For the point cloud, using an improved DBSCAN algorithm based on density peak search to remove outlier noise points, and performing density compensation on sparse areas using a variable coefficient radial basis function interpolation algorithm to generate a uniformly distributed high-density point cloud; S07. Based on the high-density point cloud, a Poisson surface reconstruction algorithm is used to optimize mesh generation in combination with local point cloud density, curvature, and normal vector direction consistency. At the same time, illumination-invariant texture is extracted from the original image, brightness differences are eliminated through adaptive illumination correction, and bilinear interpolation technology is used to map the texture to a three-dimensional mesh surface, and a three-dimensional model is output.

2. The method for rapid stitching and 3D reconstruction of drone images based on deep learning according to claim 1, characterized in that: In step S01, the multi-threaded collaborative SURF optimization algorithm constructs a scale space in parallel, and dynamically allocates thread resources according to the sub-region entropy value and gradient mean. The scale space is constructed using the formula: in, ω(x,y) and β(x,y) are weight and adjustment coefficient respectively, is the Laplace operator, is the convolution operation, σ is the scale parameter, and (x, y) represents the coordinate position in the image.

3. The method for rapid stitching and 3D reconstruction of drone images based on deep learning according to claim 1, characterized in that: The deep neural network in step S01 adopts a nested multi-scale feature pyramid structure, using the formula: in, is the corresponding feature vector component, α1, α2, α3 are fusion weight coefficients, n is the vector dimension, ⊙ is the element-by-element multiplication operation, is the shallow feature of the i-th sample, is the deep feature of the i-th sample.

4. The method for rapid stitching and 3D reconstruction of drone images based on deep learning according to claim 1, characterized in that: In step S02, the adaptive KD tree adjusts the splitting strategy according to the feature vector distribution density and dimension correlation, the improved RANSAC algorithm introduces a local structure similarity verification mechanism, and the matching point pair screening adopts the improved Mahalanobis distance formula. Among them, Σ is the covariance matrix, γ is the angle influence coefficient, angle(p i ,p j ) is the angle between the feature point direction vectors, p i and p j Indicates two feature points.

5. The method for rapid stitching and 3D reconstruction of drone images based on deep learning according to claim 1, characterized in that: In the process of fusing the overlapping areas of the multiple images in step S03, the multiple images are fused according to the multi-dimensional information of the overlapping areas and the pixels of the overlapping areas through fuzzy logic inference weights. The formula is: Among them, I1(P), I2(P) are pixel values, w1(P), w2(P), w3(P) are weights, It is a custom fusion operation.

6. The method for rapid UAV image stitching and 3D reconstruction based on deep learning according to claim 1, characterized in that: The binocular stereo vision deep learning model in step S04 adopts a spatial-channel dual-dimensional attention mechanism, and adopts a scheduling strategy based on task priority and resource utilization in parallel in blocks; The formula of the spatial-channel dual-dimensional attention mechanism is: Among them, N(x,y) represents the neighborhood centered on the pixel point (x,y), S spatial ((x,y),(m,n)) is the spatial attention function.

7. The method for rapid UAV image stitching and 3D reconstruction based on deep learning according to claim 1, characterized in that: The external parameter matrix [R|t] in step S05 ext When generating point clouds, the camera intrinsic parameter matrix K int Using multi-objective joint optimization self-calibration algorithm, the external parameter matrix [R|t] ext By integrating high-precision measurement data and the Kalman filter algorithm, the variable coefficient radial basis function interpolation formula is used in the sparse area of ​​the point cloud: Among them, P0 represents the target point to be interpolated, P i Points with known attribute values ​​and location information, k represents the number of known points involved in the interpolation calculation, w i is the i-th known point P i The weight, λ i are the parameters of the interpolation function.

8. The method for rapid UAV image stitching and 3D reconstruction based on deep learning according to claim 1, characterized in that: The calculation of optimizing mesh generation by using the Poisson surface reconstruction algorithm combined with local point cloud density, curvature, and normal vector direction consistency in step S07 is as follows: The grid topology adopts a hierarchical region growing strategy, where i is an index variable used to traverse the m points involved in the calculation, and m represents the total number of points involved in the normal vector optimization calculation. i ,y i ,z i ) represents the coordinate position of the i-th point in three-dimensional space, represents the original normal vector of the i-th point, ρ represents the density function, cos(θ i ) where θ i Represents the original normal vector of the i-th point and other reference directions.

9. The method for rapid UAV image stitching and 3D reconstruction based on deep learning according to claim 1, characterized in that: During the texture mapping in step S07, the texture is corrected using a method based on illumination clustering, and the texture is mapped using techniques such as bilinear interpolation, using the formula: Q enhanced =λ1·PSNR+λ2·SSIM+λ3·ENT Evaluate the denoising and enhancement effect, where λ1, λ2, and λ3 are three weight coefficients, ENT is information entropy, PSNR is an indicator to measure the degree of image distortion, and SSIM is a measure of the structural similarity between two images.

Images