A dual-UAV 3D reconstruction method based on cloud-edge collaboration
By adopting a cloud-edge collaborative dual-UAV 3D reconstruction method, the problems of manual intervention and adaptability in large-scene 3D reconstruction are solved by utilizing the collaborative control of dual UAVs and cloud computing power, achieving efficient and automated 3D reconstruction results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHAANXI NPU SCI PARK CO LTD
- Filing Date
- 2022-11-15
- Publication Date
- 2026-07-31
AI Technical Summary
Existing technologies for large-scale 3D reconstruction suffer from several drawbacks, including the need for manual intervention in reconstruction results, high costs, poor performance of monocular vision systems, and the inability of binocular vision stereo matching to adapt in large outdoor scenes.
A cloud-edge collaborative dual-UAV 3D reconstruction method is adopted. Through steps such as static and dynamic calibration, image preprocessing, deep neural network feature extraction and matching, depth estimation, point cloud reconstruction and texture mapping, the method utilizes the collaborative control of dual UAVs and cloud computing power to achieve fast and efficient 3D reconstruction.
It enables automated 3D reconstruction in large scenes, reduces costs, improves reconstruction efficiency and accuracy, enhances the robustness and adaptability of the system, and provides ease of use and scenario versatility.
Smart Images
Figure CN115908731B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of dual-UAV 3D reconstruction, and more particularly to a dual-UAV 3D reconstruction method based on cloud-edge collaboration. Background Technology
[0002] Based on the fundamental principles of stereo vision, this method enables 3D reconstruction of terrain contours. A region-segmentation-based 3D reconstruction method is proposed based on terrain features. First, the classic watershed algorithm is used to segment the image, and the segmented boundaries are taken as the contour edges in the actual scene. Then, the global features of adjacent segmented regions, as well as the scale, location, and grayscale distribution features of each local region, are used as stereo matching constraints. This method not only reduces the matching search space but also improves efficiency. However, since this method is based on region segmentation, the contours seen from different viewpoints can vary, leading to increased matching errors.
[0003] In current real-scene 3D reconstruction systems, mainstream oblique photogrammetry technology struggles to achieve rapid reconstruction results in large scenes due to the heavy workload of aerial triangulation and modeling, as well as the requirements for aircraft size and flight time. Furthermore, the success rate of large-scale aerial triangulation decreases rapidly with the expansion of the photographic scene, often resulting in fragmented flight paths, layering, and bending in the reconstructed images. This necessitates repeated manual intervention, increasing reconstruction costs. SFM-based monocular vision systems, limited by dimensional loss in specific scenes, also perform poorly in certain reconstruction tasks, such as scenes with repetitive textures. Mismatched feature points can lead to incorrect camera pose estimation, resulting in incomplete reconstruction models. Additionally, as mentioned above, binocular cameras or simple dual-camera platforms based on stereo matching cannot adapt to large outdoor scenes, especially in weakly textured scenes (such as lakes or curtain walls) and large-scale scenes (such as cities). Since binocular matching requires overlapping areas, fixed-lens cameras and simple dual-camera control cannot handle related tasks. Similarly, RGBD cameras, which can simultaneously acquire image depth information, are also limited by the shooting range and cannot achieve adaptive adjustment. Therefore, a dual-UAV 3D reconstruction method based on cloud-edge collaboration is proposed. Summary of the Invention
[0004] The purpose of this invention is to provide a dual-UAV 3D reconstruction method based on cloud-edge collaboration, which solves the problems of existing mainstream oblique photogrammetry technology, which requires repeated manual intervention by operators after reconstruction, increasing reconstruction costs, poor reconstruction effect of monocular vision system, and the inability of binocular cameras or simple dual-UAV platforms based on binocular vision stereo matching to achieve adaptive performance in large outdoor scenes.
[0005] To achieve the above objectives, the present invention provides the following technical solution: a dual-UAV 3D reconstruction method based on cloud-edge collaboration, comprising the following steps:
[0006] S1: Dual-UAV Vision Platform
[0007] 1) Static calibration: The two UAVs respectively calibrated the intrinsic parameters of their monocular cameras. A 260x260mm square checkerboard calibration board was used to obtain the camera's focal length, optical center, and distortion coefficients. The static calibration adopted the Zhang Zhengyou calibration method, which relies on the registration process of corresponding points between the world coordinate system and the camera coordinate system.
[0008]
[0009] Where M1 is the camera intrinsic parameter matrix, α x ,α y The normalized lens focal length is represented by u0 and v0, which are the lateral and radial distortion coefficients, respectively.
[0010] 2) Dynamic calibration: The two UAVs will simultaneously adjust their poses during the sparse point cloud reconstruction stage and the dense point cloud reconstruction stage according to mission requirements;
[0011] 3) Image acquisition and data transmission: The synchronization delay time between the two machines is ms, and the transmission rate through the 5G chip is 107-700Mbps;
[0012] S2: Unmanned Aerial Vehicle Cloud System Platform
[0013] 1) Image preprocessing and correction: mainly denoising and deblurring of binocular images, and performing a certain degree of image enhancement;
[0014] 2) End-to-end image feature extraction and image matching:
[0015] (1) The disparity value of stereo matching is obtained directly through deep neural network regression for binocular two-view images. The network takes the left and right views as input, and obtains the predicted disparity and the residual estimated by the update network through inference of two networks respectively. Both networks use an autoencoder structure containing 5 layers of convolution and deconvolution as the backbone. In the main network structure, the input image first passes through the feature extraction of the convolutional layer containing the residual branch, and the similarity of the feature vector is calculated by using the dot product operation, so as to obtain the implicit expression of disparity estimation. The correlation volume obtained by the dot product is calculated as follows:
[0016] c(x1, x2) = ∑<f1(x1),f2(x2)>
[0017] (2) Subsequently, multi-scale pyramid refinement is performed using the above features. The refined disparity result is obtained using deconvolution layers. The coordinates of the right view can be transformed, and the coordinate-transformed image and the left and right views are used as inputs to the second network. This network will regress the corresponding residuals layer by layer and update them to different scales. The update process is as follows, where s represents the number of scales:
[0018]
[0019] (3) During network inference, in order to balance learning speed and estimation accuracy, the number of feature channels in the convolutional and deconvolutional layers of the network will be changed to alter the model size:
[0020]
[0021]
[0022] 3) Depth estimation: The depth information of relevant pixels is calculated by matching point pairs, and a depth map is generated. Depth estimation requires the depth information of 3D points based on the parallax network regression result d, the equivalent baseline length b, and the camera focal length f. The specific calculation is as follows:
[0023] Z = (b*f) / d
[0024] 4) Point cloud reconstruction: Based on the calculated dense depth map, the depth information can be directly converted into a 3D point cloud using the PCL library. For outdoor scenes of different scales, corresponding point cloud filtering and point cloud merging will be performed.
[0025] 5) Triangulation: The three-dimensional structure is divided into regions using triangular faces as the basic units of the division.
[0026] 6) Texture mapping: Fill in texture information using 2D images from different perspectives and optimize texture seams. Steps 5) and 6) mentioned above can be integrated using OpenMVS or CloudCompare software.
[0027] 7) Effect evaluation: The reconstruction effect is evaluated based on the rationality of geometric mapping and semantics;
[0028] S3: Task-side level:
[0029] 1) Ground preparation: Initialize and adjust system equipment parameters, perform self-checks for faults, complete the installation and commissioning of the carrier platform's mission load, and perform static calibration on the dual-aircraft platform;
[0030] 2) Sparse point cloud reconstruction: The dual-machine platform will adjust the attitude through dynamic calibration, use the cloud system's (1)-(5) steps to complete the maximum depth estimation for the reconstructed scene, and obtain sparse point cloud;
[0031] 3) Dense point cloud reconstruction: Based on the sparse point cloud, the dual-machine platform is fine-tuned again to fill in the information of pixels affected by factors such as lighting and occlusion, so as to obtain a dense point cloud.
[0032] 4) Real-world scene mapping: This mainly involves transforming point cloud representations into real 3D objects, which is entirely accomplished by steps 6) and 7) of the UAV cloud system platform;
[0033] 5) Effect evaluation: The information of the user terminal, cloud system and carrier platform is exchanged, and the position and attitude of the carrier platform are adjusted through low latency network data link based on the accuracy of cloud system calculation and user subjective perception.
[0034] Preferably, in the dynamic calibration of S1 2), during the sparse point cloud reconstruction stage, the altitude range that the two aircraft can adjust each time is -10-10m, and the equivalent distance that can be adjusted each time (which involves fine-tuning of the heading angle) is -0.5-0.5m. During the dense point cloud reconstruction stage, the two aircraft system will adjust the attitude angles, with the pitch angle adjustment range being -20-20 and the heading angle adjustment range being -10-10.
[0035] Preferably, in the image preprocessing and correction of S2 1), the key regions of the image are segmented according to the task requirements. The segmentation algorithm is based on the BlendMask network, which can obtain a segmentation mask for the instance, which can be used for the subsequent thinning process.
[0036] Preferably, in step 2)(1) of S2, the first network structure is used to estimate the final predicted disparity, and the second network is used to update the residual estimated by the network. Both networks use an autoencoder structure containing 5 layers of convolution and deconvolution as the backbone.
[0037] Preferably, in steps 2)(3) of S2, E-ratio and D-ratio represent multiples relative to the original convolutional layer channels and the deconvolutional layer, respectively. The E-ratio is selected to include 2, 4, 8, and 16; the corresponding matching D-ratios are 1, 4, 8, and 16. The obtained model size will increase accordingly as E-ratio and D-ratio increase.
[0038] Preferably, in the triangulation of S2 (5), the discrete and disordered three-dimensional point cloud is triangulated according to the Delaunay triangulation algorithm. During the iteration of the algorithm, new edges are introduced into the original point set each time, thereby expanding to obtain a three-dimensional surface.
[0039] Preferably, in the effect evaluation of S2 7), in terms of the evaluation of the reasonableness of geometric mapping, the chamfer distance will be calculated. The first part of this item ensures that the distance between the generated point cloud and the ground truth point cloud is minimized, and the second part ensures that the coverage of the ground truth point cloud in the generated point cloud is maximized. In terms of semantic evaluation, the ground truth semantic labels provided by the partial 3D segmentation dataset are mainly used to calculate the segmentation accuracy.
[0040] Compared with related technologies, the cloud-edge collaborative dual-UAV 3D reconstruction method provided by this invention has the following advantages:
[0041] 1. Use two drones equipped with monocular cameras to achieve binocular vision imaging, reduce surveying costs, and expand the measurement range of the binocular vision system.
[0042] 2. Utilizing dual-UAV collaborative control enables simple, large-scale camera pose adjustments. Based on altitude and parallax requirements, it improves single-shot imaging efficiency while ensuring high imaging resolution, adapting to scene mapping requirements at different scales both indoors and outdoors. Simultaneously, to optimize detail, micro-scale adjustments to the viewing angle can be used to adaptively adjust scene details. The relatively simple control strategy reduces the difficulty of multi-UAV collaboration, enhances system robustness, and strengthens adaptability to outdoor environments.
[0043] 3. Staged reconstruction and algorithm fusion can effectively improve reconstruction results. Reconstructing point clouds from sparse to dense can generate a clear workflow for the system and reduce the impact of sudden external disturbances on the system. The fusion of feature matching and region matching in depth estimation can improve reconstruction accuracy.
[0044] 4. The cloud-edge collaborative system architecture utilizes low-latency networks to achieve rapid real-time data transmission between UAVs and between unmanned devices and the cloud, improving reconstruction speed. Simultaneously, cloud computing power is used to integrate algorithms, enabling centralized and efficient processing of distributed sampling. Furthermore, the UAV system's pose is corrected in real-time through cloud-based performance evaluation.
[0045] 5. The ground terminal of the drone can provide users with beyond-line-of-sight manual intervention and real-time effect presentation, and has the advantages of ease of use and versatility in various scenarios. Attached Figure Description
[0046] Figure 1 This is a flowchart of the dual-UAV vision platform for a cloud-edge collaborative dual-UAV 3D reconstruction method according to the present invention.
[0047] Figure 2 This is a flowchart of the drone cloud system platform for a dual-drone 3D reconstruction method based on cloud-edge collaboration according to the present invention.
[0048] Figure 3This is a schematic diagram of the cloud-edge collaborative system framework for a dual-UAV 3D reconstruction method based on cloud-edge collaboration according to the present invention.
[0049] Figure 4 This is a system branch diagram of a dual-UAV 3D reconstruction method based on cloud-edge collaboration according to the present invention.
[0050] Figure 5 This is a flowchart illustrating the task execution of a dual-UAV 3D reconstruction method based on cloud-edge collaboration according to the present invention. Detailed Implementation
[0051] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0052] Please see Figure 1-5 This invention provides a technical solution: a dual-UAV 3D reconstruction method based on cloud-edge collaboration, comprising the following steps:
[0053] S1: Dual-UAV Vision Platform
[0054] 1) Static calibration: The two UAVs respectively calibrated the intrinsic parameters of their monocular cameras. A 260x260mm square checkerboard calibration board was used to obtain the camera's focal length, optical center, and distortion coefficients. The static calibration adopted the Zhang Zhengyou calibration method, which relies on the registration process of corresponding points between the world coordinate system and the camera coordinate system.
[0055]
[0056] Where M1 is the camera intrinsic parameter matrix, α x ,α y The normalized lens focal length is represented by u0 and v0, which are the lateral and radial distortion coefficients, respectively.
[0057] 2) Dynamic calibration: The two UAVs will simultaneously adjust their poses during the sparse point cloud reconstruction stage and the dense point cloud reconstruction stage according to mission requirements;
[0058] 3) Image acquisition and data transmission: The synchronization delay time between the two machines is ms, and the transmission rate through the 5G chip is 107-700Mbps;
[0059] S2: Unmanned Aerial Vehicle Cloud System Platform
[0060] 1) Image preprocessing and correction: mainly denoising and deblurring of binocular images, and performing a certain degree of image enhancement;
[0061] 2) End-to-end image feature extraction and image matching:
[0062] (1) The disparity value of stereo matching is obtained directly through deep neural network regression for binocular two-view images. The network takes the left and right views as input, and obtains the predicted disparity and the residual estimated by the update network through inference of two networks respectively. Both networks use an autoencoder structure containing 5 layers of convolution and deconvolution as the backbone. In the main network structure, the input image first passes through the feature extraction of the convolutional layer containing the residual branch, and the similarity of the feature vector is calculated by using the dot product operation, so as to obtain the implicit expression of disparity estimation. The correlation volume obtained by the dot product is calculated as follows:
[0063] c(x1, x2) = ∑<f1(x1),f2(x2)>
[0064] (2) Subsequently, multi-scale pyramid refinement is performed using the above features. The refined disparity result is obtained using deconvolution layers. The coordinates of the right view can be transformed, and the coordinate-transformed image and the left and right views are used as inputs to the second network. This network will regress the corresponding residuals layer by layer and update them to different scales. The update process is as follows, where s represents the number of scales:
[0065]
[0066] (3) During network inference, in order to balance learning speed and estimation accuracy, the number of feature channels in the convolutional and deconvolutional layers of the network will be changed to alter the model size:
[0067]
[0068]
[0069] 3) Depth estimation: The depth information of relevant pixels is calculated by matching point pairs, and a depth map is generated. Depth estimation requires the depth information of 3D points based on the parallax network regression result d, the equivalent baseline length b, and the camera focal length f. The specific calculation is as follows:
[0070] Z = (b*f) / d
[0071] 4) Point cloud reconstruction: Based on the calculated dense depth map, the depth information can be directly converted into a 3D point cloud using the PCL library. For outdoor scenes of different scales, corresponding point cloud filtering and point cloud merging will be performed.
[0072] 5) Triangulation: The three-dimensional structure is divided into regions using triangular faces as the basic units of the division.
[0073] 6) Texture mapping: Fill in texture information using 2D images from different perspectives and optimize texture seams. Steps 5) and 6) mentioned above can be integrated using OpenMVS or CloudCompare software.
[0074] 7) Effect evaluation: The reconstruction effect is evaluated based on the rationality of geometric mapping and semantics;
[0075] S3: Task-side level:
[0076] 1) Ground preparation: Initialize and adjust system equipment parameters, perform self-checks for faults, complete the installation and commissioning of the carrier platform's mission load, and perform static calibration on the dual-aircraft platform;
[0077] 2) Sparse point cloud reconstruction: The dual-machine platform will adjust the attitude through dynamic calibration, use the cloud system's (1)-(5) steps to complete the maximum depth estimation for the reconstructed scene, and obtain sparse point cloud;
[0078] 3) Dense point cloud reconstruction: Based on the sparse point cloud, the dual-machine platform is fine-tuned again to fill in the information of pixels affected by factors such as lighting and occlusion, so as to obtain a dense point cloud.
[0079] 4) Real-world scene mapping: This mainly involves transforming point cloud representations into real 3D objects, which is entirely accomplished by steps 6) and 7) of the UAV cloud system platform;
[0080] 5) Effect evaluation: The information of the user terminal, cloud system and carrier platform is exchanged, and the position and attitude of the carrier platform are adjusted through low latency network data link based on the accuracy of cloud system calculation and user subjective perception.
[0081] In the dynamic calibration of S1 (2), during the sparse point cloud reconstruction phase, the altitude range that the two aircraft can adjust each time is -10 to 10m, and the equivalent distance that can be adjusted each time (which involves fine-tuning of the heading angle) is -0.5 to 0.5m. During the dense point cloud reconstruction phase, the two aircraft system will adjust the attitude angles, with the pitch angle adjustment range being -20 to 20 and the heading angle adjustment range being -10 to 10.
[0082] In S2's 1) image preprocessing and correction, the key regions of the image are segmented according to the task requirements. The segmentation algorithm is based on the BlendMask network, which can obtain a segmentation mask for the instance, which can be used for subsequent thinning processes.
[0083] In S2(2)(1), the first network structure is used to estimate the final predicted disparity, and the second network is used to update the residual estimated by the network. Both networks use an autoencoder structure containing 5 layers of convolution and deconvolution as the backbone.
[0084] In S2(2)(3), E-ratio and D-ratio represent the multiples relative to the original convolutional layer channels and the deconvolutional layer, respectively. The E-ratio is selected to include 2, 4, 8, and 16; the corresponding matching D-ratios are 1, 4, 8, and 16. The size of the obtained model will increase accordingly as E-ratio and D-ratio increase.
[0085] In the 5) triangulation of S2, according to the Delaunay triangulation algorithm, the discrete and disordered 3D point cloud is triangularly connected. During the iteration of the algorithm, new edges are introduced into the original point set each time, thereby expanding to obtain a 3D surface.
[0086] In the S2 7) effect evaluation, in terms of the geometric mapping rationality evaluation, the chamfer distance will be calculated. The first part of this item ensures that the distance between the generated point cloud and the ground truth point cloud is minimized, and the second part ensures the coverage of the ground truth point cloud in the generated point cloud. In terms of semantic evaluation, the ground truth semantic labels provided by the partial 3D segmentation dataset are mainly used to calculate the segmentation accuracy.
[0087] This method needs to include two phases: building the platform-level framework and executing the task-level process.
[0088] At the platform level, a dual-UAV vision platform and a UAV cloud system platform will be constructed. The dual-UAV platform will act as the surveying carrier, responsible for collecting data and receiving feedback from the user terminal and the cloud system to adjust its pose. The UAV cloud system will integrate various algorithms for 3D reconstruction, be responsible for processing and analyzing the data from the carrier platform, and provide visualization results back to the user terminal.
[0089] I. The workflow of the dual-UAV vision platform is as follows:
[0090] 1. Static Calibration: Two UAVs respectively calibrated the intrinsic parameters of their onboard monocular cameras, using a 260x260mm square checkerboard calibration board to obtain the camera's focal length, optical center, and distortion coefficients. The static calibration employed the Zhang Zhengyou calibration method, which relies on the registration process between corresponding points in the world coordinate system and the camera coordinate system.
[0091]
[0092] Where M1 is the camera intrinsic parameter matrix, α x ,α y For the normalized lens focal length, u0 and v0 are the lateral and radial distortion coefficients, respectively.
[0093] 2. Dynamic Calibration: Both UAVs will simultaneously adjust their attitude according to mission requirements. During the sparse point cloud reconstruction phase, the altitude range that the two UAVs can adjust each time is -10 to 10m, and the equivalent distance that can be adjusted each time (involving fine-tuning of the heading angle) is -0.5 to 0.5m. During the dense point cloud reconstruction phase, the dual-UAV system will adjust its attitude angles, with the pitch angle adjustment range being -20 to 20° and the heading angle adjustment range being -10 to 10°.
[0094] 3. Image acquisition and data transmission: The synchronization delay time between the two machines is ms, and the transmission rate through the 5G chip is 107-700Mbps.
[0095] II. The workflow of the UAV cloud system platform is as follows:
[0096] 1. Image Preprocessing and Correction: This mainly involves denoising and deblurring the binocular images, and performing some image enhancement. Based on task requirements, key regions of the image are segmented (e.g., reconstructing the foreground and background). The segmentation algorithm will be based on the BlendMask network, which will obtain a segmentation mask specific to the instance, which can be used for subsequent thinning processes.
[0097] 2. End-to-end image feature extraction and image matching: For binocular two-view images, the disparity value for stereo matching is directly obtained through deep neural network regression. The network takes the left and right views as input and passes them through two separate networks for inference. The first network structure estimates the final predicted disparity, and the second network updates the residuals estimated by the network. Both networks use an autoencoder structure containing 5 layers of convolution and deconvolution as the backbone. In the main network structure, the input image first undergoes feature extraction through convolutional layers containing residual branches, and the similarity of feature vectors is calculated using dot product operations to obtain an implicit representation of the disparity estimation. The correlation volume obtained after dot product is calculated as follows:
[0098] c(x1, x2) = ∑<f1(x1),f2(x2)>
[0099] Subsequently, multi-scale pyramid refinement is performed using the aforementioned features, and the refined disparity result is obtained using deconvolution layers. The initial disparity is used to perform coordinate transformation on the right view, and the transformed image, along with the left and right views, is used as input to a second network. This network regresses the corresponding residuals layer by layer and updates them at different scales. The update process is as follows, where 's' represents the number of scales:
[0100]
[0101] During network inference, in order to balance learning speed and estimation accuracy, the number of feature channels in the convolutional and deconvolutional layers in the network is changed to alter the model size. E-ratio and D-ratio represent the multiples of the original convolutional and deconvolutional layers, respectively. E-ratio is selected from 2, 4, 8, and 16; the corresponding matching D-ratios are 1, 4, 8, and 16. The resulting model size will increase accordingly as E-ratio and D-ratio increase.
[0102]
[0103]
[0104] 3. Depth Estimation: Depth information of relevant pixels is calculated by matching point pairs, and a depth map is generated. Depth estimation requires the depth information of 3D points based on the parallax network regression result d, the equivalent baseline length b, and the camera focal length f. The specific calculation is as follows:
[0105] Z = (b*f) / d
[0106] 4. Point Cloud Reconstruction: Based on the calculated dense depth map, the depth information can be directly converted into a 3D point cloud using the PCL library. For outdoor scenes of different scales, corresponding point cloud filtering and point cloud merging will be performed.
[0107] 5. Triangulation: This involves partitioning a 3D structure into regions using triangular faces as partitioning primitives. Based on the Delaunay partitioning algorithm, the discrete, unordered 3D point cloud is triangularly connected. During the algorithm's iterations, new edges are introduced into the existing point set each time, thus expanding the structure to obtain a 3D surface.
[0108] 6. Texture mapping: Fill texture information using 2D images from different perspectives and optimize texture seams. Steps 5) and 6) mentioned above can be integrated using OpenMVS or CloudCompare software.
[0109] 7. Performance Evaluation: The reconstruction performance is evaluated based on both geometric mapping rationality and semantics. For geometric evaluation, the Chamfer Distance is calculated. The first part of this term ensures the minimum distance between the generated point cloud and the ground truth point cloud, while the second part ensures the coverage of the ground truth point cloud within the generated point cloud. For semantic evaluation, the segmentation accuracy is primarily calculated using ground truth semantic labels provided by a portion of the 3D segmentation dataset.
[0110] Third, at the mission level, a complete cloud-edge collaborative system will involve five stages: ground preparation, sparse point cloud reconstruction, dense point cloud reconstruction, real-world scene mapping, and effect evaluation.
[0111] 1. Ground preparation: Initialize and adjust system equipment parameters, perform self-checks for faults, complete the installation and commissioning of the carrier platform's mission load, and perform static calibration of the dual-aircraft platform.
[0112] 2. Sparse point cloud reconstruction: The dual-machine platform will adjust the attitude through dynamic calibration, use the cloud system's (1)-(5) steps to complete the maximum depth estimation for the reconstructed scene, and obtain sparse point clouds.
[0113] 3. Dense point cloud reconstruction: Based on the sparse point cloud, the dual-machine platform is fine-tuned again to fill in the information of pixels affected by factors such as lighting and occlusion, so as to obtain a dense point cloud.
[0114] 4. Real-world scene mapping: This mainly involves transforming point cloud representations into real 3D objects, which is entirely accomplished by steps 6) and 7) of the cloud system.
[0115] 5. Effect Evaluation: The user terminal, cloud system and carrier platform interact with each other, and the carrier platform's posture is adjusted through low-latency network data link based on the accuracy of cloud system calculation and the user's subjective perception.
[0116] Furthermore, this method uses two UAVs equipped with monocular cameras to achieve binocular vision imaging, reducing surveying costs and expanding the measurement range of the binocular vision system;
[0117] By utilizing the collaborative control of dual drones, simple and wide-range camera pose adjustment can be performed. Based on altitude and parallax requirements, while ensuring high imaging resolution, the single imaging efficiency can be improved, adapting to the scene mapping requirements of different scales indoors and outdoors. At the same time, based on the requirements of detail optimization, micro-scale adjustment of the viewing angle can be used to achieve adaptive scene details. The relatively easy control strategy can reduce the difficulty of multi-drone collaboration, improve the robustness of the system, and enhance its adaptability to outdoor environments.
[0118] Phased reconstruction and algorithm fusion can effectively improve reconstruction results. Reconstructing point clouds from sparse to dense can generate a clear workflow for the system and reduce the impact of sudden external disturbances on the system. The fusion of feature matching and region matching in depth estimation can improve reconstruction accuracy.
[0119] The cloud-edge collaborative system architecture utilizes low-latency networks to achieve fast real-time data transmission between drones and between unmanned devices and the cloud, improving the reconstruction speed. At the same time, it uses cloud computing power to integrate algorithms, enabling distributed sampling to be processed centrally and efficiently, and the pose of the drone system to be corrected in real time through cloud effect evaluation.
[0120] The ground terminal of the drone can provide users with beyond-line-of-sight manual intervention and real-time effect presentation, and is easy to use and applicable to various scenarios.
Claims
1. A method for three-dimensional reconstruction based on cloud-edge collaboration of dual unmanned aerial vehicles, characterized in that, Includes the following steps: S1: Dual-UAV Vision Platform 1) Static calibration: The two UAVs respectively calibrated the intrinsic parameters of their monocular cameras. A 260x260mm square checkerboard calibration board was used to obtain the camera's focal length, optical center, and distortion coefficients. The static calibration adopted the Zhang Zhengyou calibration method, which relies on the registration process of corresponding points between the world coordinate system and the camera coordinate system. ； where M1 is an intrinsic camera matrix, , is a normalized lens focal length, u0, v0 are transverse and radial distortion coefficients; 2) Dynamic calibration: The two UAVs will simultaneously adjust their poses during the sparse point cloud reconstruction stage and the dense point cloud reconstruction stage according to mission requirements; 3) Image acquisition and data transmission: The synchronization delay time between the two machines is ms, and the transmission rate through the 5G chip is 107-700Mbps; S2: Unmanned Aerial Vehicle Cloud System Platform 1) Image preprocessing and correction: Denoising and deblurring are performed on the binocular images, and image enhancement is also performed; 2) End-to-end image feature extraction and image matching: (1) The disparity value of stereo matching is obtained directly through deep neural network regression for binocular two-view images. The network takes the left and right views as input and obtains the predicted disparity and the residual estimated by the update network through inference of two networks respectively. Both networks use an autoencoder structure containing 5 layers of convolution and deconvolution as the backbone. In the main network structure, the input image first goes through the feature extraction of the convolutional layer containing the residual branch, and the similarity of the feature vector is calculated by using the dot product operation, so as to obtain the implicit expression of disparity estimation. The correlation volume obtained by the dot product is calculated as follows: ； (2) Subsequently, the above features are used for multi-scale pyramid refinement. The refined disparity results are obtained using deconvolution layers. The coordinates of the right view are transformed, and the coordinate-transformed image and the left and right views are used as inputs to the second network. This network will regress the corresponding residuals layer by layer and update them to different scales. The update process is as follows, where s represents the number of scales: ； (3) In the process of network inference, in order to balance the learning speed and estimation accuracy, the number of feature channels in the convolutional and deconvolutional layers in the network will be changed to change the size of the model: ； 3) Depth Estimation: Depth information of relevant pixels is calculated by matching point pairs, and a depth map is generated. Depth estimation requires the depth information of 3D points based on the parallax network regression result d, the equivalent baseline length b, and the camera focal length f. The specific calculation is as follows: ； 4) Point cloud reconstruction: Based on the calculated dense depth map, the depth information is directly converted into a 3D point cloud using the PCL library. For outdoor scenes of different scales, corresponding point cloud filtering and point cloud merging will be performed. 5) Triangulation: The three-dimensional structure is divided into regions using triangular faces as the basic units of the division. 6) Texture mapping: Fill in texture information using 2D images from different perspectives and optimize texture seams. Steps 5) and 6) mentioned above will be integrated using OpenMVS or CloudCompare software. 7) Effect evaluation: The reconstruction effect is evaluated based on the rationality of geometric mapping and semantics; S3: Task-side level: 1) Ground preparation: Initialize and adjust system equipment parameters, perform self-checks for faults, complete the installation and commissioning of the carrier platform's mission load, and perform static calibration of the dual-aircraft platform; 2) Sparse point cloud reconstruction: The dual-machine platform will adjust the attitude through dynamic calibration, and use steps 1) to 5) in step S2 to complete the maximum depth estimation for the reconstructed scene and obtain sparse point cloud; 3) Dense point cloud reconstruction: Based on the sparse point cloud, the dual-machine platform is fine-tuned again to fill in the information of pixels affected by lighting and occlusion, so as to obtain a dense point cloud. 4) Real-world scene mapping: The point cloud representation is transformed into a real 3D object, which is entirely accomplished by steps 6) and 7) of the UAV cloud system platform; 5) Effect evaluation: The information of the user terminal, cloud system and carrier platform is exchanged, and the position and attitude of the carrier platform are adjusted through low latency network data link based on the accuracy of cloud system calculation and user subjective perception.
2. The dual-UAV 3D reconstruction method based on cloud-edge collaboration according to claim 1, characterized in that, In the dynamic calibration of S1 2), during the sparse point cloud reconstruction stage, the altitude range of each adjustment by the dual aircraft is -10m to 10m, and the equivalent distance of each adjustment is -0.5m to 0.5m. During the dense point cloud reconstruction stage, the dual aircraft system will adjust the attitude angles, with the pitch angle adjustment range being -20° to 20° and the heading angle adjustment range being -10° to 10°.
3. The method according to claim 2, wherein, In the image preprocessing and correction of S2 1), the key regions of the image are segmented according to the task requirements. The segmentation algorithm is based on the BlendMask network, which will obtain a segmentation mask for the instance, which will be used for the subsequent thinning process.
4. The dual-UAV 3D reconstruction method based on cloud-edge collaboration according to claim 3, characterized in that, In S2(2)(1), the first network structure is used to estimate the final predicted disparity, and the second network is used to update the residual estimated by the network. Both networks use an autoencoder structure containing 5 layers of convolution and deconvolution as the backbone.
5. The dual-UAV 3D reconstruction method based on cloud-edge collaboration according to claim 4, characterized in that, In steps 2)(3) of S2, E-ratio and D-ratio represent the multiples relative to the original convolutional layer channels and the deconvolutional layer, respectively. The selected E-ratios are 2, 4, 8, and 16; the corresponding matching D-ratios are 1, 4, 8, and 16. The size of the obtained model will increase accordingly as E-ratio and D-ratio increase.
6. The method according to claim 5, wherein, In the triangulation of S2 (5), according to the Delaunay triangulation algorithm, the discrete and disordered three-dimensional point cloud is triangularly connected. During the iteration of the algorithm, new edges are introduced into the original point set each time, thereby expanding to obtain a three-dimensional surface.
7. The method according to claim 6, wherein, In the effect evaluation of S2(7), in terms of the reasonableness evaluation of geometric mapping, the chamfer distance will be calculated. The first part of the chamfer distance ensures that the distance between the generated point cloud and the ground truth point cloud is minimized, and the second part ensures the coverage of the ground truth point cloud in the generated point cloud. In terms of semantic evaluation, the ground truth semantic labels provided by the partial 3D segmentation dataset are used to calculate the segmentation accuracy.