An unmanned aerial vehicle cluster relative positioning and real-time dense reconstruction algorithm

Abstract

The application discloses a kind of unmanned aerial vehicle cluster relative positioning and real-time dense reconstruction algorithm, it is related to unmanned aerial vehicle remote sensing surveying and mapping and spatial information processing technical field, including the following steps: image sequence collected to unmanned aerial vehicle is carried out visual odometer processing, feature is extracted and camera pose is estimated, generates key frame and constructs local map including camera pose and sparse map point.The application proposes multi-unmanned aerial vehicle cooperative real-time three-dimensional reconstruction method, through cluster cooperation and low-altitude synchronous acquisition, realize wide-range scene fast modeling, improve reconstruction efficiency and real-time performance.Multiple scene verification, while maintaining high accuracy and stability in shortening processing time.Combining visual odometer, diffusion depth estimation and two-stage fusion and global optimization, effectively solve the scale inconsistency and trajectory drift problem, realize high-quality dense reconstruction.

Description

Technical Field

[0001] This invention relates to the field of UAV remote sensing mapping and spatial information processing technology, specifically to an algorithm for relative positioning and real-time dense reconstruction of UAV swarms. Background Technology

[0002] With the widespread application of UAV remote sensing technology in emergency rescue, land surveys, and regional inspections, the demand for large-scale, high-time-efficiency 3D spatial information acquisition is becoming increasingly urgent. Traditional single-UAV aerial surveying methods are limited by endurance and operational efficiency, often requiring multiple flights for data stitching in large survey areas. This not only increases operational time but also makes data consistency susceptible to factors such as changes in lighting and positioning errors. Furthermore, existing offline 3D reconstruction methods based on Structure of Motion (SfM) have complex processing procedures and long computation times, making it difficult to meet the requirements for real-time data acquisition in disaster emergency scenarios. In recent years, real-time mapping methods combining Visual Simultaneous Localization and Mapping (SLAM) technology have gradually developed, which can improve UAV surveying efficiency to some extent. However, existing research mainly focuses on single-UAV 2D map generation, which is insufficient to meet the application needs of large-scale 3D reconstruction.

[0003] Existing UAV collaborative mapping technologies primarily rely on multi-sensor fusion schemes (such as LiDAR, depth cameras, or ultra-wideband positioning) to achieve high-precision relative positioning. However, limitations in payload weight, power consumption, and cost make them difficult to apply to consumer-grade UAV swarms in large-scale remote sensing scenarios. Furthermore, relying solely on monocular cameras and GPS, multi-UAV collaborative mapping faces challenges such as scale inconsistencies, missing depth information, and difficulties in cross-platform data matching, leading to inconsistent 3D reconstruction accuracy and stability. In addition, the massive amount of image data and high texture similarity in large-scale scenarios further increase data transmission pressure and feature matching difficulty, making it difficult for existing methods to achieve efficient, real-time, dense 3D reconstruction. Therefore, how to achieve efficient collaborative positioning and high-quality real-time 3D reconstruction among multiple UAVs under limited sensing conditions remains a critical technical problem that urgently needs to be solved.

[0004] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide an algorithm for relative positioning and real-time dense reconstruction of unmanned aerial vehicle (UAV) swarms to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an algorithm for relative positioning and real-time dense reconstruction of UAV swarms, comprising the following steps: Visual odometry is performed on the image sequence acquired by the UAV to extract features and estimate the camera pose, generate keyframes and construct a local map containing the camera pose and sparse map points. Based on local maps and keyframes, cross-UAV co-view detection is performed. When satellite positioning information is available, baseline distance is calculated based on flight altitude and camera parameters to filter co-view frames. When satellite positioning information is unavailable, co-view relationships are determined by image feature retrieval combined with time constraints and spatial verification to obtain a set of matching keyframes. Multi-view depth estimation is performed based on the set of matching keyframes. The search range is determined by the depth distribution of sparse map points. The depth map is obtained by combining camera pose and image data. Valid points are selected based on confidence and reprojection consistency to generate local dense point clouds. Cross-UAV alignment is achieved by using local maps and local dense point clouds. First, an initial transformation including scale is solved by sparse point matching. Then, the dense point cloud is registered based on the initial transformation to achieve sub-map fusion. Based on the fusion results, global pose constraints are constructed, and the pose relationships within a single UAV and the alignment relationships across UAVs are jointly optimized to obtain pose results in a unified coordinate system and output global dense 3D reconstruction results.

[0007] Preferably, focusing on improving the stability of pose estimation and the ability to represent features during image sequence processing, the key processing steps of visual odometry are refined to make local map construction more reliable. The steps are as follows: A multi-scale image pyramid is constructed for the input image sequence. ORB feature points are extracted at each scale layer, and a corresponding binary descriptor is generated for each feature point. Feature descriptor matching is performed between consecutive frames, matching point pairs are filtered by distance metric, and camera pose transformation is calculated based on the matching relationship; Based on the pose change and feature point disparity relationship between adjacent frames, determine whether the current frame meets the keyframe insertion conditions, and mark the frames that meet the conditions. Triangulation calculations are performed on the matching feature points between keyframes to recover the coordinates of the three-dimensional points in space, and the generated three-dimensional points are associated with the camera pose and stored to form a local map structure.

[0008] Preferably, for the problem of determining spatial relationships between images from different UAVs, a geometric constraint mechanism based on flight state and imaging parameters is introduced to improve the accuracy of common-view frame selection. The steps are as follows: Collect flight altitude information and camera intrinsic parameters for each UAV at the corresponding time, including sensor size and focal length; Calculate the spatial baseline distance constraint range between the drones based on the acquired altitude information and camera parameters; Image frames acquired by different drones are combined in chronological order, and candidate frames that meet the spatial overlap condition are selected based on baseline distance constraints. The selected candidate frames are further processed for common-view determination to establish a set of keyframe matching relationships across UAVs.

[0009] Preferably, for cross-platform image matching needs in environments lacking satellite positioning information, stable co-view recognition is achieved through visual feature retrieval and multi-constraint fusion, with the following steps: Global feature vectors and local feature descriptors are extracted from keyframe images, and a feature index database is established. Based on the similarity measure between global features, a retrieval operation is performed on the input image to obtain several candidate image frames; Candidate frames are filtered by combining image timestamp information, and image pairs with a time interval lower than a preset threshold are excluded; Local feature matching is performed on the remaining candidate frames, and the results are verified in conjunction with spatial geometric consistency constraints to determine the cross-UAV co-view relationship.

[0010] Preferably, consistency enhancement constraints are applied to the matching results of cross-UAV co-view relationships, a feature matching relationship set is constructed, the spatial distribution consistency between matching point pairs is calculated, the matching relationships are filtered based on reprojection errors, and the co-view relationship set is updated according to geometric constraints to obtain stable matching results. Preferably, to address the problem of insufficient scale adaptability in multi-view depth estimation, the depth search range and estimation process are optimized by introducing depth distribution constraints, as follows: Statistical analysis is performed on sparse 3D points in the local map to obtain their depth distribution information in the camera coordinate system; The upper and lower bounds of the depth search range are determined based on the statistical results of the depth distribution, which are used to constrain the subsequent depth estimation process; The image data, camera pose parameters, and depth search range are input into the depth estimation model to calculate the depth value of the corresponding pixel. The output depth map is filtered using confidence information, and depth results corresponding to low-confidence regions are removed.

[0011] Preferably, regarding error suppression and consistency assurance during the construction of dense point clouds, effective spatial points are selected through multi-view geometric constraints to improve the reliability of point cloud data. The steps are as follows: Based on the pixel coordinates and corresponding depth values, the two-dimensional pixel points are back-projected into three-dimensional space using camera intrinsic parameters to obtain the spatial point coordinates; The obtained spatial points are mapped to adjacent views according to the camera pose relationship, and the corresponding reprojected pixel positions are calculated. Compare the differences between the original pixel positions and the reprojected positions, and calculate the corresponding depth deviation; Spatial points that meet the preset pixel error threshold and depth error threshold are selected and constructed into a local dense point cloud.

[0012] Preferably, to address the scale differences and alignment errors between local maps from multiple UAVs, a phased registration strategy is employed to achieve unified coordinate representation, with the following steps: Feature matching is performed on sparse map points generated by different drones, and valid matching point pairs are selected through consistency constraints. Geometric constraints are constructed based on matching point pairs, and the initial alignment transformation is estimated using the random sampling consensus method. Using the initial alignment transformation as input, perform iterative registration calculations on the corresponding dense point cloud and update the transformation parameters; The updated transformation is applied to the point cloud data to fuse the point clouds from multiple UAVs and unify them into the same coordinate system.

[0013] Preferably, to address the initial alignment transformation accuracy and the stability of dense point cloud registration, further constraints are imposed on the transformation solution and point cloud registration process. By filtering errors in the sparse point matching results and constructing weighted constraint relationships, and combining the convergence judgment conditions in the transformation parameter optimization process, the correspondence in the dense point cloud registration process is constrained and updated, thus completing a unified expression of coordinate consistency in the point cloud fusion process.

[0014] Preferably, to address the issue of accumulated errors and scale inconsistencies during multi-UAV collaborative mapping, global pose unification is achieved by constructing multi-level constraint relationships, as follows: Establish pose constraints between consecutive keyframes within a single UAV and construct loop closure detection constraints; The pose variables inside a single UAV are optimized and calculated to update the pose of keyframes. Integrate the alignment relationships between different drones and incorporate cross-drone constraints into the overall optimization structure; The pose variables of all UAVs are jointly optimized to obtain pose results in a unified coordinate system.

[0015] The technical effects and advantages provided by the present invention in the above technical solution are as follows: This invention establishes a cluster collaboration mechanism, enabling multiple drones to synchronously acquire data under low-altitude conditions and achieve rapid 3D modeling of large-scale scenes through collaborative processing. Compared to the traditional single-drone multi-operation method, this method effectively shortens the data acquisition and processing cycle, significantly improves remote sensing efficiency, and meets the application needs of scenarios with high timeliness requirements, such as emergency mapping, thereby improving the real-time performance and processing efficiency of large-scale 3D reconstruction as a whole.

[0016] This invention validated the effectiveness and stability of its method through multi-region, multi-UAV collaborative experiments. Under different data scales, the 3D reconstruction time was consistently kept within a short range, significantly reducing processing time compared to traditional methods. Simultaneously, in terms of accuracy, the reconstruction error remained within a small range, demonstrating good geometric consistency and stability. Experimental results show that this method improves processing efficiency while maintaining high reconstruction accuracy, making it suitable for large-scale 3D reconstruction tasks in complex environments and possessing significant engineering application value.

[0017] This invention constructs a collaborative computing architecture that enables UAV swarms to achieve real-time pose estimation using visual odometry, and combines a block processing strategy with a conditional diffusion model for efficient depth estimation, effectively reducing the computational overhead of multi-viewpoint mapping. Simultaneously, by introducing a two-stage fusion method combining sparse features and dense point clouds, and a cross-UAV global pose map optimization strategy, it solves the problems of scale inconsistency and trajectory drift in multi-UAV mapping. This method overcomes the limitations of single UAVs in terms of coverage and processing power, achieving high-quality real-time dense 3D reconstruction in large-scale environments, and significantly improving the overall performance of UAV remote sensing reconstruction. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0019] Figure 1 This is a flowchart of the relative positioning and real-time dense reconstruction method for UAV swarms according to the present invention.

[0020] Figure 2 This is a schematic diagram of the multi-view depth estimation and dense reconstruction based on image segmentation according to the present invention.

[0021] Figure 3 This is a schematic diagram illustrating the relative positioning and coordinate alignment of the multi-UAV sub-maps of the present invention. Detailed Implementation

[0022] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the description of this disclosure will be more complete and fully convey the concept of the exemplary embodiments to those skilled in the art.

[0023] This invention provides, for example Figures 1-3The algorithm for relative positioning and real-time dense reconstruction of a drone swarm shown below has the following specific steps: This technical solution adopts a centralized collaborative processing architecture, defining the set of drones participating in the collaborative operation as: in, Indicates the first A drone, This indicates the number of drones. All drones perform autonomous flight missions according to pre-planned flight paths, continuously collecting ground imagery data during flight. Each drone acquires continuous image sequences through onboard sensors, and simultaneously obtains corresponding pose information in conjunction with the flight control system. The image stream and pose data are transmitted in real-time to the ground-based central processing unit via a high-bandwidth communication link. In this collaborative architecture, each drone independently performs visual odometry front-end processing for real-time perception and modeling of the local environment. Specifically, for each drone... The sequence of images it acquires is denoted as The visual odometry algorithm continuously processes the sequence, achieving real-time tracking of the UAV's motion state through feature extraction, feature matching, and pose estimation. In the feature extraction stage, a multi-scale ORB (Oriented Fast and Rotated Brief) feature detection method is used to construct a pyramid structure for image processing, where the number of pyramid layers is set as follows: In each frame, a maximum of 3000 feature points are extracted to balance feature representation capability and real-time processing efficiency. This multi-scale feature extraction strategy allows for the acquisition of stable and rotation-invariant key features at different resolution levels, thereby enhancing the system's robustness in complex environments. Subsequently, in the feature matching stage, the current frame image is matched with the previous frame image, and feature correspondences are established through descriptor similarity calculation. Based on this, camera pose estimation is performed using the matching results to obtain the camera's pose representation in the world coordinate system. in, This represents a rigid body transformation from the camera coordinate system to the world coordinate system, including rotation and translation information, and belongs to a special Euclidean group. This pose is used to describe the UAV's motion in space and is the foundation for subsequent map building and multi-UAV collaboration.

[0024] During pose estimation, when the keyframe insertion strategy is satisfied, the current frame is set as the keyframe, denoted as: in, Indicates the drone's serial number. This indicates the drone's first Keyframes are introduced to reduce computational redundancy while ensuring the stability and integrity of map construction. After keyframe generation, triangulation is performed on successfully matched feature points to recover the 3D coordinates of spatial points through multi-view geometric relationships, thereby constructing sparse map points. Through this process, 2D image features can be enhanced into 3D spatial structure information. Thus, each UAV generates its corresponding local map, represented as: in: Indicates the first A set of camera poses for a drone; This represents the corresponding set of sparse map points. In this local map, each sparse map point contains its three-dimensional spatial coordinates and a corresponding binary feature descriptor. The three-dimensional coordinates describe the point's position in space, while the binary descriptor is used for subsequent feature matching across frames and between drones. Through this processing flow, each drone can independently complete the entire process from image acquisition, feature extraction, pose estimation to sparse map construction, achieving real-time modeling of the local environment. This distributed front-end processing method effectively reduces the computational burden on the central processing unit while ensuring the real-time nature of data acquisition. Furthermore, because each drone observes the environment at different times and from different perspectives, the generated local maps often differ in coordinate systems. A unified pose representation is used to address this. and structured local map representation This provides a standardized data foundation for subsequent consensus detection, map fusion, and global optimization across drones.

[0025] In terms of data transmission, a high-bandwidth communication network is used to ensure that image streams and pose data can be transmitted to the central processing unit with low latency, thereby supporting depth estimation and global mapping in subsequent steps. Through this collaborative mechanism of "front-end distributed processing + back-end centralized optimization," the overall mapping accuracy can be improved while ensuring real-time performance.

[0026] In summary, this step enables efficient environmental perception and sparse 3D structure reconstruction by the UAV. Through standardized pose representation and map organization, it provides a unified data interface and basic support for collaborative mapping among multiple UAVs, and lays a key foundation for subsequent cross-UAV data association and fusion processing.

[0027] The cross-UAV adaptive common-view detection and relocalization process is crucial for determining whether images acquired by different UAVs originate from the same spatial region. Due to differences in perspective, time, and positioning errors during UAV flight, relying solely on single pieces of information is insufficient to accurately establish correspondences between cross-platform images. Therefore, an environmentally adaptable common-view detection mechanism is required. This mechanism divides the processing path into two modes based on the availability of external positioning information: GNSS available mode and GNSS denied mode, ensuring stable common-view recognition capabilities across various application environments.

[0028] Under GNSS availability, the UAV's flight altitude and spatial position information are obtained using its own satellite positioning information, and a dynamic distance determination model is constructed through geometric constraints. The determination of common-view relationships is based on the degree of overlap of image projections on the ground. To ensure the reliability of subsequent image stitching and 3D reconstruction, the linear overlap rate of the orthographic projection must be maintained at no less than a preset threshold. This threshold is typically set as follows: To meet this overlap requirement, the spatial baseline distance between drones needs to satisfy the following constraints: in: Indicates drone With drones Spatial baseline distance between them; and These represent the flight altitudes of the two drones, respectively. Indicates the physical width of the camera sensor; Indicates the camera's focal length; This represents the image overlap rate threshold.

[0029] This inequality constrains the maximum spacing between UAVs from an imaging geometry perspective, ensuring sufficient overlap between images given altitude and camera parameters. When this condition is met, the corresponding frame is identified as a co-view candidate frame. This strategy is dynamically adaptive, automatically adjusting the decision threshold as flight altitude changes, thus making it suitable for scenarios of different scales.

[0030] In practical applications, since the flight altitude of drones may vary, this model introduces... This approach achieves unified constraints on different altitude combinations, enabling stable common-view determination capabilities even in low-altitude collaborative scenarios. Simultaneously, it incorporates camera physical parameters... and This enables the model to generalize across devices, allowing it to be adapted to imaging devices carried by different types of drones.

[0031] In GNSS-denied environments, since external positioning information cannot be relied upon, it is necessary to establish correspondences between UAVs using visual information. In this case, a coarse-to-fine hierarchical matching strategy based on visual position recognition is adopted to improve matching efficiency and reduce the probability of false matches.

[0032] First, in the feature representation stage, a visual model is used to extract global and local features from the image. Global features are used to describe the overall semantic information of the image, enabling rapid filtering of potential matching candidates; local features are used for fine-grained matching, ensuring geometric consistency. During processing, keyframe features are continuously added to the database incrementally, thus forming a retrieval-ready feature set.

[0033] To avoid mismatches between consecutive frames from the same drone, a time constraint mechanism is introduced during the candidate frame selection stage to limit the image acquisition time. Specifically, the time difference must satisfy: Frame pairs are excluded to avoid misclassifying adjacent frames on the same flight path as cross-drone shared-view frames. This strategy effectively reduces matching interference caused by temporal correlation.

[0034] During the candidate retrieval phase, cosine similarity between global descriptors is used for fast retrieval, selecting the Top-K candidate frames with the highest similarity from the database. This process can efficiently filter potential matching objects in large-scale data, significantly reducing the subsequent computational burden.

[0035] Subsequently, in the fine matching stage, local feature matching is performed on candidate frames. Preliminary correspondences are established through descriptor matching and verified in conjunction with spatial geometric constraints. A re-ranking mechanism prioritizes matching results that satisfy geometric consistency, thereby improving matching accuracy.

[0036] During spatial verification, consistency checks are performed on matching points by combining multi-view geometric relationships. By eliminating abnormal matches that do not meet the constraints, the reliability of the final matching results is ensured. This process achieves a step-by-step screening from semantic similarity to geometric consistency, effectively improving the accuracy of cross-UAV relocalization.

[0037] Through the collaborative design of the above-mentioned GNSS availability and GNSS denial modes, stable determination of co-view relationships under different environmental conditions is achieved. On the one hand, geometric constraints are used to achieve rapid screening and improve processing efficiency; on the other hand, visual features are combined to achieve accurate matching and enhance system robustness.

[0038] Overall, this adaptive co-view detection mechanism not only effectively solves the matching difficulties caused by differences in viewpoints and time between multiple UAVs, but also maintains high matching accuracy and stability in complex environments, providing a reliable data association foundation for subsequent multi-view depth estimation and map fusion. Furthermore, this mechanism balances computational efficiency and matching accuracy, demonstrating good scalability and practical value in large-scale UAV swarm applications.

[0039] In multi-UAV collaborative mapping, the recovery of dense 3D structures relies on high-quality depth estimation results. To fully utilize the sparse map information generated by front-end visual odometry and improve the adaptability of multi-view depth estimation in large-scale complex scenes, the visual odometry output data is divided into multiple image patches with spatial overlap, and a conditional diffusion probability model is introduced for depth estimation. This block-based processing strategy improves overall computational efficiency while maintaining local consistency and enhances the model's adaptability to structural differences in different regions.

[0040] At the data organization level, continuous image sequences are divided into several image patches based on spatial proximity. Each image patch contains several image frames with overlapping viewpoints. A certain proportion of overlapping areas is allowed between different image patches to ensure the continuity and consistency of depth information in the subsequent fusion stage. Each image patch corresponds to a set of camera pose information and sparse map points, which together constitute the prior input for depth estimation.

[0041] In the depth estimation process, adaptive initialization of the depth search range is performed first. For each image patch, sparse point cloud data is extracted from its corresponding local map, and these 3D points are transformed to the corresponding camera coordinate system to analyze their depth distribution along the Z-axis. The 1st and 99th percentiles of this depth distribution are statistically determined and used as the minimum and maximum initial search ranges for the depth estimation process, respectively. This range constraint method based on data distribution effectively avoids the lack of adaptability caused by a fixed depth range, enabling the depth estimation process to dynamically adjust according to scale changes in different scenes, thereby improving estimation accuracy and stability.

[0042] After initializing the depth range, the multi-view depth estimation stage begins. For each image patch, its contained image data, corresponding camera pose parameters, camera intrinsic information, and a pre-determined depth search range are used as input and fed into the conditional diffusion probability model for processing. This model jointly models multi-view information and outputs the depth value and corresponding depth confidence score for each image pixel. The depth confidence score reflects the reliability of the model's depth estimation result, providing a basis for subsequent error filtering and consistency verification.

[0043] After obtaining the initial depth estimation results, further filtering and optimization are needed using geometric and photometric constraints. For any pixel in the reference keyframe... Combined with its corresponding estimated depth The camera projection model is then back-projected into 3D space to obtain the corresponding spatial points. This back-projection process is accomplished using the following expression: in: This represents the coordinates of a 3D point obtained through back projection in the world coordinate system; Indicates camera pose; Represents the rotation matrix; Represents the translation vector; Represents the camera intrinsic parameter matrix; Represents the pixel coordinates of the image; This represents the depth value corresponding to that pixel.

[0044] This back-projection process can restore pixels in a two-dimensional image to points in three-dimensional space, providing a basis for subsequent consistency verification.

[0045] After obtaining the 3D point, the point is reprojected into adjacent views. By comparing its projection position and depth information in different views, multi-view processing is achieved. Figure 1 Consistency verification. In this process, pixel reprojection error and relative depth error are calculated separately, and the results are filtered according to preset thresholds. Specific judgment criteria are as follows: in: This represents the reprojected pixel coordinates of a 3D point in an adjacent view; This indicates the depth value of the point in the adjacent view; Indicates the pixel error threshold; This represents the depth error threshold.

[0046] The aforementioned dual constraint mechanism effectively eliminates outliers caused by matching errors or inaccurate estimations. Pixel error constraints ensure the consistency of spatial projection, while depth error constraints ensure the consistency of geometric structure. Only points that simultaneously satisfy both conditions are considered valid 3D points.

[0047] After the consistency screening is completed, all verified 3D points are aggregated to form a local dense point cloud for the corresponding image patch. Compared with the sparse point cloud generated by the front end, this point cloud has a higher point density and richer geometric details, and can more accurately describe the scene structure.

[0048] The above process completes the entire process from sparse map prior guidance to multi-view depth estimation, and then to consistency filtering to generate dense point clouds. This method fully utilizes existing sparse structure information, effectively constrains the depth estimation range, and combines multi-view geometric relationships for error filtering, thus obtaining high-quality dense 3D reconstruction results even in complex environments.

[0049] Overall, this process significantly improves the accuracy and robustness of depth estimation while ensuring computational efficiency, providing high-quality geometric foundation data for subsequent cross-UAV map fusion and global optimization.

[0050] In multi-UAV collaborative mapping, because each UAV independently performs visual odometry, the resulting local maps are often in different coordinate systems and exhibit scale inconsistencies. Therefore, a cross-UAV map fusion process is needed to align the local maps from different UAVs to a unified coordinate system, thereby achieving a consistent representation of the overall spatial structure. This process is based on multi-frame co-view relationships and uses a two-stage alignment strategy to progressively improve registration accuracy, balancing computational efficiency and alignment accuracy.

[0051] During the fusion process, based on the common-view detection results from the previous stage, a set of continuous common-view keyframes and corresponding local maps between the query drone and the matching drone are first obtained. By analyzing the observation relationships between these keyframes, the overlapping area between the two drones in space can be determined. Based on this common-view information, a similarity transformation relationship is constructed: This transformation includes rotation. Translation and scale This similarity transformation is used to describe the mapping relationship between different UAV coordinate systems. Since monocular visual odometry cannot recover absolute scale, this similarity transformation plays a crucial role in the fusion process.

[0052] The entire map fusion process employs a two-stage strategy: the first stage performs initial alignment of the sparse point cloud, and the second stage performs fine-grained registration of the dense point cloud based on this alignment. This two-stage strategy effectively reduces computational complexity while improving the final registration accuracy.

[0053] In the first stage, matching is performed on the sparse map point sets of the two UAVs. First, the Hamming distance of the feature descriptors is used to match sparse points, and then a ratio test and bidirectional consistency constraints are combined to filter reliable matching pairs, thus obtaining an initial set of matched points. This set, after rigorous screening, effectively reduces the interference of mismatches on subsequent estimations.

[0054] After obtaining the set of matching points, a Random Sample Consensus (RANSAC) framework is introduced to enhance robustness to outlier matches. Within this framework, candidate transformations are estimated through multiple random samplings, and the model with the most interior points is selected as the optimal estimate. Subsequently, based on the optimal interior point set, the transformation parameters are further optimized using a weighted least squares method.

[0055] In the optimization process, the Umeayama algorithm is used to solve for the similarity transformation parameters, and its objective function is expressed as: in: This represents the set of filtered matching pairs; and These represent the three-dimensional feature points in the local maps of the two UAVs, respectively. , , Let represent the rotation, translation, and scaling parameters, respectively. The objective function achieves optimal alignment estimation for the two sets of point clouds by minimizing the squared Euclidean distance between corresponding points. This step yields the initial transformation matrix: This matrix has corrected for scale differences between different UAVs, providing reliable initial values ​​for subsequent fine registration.

[0056] In the second stage, dense point cloud information is further utilized for fine-grained alignment. Compared to sparse point clouds, dense point clouds contain richer geometric structure information, which can significantly improve registration accuracy. In this stage, the locally dense point cloud generated by the depth estimation in the previous stage is selected, denoted as... and These correspond to querying drones and matching drones, respectively.

[0057] With initial transformation As a starting point, the voxelized generalized iterative nearest point algorithm (FastVGICP) is used to perform point cloud registration. This method improves registration stability under complex geometries by modeling the point cloud as a local Gaussian distribution and utilizing the Mahalanobis distance metric to measure the matching error between points.

[0058] During the optimization process, the transformation parameters are iteratively updated to gradually converge the correspondences between point clouds, thereby obtaining more accurate alignment results. This stage ultimately outputs the globally optimal alignment transformation: This transformation has higher accuracy than the initial transformation and can more accurately reflect the spatial relationship between the two UAVs.

[0059] After obtaining the fine alignment transformation, it is applied to the dense point cloud of the query drone. and the point cloud of the matching drone The points are then fused. During the fusion process, voxel downsampling is performed on the point cloud to reduce redundant points and improve data processing efficiency. The final result is a fused point cloud in a unified coordinate system. The fused point cloud exhibits good continuity and consistency in its spatial structure, and can realistically reflect the three-dimensional geometric information of the scene.

[0060] Overall, the two-stage map fusion strategy effectively addresses the issues of scale inconsistency and alignment accuracy in multi-UAV collaborative mapping through a combination of sparse alignment and dense optimization. The first stage utilizes sparse features to achieve rapid coarse registration, reducing computational complexity; the second stage uses dense information for fine optimization, improving overall accuracy. This combination not only enhances the fusion effect but also ensures the algorithm's scalability in large-scale scenarios.

[0061] Furthermore, this method is designed to support the gradual fusion of multiple UAVs. By continuously introducing new local maps and executing the aforementioned two-stage alignment process, a complete large-scale 3D scene model can be gradually constructed. With the support of a multi-threaded execution mechanism, the fusion process can be performed in parallel with the data acquisition process, thereby meeting real-time requirements.

[0062] Through the above processing flow, high-precision fusion of local maps from different UAVs was achieved, providing a solid foundation for subsequent global pose optimization and unified 3D modeling.

[0063] In multi-UAV collaborative mapping, since each UAV independently performs visual odometry and local map construction, cumulative errors inevitably exist in their pose estimation. Furthermore, scale inconsistencies and relative pose deviations exist between different UAVs. To achieve a globally consistent representation in a unified coordinate system, the pose relationships of all UAVs need to be optimized holistically. This process employs a multi-level constraint strategy, gradually eliminating local drift and cross-UAV alignment errors through phased processing, thereby obtaining globally consistent pose estimation results.

[0064] The overall optimization process is divided into two levels: intra-UAV optimization and inter-UAV global optimization. The former is mainly used to eliminate the cumulative error within a single UAV, while the latter is used to unify the coordinate relationships and scale information between different UAVs.

[0065] During the internal optimization phase of the UAV, visual loop closure information is used to constrain local trajectories. When the same UAV is detected repeatedly observing the same area at different times, loop closure constraints can be established to correct drift. In this process, a graph structure is constructed, consisting of odometry constraints between consecutive keyframes and constraints generated by loop closure detection. The constraints between consecutive frames reflect short-term motion relationships, while the loop closure constraints provide long-term consistency information.

[0066] The optimization objective is achieved by minimizing the residuals corresponding to these constraints, which can be expressed as: in: This represents the set of odometry edges consisting of consecutive keyframes; This represents the constraint set obtained from loop closure detection; This indicates the relative pose error provided by the visual odometry. This represents the pose error corresponding to the loop closure constraint. This represents the information matrix, used to describe measurement uncertainty; This represents the robust kernel function, used to reduce the impact of outlier constraints on the optimization results. Regarding the error definition, the relative pose error is represented in Lie algebra form, obtained by performing a logarithmic mapping on the pose transformation, and is expressed as: This expression maps the pose error to the Lie algebra space, facilitating linearization during optimization. Through this optimization process, the accumulated error in the UAV trajectory can be significantly reduced, improving the consistency of the local map.

[0067] After completing the internal optimization of the UAVs, the next stage is global optimization among the UAVs. The core objective of this stage is to resolve the scale drift problem caused by the monocular vision system between different UAVs and to further unify the pose representation of each UAV. Since monocular vision cannot directly obtain the absolute scale, the trajectories of different UAVs usually have scale deviations, so joint optimization needs to be performed in a similarity transformation space.

[0068] In this stage, the pose representations of all UAVs are extended to the Sim(3) space, thereby simultaneously estimating rotation, translation, and scale parameters during the optimization process. The optimization process relies on cooperative loop constraints across UAVs, which are derived from the alignment relationships obtained from the map fusion in the previous stage.

[0069] The optimization objective is achieved by jointly minimizing the internal constraints and cross-UAV constraints of the UAV, and its expression is as follows: in: This represents the set of drones participating in the optimization; Indicates the first The set of odometry constraints inside the drone; Represents the set of cooperative lapsing constraints across drones; This represents the pose residual between drones, which is derived from the relative transformation calculated during the map fusion stage. This represents the corresponding measurement information matrix; This represents a robust kernel function.

[0070] By introducing cross-UAV constraints, the trajectories of different UAVs can be connected into a unified topology, thereby achieving globally consistent optimization. This optimization process not only eliminates scale biases between different UAVs but also further corrects errors that were not completely eliminated in the local optimization stage.

[0071] After optimization, the poses of all UAVs are uniformly mapped to the same global coordinate system, and the corresponding dense point clouds are transformed and fused. To reduce data redundancy and improve processing efficiency, voxel downsampling is performed on the fused point clouds to obtain the final globally dense 3D map. This map has good continuity and consistency in spatial structure and can accurately reflect the geometric features of the real scene.

[0072] From an overall process perspective, the multi-level constraint optimization strategy effectively solves key problems in multi-UAV collaborative mapping through a combination of local and global optimization. The internal optimization phase focuses on improving the trajectory accuracy of individual UAVs, while the cross-UAV optimization phase focuses on global consistency and scale uniformity. The two work together to form a complete optimization system.

[0073] Furthermore, this optimization framework exhibits excellent scalability, enabling collaborative mapping tasks involving any number of drones. As more drone data is added, the scale of the optimization problem increases, but due to the graph optimization structure, computation can be efficiently completed using sparse matrix solving methods, thus ensuring the overall system's availability in large-scale scenarios.

[0074] The above processing steps can significantly improve the accuracy and stability of multi-UAV collaborative reconstruction results, maintaining good geometric consistency in complex environments and large-scale scenes, and providing a reliable data foundation for subsequent 3D model applications.

[0075] This invention establishes a cluster collaboration mechanism, enabling multiple drones to synchronously acquire data under low-altitude conditions and achieve rapid 3D modeling of large-scale scenes through collaborative processing. Compared to the traditional single-drone multi-operation method, this method effectively shortens the data acquisition and processing cycle, significantly improves remote sensing efficiency, and meets the application needs of scenarios with high timeliness requirements, such as emergency mapping, thereby improving the real-time performance and processing efficiency of large-scale 3D reconstruction as a whole.

[0076] This invention validated the effectiveness and stability of its method through multi-region, multi-UAV collaborative experiments. Under different data scales, the 3D reconstruction time was consistently kept within a short range, significantly reducing processing time compared to traditional methods. Simultaneously, in terms of accuracy, the reconstruction error remained within a small range, demonstrating good geometric consistency and stability. Experimental results show that this method improves processing efficiency while maintaining high reconstruction accuracy, making it suitable for large-scale 3D reconstruction tasks in complex environments and possessing significant engineering application value.

[0077] This invention constructs a collaborative computing architecture that enables UAV swarms to achieve real-time pose estimation using visual odometry, and combines a block processing strategy with a conditional diffusion model for efficient depth estimation, effectively reducing the computational overhead of multi-viewpoint mapping. Simultaneously, by introducing a two-stage fusion method combining sparse features and dense point clouds, and a cross-UAV global pose map optimization strategy, it solves the problems of scale inconsistency and trajectory drift in multi-UAV mapping. This method overcomes the limitations of single UAVs in terms of coverage and processing power, achieving high-quality real-time dense 3D reconstruction in large-scale environments, and significantly improving the overall performance of UAV remote sensing reconstruction.

[0078] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. An algorithm for relative positioning and real-time dense reconstruction of UAV swarms, characterized in that, Includes the following steps: Visual odometry is performed on the image sequence acquired by the UAV to extract features and estimate the camera pose, generate keyframes and construct a local map containing the camera pose and sparse map points. Based on local maps and keyframes, cross-UAV co-view detection is performed. When satellite positioning information is available, baseline distance is calculated based on flight altitude and camera parameters to filter co-view frames. When satellite positioning information is unavailable, co-view relationships are determined by image feature retrieval combined with time constraints and spatial verification to obtain a set of matching keyframes. Multi-view depth estimation is performed based on the set of matching keyframes. The search range is determined by the depth distribution of sparse map points. The depth map is obtained by combining camera pose and image data. Valid points are selected based on confidence and reprojection consistency to generate local dense point clouds. Cross-UAV alignment is achieved by using local maps and local dense point clouds. First, an initial transformation including scale is solved by sparse point matching. Then, the dense point cloud is registered based on the initial transformation to achieve sub-map fusion. Based on the fusion results, global pose constraints are constructed, and the pose relationships within a single UAV and the alignment relationships across UAVs are jointly optimized to obtain pose results in a unified coordinate system and output global dense 3D reconstruction results.

2. The algorithm for relative positioning and real-time dense reconstruction of UAV swarms according to claim 1, characterized in that, To improve the stability of pose estimation and the ability to represent features during image sequence processing, the key processing flow of visual odometry is designed in detail, with the following steps: A multi-scale image pyramid is constructed for the input image sequence. ORB feature points are extracted at each scale layer, and a corresponding binary descriptor is generated for each feature point. Feature descriptor matching is performed between consecutive frames, matching point pairs are filtered by distance metric, and camera pose transformation is calculated based on the matching relationship; Based on the pose change and feature point disparity relationship between adjacent frames, determine whether the current frame meets the keyframe insertion conditions, and mark the frames that meet the conditions. Triangulation calculations are performed on the matching feature points between keyframes to recover the coordinates of the three-dimensional points in space, and the generated three-dimensional points are associated with the camera pose and stored to form a local map structure.

3. The algorithm for relative positioning and real-time dense reconstruction of UAV swarms according to claim 2, characterized in that, To address the problem of determining spatial relationships between images from different UAVs, a geometric constraint mechanism based on flight state and imaging parameters is introduced, with the following steps: Collect flight altitude information and camera intrinsic parameters for each UAV at the corresponding time, including sensor size and focal length; Calculate the spatial baseline distance constraint range between the drones based on the acquired altitude information and camera parameters; Image frames acquired by different drones are combined in chronological order, and candidate frames that meet the spatial overlap condition are selected based on baseline distance constraints. The selected candidate frames are further processed for common-view determination to establish a set of keyframe matching relationships across UAVs.

4. The algorithm for relative positioning and real-time dense reconstruction of UAV swarms according to claim 3, characterized in that, To address the cross-platform image matching requirements in environments lacking satellite positioning information, stable co-view recognition is achieved through visual feature retrieval and multi-constraint fusion. The steps are as follows: Global feature vectors and local feature descriptors are extracted from keyframe images, and a feature index database is established. Based on the similarity measure between global features, a retrieval operation is performed on the input image to obtain several candidate image frames; Candidate frames are filtered by combining image timestamp information, and image pairs with a time interval lower than a preset threshold are excluded; Local feature matching is performed on the remaining candidate frames, and the results are verified in conjunction with spatial geometric consistency constraints to determine the cross-UAV co-view relationship.

5. The algorithm for relative positioning and real-time dense reconstruction of UAV swarms according to claim 4, characterized in that, Consistency enhancement constraints are applied to the matching results of cross-UAV co-view relationships. A set of feature matching relationships is constructed, the spatial distribution consistency between matching point pairs is calculated, the matching relationships are filtered in combination with reprojection error, and the co-view relationship set is updated according to geometric constraints to obtain stable matching results.

6. The algorithm for relative positioning and real-time dense reconstruction of UAV swarms according to claim 4, characterized in that, To address the issue of insufficient scale adaptability in multi-view depth estimation, a depth distribution constraint is introduced to optimize the depth search range and estimation process. The steps are as follows: Statistical analysis is performed on sparse 3D points in the local map to obtain their depth distribution information in the camera coordinate system; The upper and lower bounds of the depth search range are determined based on the statistical results of the depth distribution, which are used to constrain the subsequent depth estimation process; The image data, camera pose parameters, and depth search range are input into the depth estimation model to calculate the depth value of the corresponding pixel. The output depth map is filtered using confidence information, and depth results corresponding to low-confidence regions are removed.

7. The algorithm for relative positioning and real-time dense reconstruction of UAV swarms according to claim 6, characterized in that, To address error suppression and consistency assurance during the construction of dense point clouds, effective spatial points are selected through multi-view geometric constraints. The steps are as follows: Based on the pixel coordinates and corresponding depth values, the two-dimensional pixel points are back-projected into three-dimensional space using camera intrinsic parameters to obtain the spatial point coordinates; The obtained spatial points are mapped to adjacent views according to the camera pose relationship, and the corresponding reprojected pixel positions are calculated. Compare the differences between the original pixel positions and the reprojected positions, and calculate the corresponding depth deviation; Spatial points that meet the preset pixel error threshold and depth error threshold are selected and constructed into a local dense point cloud.

8. The algorithm for relative positioning and real-time dense reconstruction of UAV swarms according to claim 7, characterized in that, To address the scale differences and alignment errors between local maps from multiple UAVs, a phased registration strategy is employed to achieve unified coordinate representation. The steps are as follows: Feature matching is performed on sparse map points generated by different drones, and valid matching point pairs are selected through consistency constraints. Geometric constraints are constructed based on matching point pairs, and the initial alignment transformation is estimated using the random sampling consensus method. Using the initial alignment transformation as input, perform iterative registration calculations on the corresponding dense point cloud and update the transformation parameters; The updated transformation is applied to the point cloud data to fuse the point clouds from multiple UAVs and unify them into the same coordinate system.

9. The algorithm for relative positioning and real-time dense reconstruction of UAV swarms according to claim 8, characterized in that, To address the initial alignment transformation accuracy and the stability of dense point cloud registration, further constraints are imposed on the transformation solution and point cloud registration process. By filtering errors in the sparse point matching results and constructing weighted constraint relationships, and combining the convergence criteria in the transformation parameter optimization process, the correspondence relationships in the dense point cloud registration process are updated with constraints.

10. The algorithm for relative positioning and real-time dense reconstruction of UAV swarms according to claim 8, characterized in that, To address the issues of accumulated errors and scale inconsistencies during multi-UAV collaborative mapping, global pose unification is achieved by constructing multi-level constraint relationships. The steps are as follows: Establish pose constraints between consecutive keyframes within a single UAV and construct loop closure detection constraints; The pose variables inside a single UAV are optimized and calculated to update the pose of keyframes. Integrate the alignment relationships between different drones and incorporate cross-drone constraints into the overall optimization structure; The pose variables of all UAVs are jointly optimized to obtain pose results in a unified coordinate system.

Images