Single-region unmanned aerial vehicle cluster incremental SfM real-time point cloud reconstruction method

Abstract

The application provides a single-region unmanned aerial vehicle cluster incremental SfM real-time point cloud reconstruction method, which comprises the following steps: reconstructing the incremental SfM real-time point cloud of each unmanned aerial vehicle in a research region; based on a multi-dimensional coordinate unification and global consistency optimization scheme, registering and fusing the incremental SfM real-time point cloud of each unmanned aerial vehicle in the research region to obtain the reconstructed unmanned aerial vehicle cluster incremental SfM real-time point cloud in the research region; wherein the reconstruction process of the incremental SfM real-time point cloud of a single unmanned aerial vehicle comprises the following steps: receiving images taken by a single unmanned aerial vehicle in real time, and fusing a sliding window mechanism and deep learning to obtain matching image pairs corresponding to the latest images and matching point pairs; screening the matching image pairs corresponding to the latest images for reconstruction based on multiple geometric constraints and a local optimization strategy; and iteratively reconstructing the incremental SfM real-time point cloud of the unmanned aerial vehicle according to the matching image pairs corresponding to the latest images for reconstruction and the matching point pairs.

Description

Technical Field

[0001] This invention belongs to the field of point cloud reconstruction technology, specifically relating to a method for incremental SfM real-time point cloud reconstruction of a single-area UAV cluster. Background Technology

[0002] UAV swarm 3D reconstruction technology is widely used in surveying and mapping, emergency rescue, and other fields. The core requirement is to achieve high-frequency, low-latency, and structurally continuous 3D point cloud reconstruction. Structure of Motion (SfM) is a core technology and a key support for UAV swarm reconstruction systems.

[0003] Traditional SfM methods rely on offline image sets for global matching and optimization, resulting in high computational costs and extended processing time. This makes them unsuitable for the real-time requirements of dynamic image transmission from drone swarms. In asynchronous multi-drone shooting scenarios, they also face issues such as feature drift and strong initialization dependencies. While incremental SfM improves real-time performance through stepwise registration and fusion, it still suffers from drawbacks such as error accumulation, insufficient robustness in image matching, and unreasonable selection of initial image pairs, directly impacting reconstruction accuracy.

[0004] In multi-UAV collaborative scenarios, the local models generated independently by each UAV suffer from problems such as inconsistent coordinate systems, inconsistent scales, and structural redundancy. Directly stitching these models together can easily lead to difficulties in positioning and registration, and exacerbate attitude drift. Existing technologies have failed to effectively utilize GPS-assisted information and efficient global optimization mechanisms, making it difficult to solve the accuracy and consistency issues after multi-model fusion. Summary of the Invention

[0005] To overcome the problems of inconsistent coordinate systems, scale discrepancies, and structural redundancy in the local models independently generated by each UAV in multi-UAV collaborative scenarios, direct stitching can easily lead to difficulties in positioning and registration and exacerbate attitude drift. Existing technologies fail to effectively utilize GPS-assisted information and efficient global optimization mechanisms, making it difficult to solve the accuracy and consistency issues after multi-model fusion. This invention provides a single-region UAV swarm incremental SfM real-time point cloud reconstruction method, achieving high-frequency, low-latency, and structurally continuous 3D point cloud reconstruction. It is suitable for collaborative operation scenarios with asynchronous shooting and real-time data transmission from multiple UAVs, balancing reconstruction accuracy, real-time performance, and robustness.

[0006] According to one aspect of the present invention, a method for incremental SfM real-time point cloud reconstruction in a single-area UAV swarm is provided, comprising:

[0007] Reconstruct incremental SfM real-time point clouds for each UAV within the study area;

[0008] Based on the multi-dimensional coordinate unification and global consistency optimization scheme, the incremental SfM real-time point clouds of each UAV in the study area are registered and fused to obtain the reconstructed incremental SfM real-time point cloud of the UAV cluster in the study area.

[0009] The incremental SfM real-time point cloud reconstruction process for a single UAV includes:

[0010] It receives images captured by a single drone in real time and combines a sliding window mechanism with deep learning to obtain the latest image matching image pair and its matching point pair;

[0011] The matching image pairs for reconstruction are selected based on multiple geometric constraints and local optimum strategies;

[0012] The incremental SfM real-time point cloud of the UAV is iteratively reconstructed based on the matching image pairs and matching point pairs corresponding to the latest image. The iterative reconstruction is optimized by combining local BA and global BA, and is registered with GPS information.

[0013] As a further technical solution, the steps of integrating the sliding window mechanism and deep learning to obtain the latest image corresponding to the matching image pair and its matching point pair include:

[0014] According to the shooting time sequence, the images captured by the drone are received frame by frame using a sliding window. When the number of images in the sliding window reaches the preset maximum number of images in the sliding window, whenever the latest image is received, the sliding window moves one frame and the earliest image in the sliding window is removed.

[0015] Within the sliding window, whenever a new image is received, it is paired with other images in the sliding window one by one to obtain multiple matching image pairs. SuperGlue feature matching is performed on each matching image pair to extract the matching point pairs of the matching image pairs, and all matching image pairs and their matching point pairs corresponding to the latest image in the sliding window are output in real time.

[0016] As a further technical solution, the steps for selecting the matching image pair for reconstruction corresponding to the latest image based on multiple geometric constraints and local optimum strategies include:

[0017] Based on all matching image pairs and their matching point pairs within the sliding window, perform multi-constraint filtering to select reliable image pairs from all matching image pairs within the sliding window;

[0018] Within a sliding window, calculate the total number of matching points and the number of geometrically verified inliers for a reliable image pair. Calculate the inlier ratio based on the total number of matching points and the number of geometrically verified inliers. Select reliable image pairs that simultaneously meet the criteria of having a geometrically verified inlier number not less than a preset threshold and an inlier ratio not less than a preset threshold. When a reliable image pair that meets the selection requirements exists, select the reliable image pair with the highest inlier ratio as the matching image pair for the reconstruction of the latest image; otherwise, no matching image pair for reconstruction exists for the latest image.

[0019] As a further technical solution, multiple constraint screening includes homography monitoring constraints, baseline constraints, and takeoff image screening constraints.

[0020] As a further technical solution, the iterative reconstruction process, which combines local and global data analysis (BA) for optimization, includes:

[0021] Select a preset number of historical images captured at adjacent times to the latest image, and based on the matching image pairs and matching point pairs corresponding to the latest image, obtain the three-dimensional points jointly observed by the latest image and the selected historical images to form a local three-dimensional point set;

[0022] Construct an objective function to minimize reprojection error, and use the Levenberg-Marquardt algorithm to optimize the local 3D point set. Optimization stops when the preset maximum number of iterations is reached or the error is less than the preset convergence threshold. The local optimal 3D point set after local BA optimization is obtained, and the incremental SfM real-time point cloud of the UAV is updated using the local optimal 3D point set.

[0023] The cumulative total number of the latest images and the average reprojection error during the statistical reconstruction process are calculated. When the drift of the reprojection error of the latest images or the local optimal 3D point set relative to the average reprojection error exceeds a preset level for each newly added preset number of images, the global 3D point set of the current UAV's incremental SfM real-time point cloud is obtained. The Levenberg-Marquardt algorithm is then used to optimize the global 3D point set according to the objective function of minimizing the reprojection error. The global BA optimization is terminated when the decrease in the average reprojection error is less than a preset threshold or the maximum number of iterations is reached. The global BA-optimized global 3D point set is then obtained, and the UAV's incremental SfM real-time point cloud is updated using the global 3D point set.

[0024] As a further technical solution, the steps of iterative reconstruction combined with GPS information registration include:

[0025] The GPS coordinates of each image in the reconstructed matching image pair are converted into local coordinates and used as the initial position of the camera center. The initial pose estimation of the camera for each image is obtained by combining the image matching.

[0026] In the local BA optimization process, the GPS soft constraint term is introduced into the objective function of minimizing the reprojection error. Based on the initial pose estimation of the camera in each image, the Levenberg-Marquardt algorithm is used to optimize the local 3D point set, and the local optimal 3D point set after local BA optimization combined with GPS information registration is obtained.

[0027] As a further technical solution, the steps for reconstructing the incremental SfM real-time point cloud of the UAV swarm within the study area include:

[0028] The latitude and longitude coordinates of the incremental SfM real-time point cloud of each UAV are converted into the geocentric Cartesian coordinate system. Combined with the GPS position information of the UAV, the incremental SfM real-time point cloud of all UAVs is translated to the same spatial area by rigid body transformation. The origin of the unified local coordinate system is selected, and the three-dimensional point coordinates of the incremental SfM real-time point cloud of each UAV in the spatial area and the camera pose of each UAV are recalculated based on the origin of the unified local coordinate system.

[0029] Based on the 3D point coordinates of the incremental SfM real-time point cloud of each UAV and the camera pose of each UAV, the incremental SfM real-time point cloud of each UAV is fused, and the reprojection error term and GPS constraint term are fused to construct a globally consistent objective function. The Levenberg-Marquardt algorithm is used to optimize the fusion result of the incremental SfM real-time point cloud of each UAV, so as to obtain the reconstructed incremental SfM real-time point cloud of the UAV cluster in the study area.

[0030] According to another aspect of this specification, a single-area UAV swarm incremental SfM real-time point cloud reconstruction system is provided, comprising:

[0031] A single UAV SfM real-time point cloud reconstruction module is used to reconstruct incremental SfM real-time point clouds of each UAV within the research area.

[0032] The UAV swarm SfM real-time point cloud reconstruction module is used to register and fuse the incremental SfM real-time point clouds of each UAV in the study area based on a multi-dimensional coordinate unification and global consistency optimization scheme, so as to obtain the reconstructed incremental SfM real-time point cloud of the UAV swarm in the study area.

[0033] The incremental SfM real-time point cloud reconstruction process for a single UAV includes:

[0034] It receives images captured by a single drone in real time and combines a sliding window mechanism with deep learning to obtain the latest image matching image pair and its matching point pair;

[0035] The matching image pairs for reconstruction are selected based on multiple geometric constraints and local optimum strategies;

[0036] The incremental SfM real-time point cloud of the UAV is iteratively reconstructed based on the matching image pairs and matching point pairs corresponding to the latest image. The iterative reconstruction is optimized by combining local BA and global BA, and is registered with GPS information.

[0037] According to another aspect of this specification, an electronic device is provided, including a memory and a processor, the memory storing program instructions executed by the processor, the processor invoking the program instructions to perform a single-area UAV swarm incremental SfM real-time point cloud reconstruction method.

[0038] According to another aspect of this specification, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium storing computer instructions that cause the computer to execute a single-area UAV swarm incremental SfM real-time point cloud reconstruction method.

[0039] Compared with existing technologies, the advantages of this invention are as follows: First, existing technologies mostly employ traditional manual feature matching or single matching strategies, while this invention integrates SuperGlue deep learning matching and sliding window strategies to achieve a better balance between matching accuracy and real-time performance in complex scenes; Second, existing technologies rely on global filtering or directly take the first two frames for initial image pair selection, while this invention achieves fast and reliable initialization through quadruple geometric constraints and sliding window local filtering, avoiding reconstruction failure; Third, existing incremental SfM technologies lack efficient error suppression mechanisms, while this invention adopts a hierarchical optimization strategy combining local BA and global BA to accurately control the accumulation of local and global errors; Fourth, existing technologies are difficult to adapt to multi-UAV collaborative scenarios, while this invention establishes a unified coordinate reference system with the assistance of GPS information, combined with global consistency optimization, effectively solving the fusion problems such as inconsistent coordinate systems and scales among multiple models; Fifth, existing technologies are mostly offline processing or single UAV operations, while this invention achieves a real-time reconstruction mode of "processing while flying," significantly shortening the total project time.

[0040] Accordingly, this invention has the following advantages: strong real-time performance, supporting real-time transmission and synchronous processing of UAV images, significantly reducing the total engineering time compared to traditional methods; high reconstruction accuracy, with reprojection errors lower than traditional FLANN matching and single-UAV SfM methods, and better robustness in complex scenarios; 100% initialization success rate, effectively eliminating invalid image pairs through multiple constraints, avoiding reconstruction failures caused by pure rotation, excessively short baselines, etc.; good multi-UAV adaptability, capable of handling heterogeneous and asynchronous data from multiple UAVs, resulting in a coherent model structure and consistent positioning after fusion; low deployment cost, achievable based on conventional UAVs and computing units, without relying on expensive dedicated equipment; wide applicability, suitable for 3D reconstruction needs in multiple fields such as surveying and exploration, emergency rescue, and environmental monitoring; the solution of this invention is simple and convenient to implement, effectively solving the problems of poor real-time performance, difficulty in multi-UAV fusion, and insufficient robustness in complex scenarios of traditional SfM, possessing significant engineering application value and market prospects. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of a method for incremental SfM real-time point cloud reconstruction of a single-area UAV cluster provided by an embodiment of the present invention;

[0043] Figure 2 This is a flowchart illustrating the incremental SfM real-time point cloud reconstruction process for a single UAV in an embodiment of the present invention.

[0044] Figure 3 This is a schematic diagram illustrating the comparison of matching effects in an embodiment of the present invention: Figure 3 (a) in the image is an example of the matching effect comparison in some scenes with large perspective changes. Figure 3 (b) in the figure is an example of the matching effect comparison in a complex scene;

[0045] Figure 4 This is a comparison chart of the results after global optimization of SfM in an embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of a single-area UAV swarm incremental SfM real-time point cloud reconstruction system provided in an embodiment of the present invention;

[0047] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0048] It should be noted that:

[0049] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0050] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be decomposed, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0052] like Figure 1 , Figure 2 As shown, this invention provides a method for incremental SfM real-time point cloud reconstruction in a single-area UAV swarm, comprising:

[0053] Step 1: Reconstruct the incremental SfM real-time point cloud of each UAV within the study area;

[0054] The incremental SfM real-time point cloud reconstruction process for a single UAV includes:

[0055] Step 1-1: Receive images captured by a single drone in real time, and combine the sliding window mechanism with deep learning to obtain the matching image pairs and matching point pairs corresponding to the latest images;

[0056] Steps 1-2: Select matching image pairs for reconstruction based on multiple geometric constraints and local optimum strategies;

[0057] Steps 1-3: Iteratively reconstruct the incremental SfM real-time point cloud of the UAV based on the matching image pairs and matching point pairs corresponding to the latest image. The iterative reconstruction is combined with GPS information registration and optimized by combining local BA and global BA.

[0058] Step 2: Based on multi-dimensional coordinate unification and global consistency, the incremental SfM real-time point clouds of each UAV in the study area are registered and fused to obtain the reconstructed incremental SfM real-time point cloud of the UAV cluster in the study area.

[0059] In step 1-1, the steps of integrating the sliding window mechanism and deep learning to obtain the matching image pair and its matching point pair corresponding to the latest image include:

[0060] Step 1-1-1: According to the shooting time sequence, the images captured by the drone are received frame by frame using a sliding window. When the number of images in the sliding window reaches the preset maximum number of images in the sliding window, whenever the latest image is received, the sliding window moves one frame and the earliest captured image in the sliding window is removed.

[0061] Step 1-1-2: Within the sliding window, whenever a new image is received, the new image is paired with other images in the sliding window one by one to obtain multiple matching image pairs. SuperGlue feature matching is performed on each matching image pair to extract the matching point pairs of the matching image pairs, and all matching image pairs and their matching point pairs of the newest image in the sliding window are output in real time.

[0062] Specifically, step 1-1-1 essentially involves using a sliding window to limit the matching range: assuming the received image set... for: (I represents UAV imagery, and the subscript is the sequence number), define the sliding window size as K, and the set of images within the sliding window. for: The latest image is only matched with historical images within the sliding window, avoiding meaningless global matching and reducing computational overhead and incorrect matching.

[0063] Step 1-1-2 of SuperGlue feature matching includes: using a convolutional neural network (CNN, such as SuperPoint) to extract key points and descriptors of each image in the image matching pair, constructing key points into graph structure nodes, iteratively updating context information through a graph neural network (GNN), and combining a cross-attention mechanism to find the correspondence between images to obtain matching point pairs, thereby improving the matching accuracy in scenes with occlusion, repeated textures, and drastic changes in viewpoint.

[0064] Furthermore, in steps 1-2, the step of selecting the matching image pair for reconstruction corresponding to the latest image based on multiple geometric constraints and local optimum strategies includes:

[0065] Based on all matching image pairs and their matching point pairs within the sliding window, perform multi-constraint filtering to select reliable image pairs from all matching image pairs within the sliding window;

[0066] Within a sliding window, calculate the total number of matching points and the number of geometrically verified inliers for a reliable image pair. Calculate the inlier ratio based on the total number of matching points and the number of geometrically verified inliers. Select reliable image pairs that simultaneously meet the criteria of having a geometrically verified inlier number not less than a preset threshold and an inlier ratio not less than a preset threshold. When a reliable image pair that meets the selection requirements exists, select the reliable image pair with the highest inlier ratio as the matching image pair for the reconstruction of the latest image; otherwise, no matching image pair for reconstruction exists for the latest image.

[0067] The multiple constraint screening includes homography monitoring constraints, baseline constraints, and takeoff image screening constraints.

[0068] Specifically, firstly, homography monitoring constraints include: for matched image pairs ( The homography matrix H and the fundamental matrix F are estimated using the RANSAC method, and the corresponding number of interior points is calculated. , This represents the number of interior points in the corresponding homography matrix. The number of interior points of the corresponding fundamental matrix; when hour( (Using a preset threshold), the matching image pairs are determined to be either nearly coplanar or purely rotated image pairs and are then discarded.

[0069] Secondly, baseline constraints include: calculating the essential matrix based on the matching point pairs of the matched image pairs, and recovering the camera pose based on the essential matrix, using triangulated 3D points. Computational camera center , The included angle Statistical mean angle Remove image pairs with excessively short baselines;

[0070] Third, the constraints for screening takeoff images include: removing images dominated by forward motion based on the forward motion ratio, and removing images with abnormal perspectives caused by vertical takeoff based on the camera pitch angle.

[0071] Fourth, the mathematical representation of the local optimum strategy selection is: calculating the total number of matching points for image pairs within a sliding window. Geometric verification of the number of interior points and interior point ratio Filter to meet , ( , These are the preset thresholds for the number of geometric verification interior points and the preset threshold for the proportion of interior points, respectively. The highest-valued image pair is used as the initial image pair; if the threshold is not reached, the window waits for new images and moves backward.

[0072] Steps 1-3 also include the initialization process: when the number of matching image pairs for reconstruction of the latest selected image meets the preset number, reconstruction is started. Relative pose estimation: using the selected matching points, the essential matrix or fundamental matrix is ​​estimated by the five-point method or the eight-point method, and the relative pose of the two cameras (relative rotation R and relative translation t) is calculated. Based on the relative pose, the matching points corresponding to the matching image pairs for reconstruction are triangulated into a three-dimensional point cloud, thus completing the construction of the initial point cloud.

[0073] Furthermore, in steps 1-3, iterative reconstruction combined with GPS information registration is optimized through a combination of local BA and global BA. The optimization process includes:

[0074] Select a preset number of historical images captured at adjacent times to the latest image, and based on the matching image pairs and matching point pairs corresponding to the latest image, obtain the three-dimensional points jointly observed by the latest image and the selected historical images to form a local three-dimensional point set;

[0075] A reprojection error minimization objective function is constructed. Combined with GPS information registration, the Levenberg-Marquardt algorithm is used to locally optimize the local 3D point set. The optimization stops when the preset maximum number of iterations is reached or the error is less than the preset convergence threshold. The local optimal 3D point set after local BA optimization combined with GPS information registration is obtained. The local optimal 3D point set is then used to update the UAV's incremental SfM real-time point cloud.

[0076] The cumulative total number of the latest images and the average reprojection error during the statistical reconstruction process are calculated. When the drift of the reprojection error of the latest images or the local optimal 3D point set relative to the average reprojection error exceeds a preset level for each additional preset number of latest images or the local optimal 3D point set, the global 3D point set of the incremental SfM real-time point cloud of the current UAV is obtained. The Levenberg-Marquardt algorithm is then used to optimize the reprojection error based on the objective function of minimizing the reprojection error until the decrease in the average reprojection error is less than a preset threshold or the maximum number of iterations is reached, at which point the global BA optimization is terminated.

[0077] Among them, the degree of drift of the reprojection error of the locally optimal 3D point set relative to the average reprojection error. The mathematical representation of is:

[0078]

[0079] In the formula, This represents the observed pixel coordinates of the j-th 3D point in the i-th frame of the image; Represents the drone camera pose of the i-th frame; The projection function represents the predicted coordinates obtained by projecting 3D points onto the image plane according to the UAV pose; O represents the set of local images used for reconstruction, which contains a preset number of the latest images; This represents the set of images used for reconstruction, containing all images used for reconstruction.

[0080] Furthermore, in steps 1-3, the steps of combining GPS information registration and using the Levenberg-Marquardt algorithm for local BA optimization of the local 3D point set include:

[0081] The GPS coordinates of each image in the reconstructed matching image pair are converted into local coordinates and used as the initial position of the camera center. The initial pose estimation of the camera for each image is obtained by combining the image matching.

[0082] In the local BA optimization process, the GPS soft constraint error is calculated based on the camera initial pose estimation of each image. The GPS soft constraint error is introduced as a GPS soft constraint term into the objective function of minimizing the reprojection error, and the Levenberg-Marquardt algorithm is used to optimize the local 3D point set.

[0083] Step 2, the steps for reconstructing the incremental SfM real-time point cloud of the UAV swarm within the study area, include:

[0084] The latitude and longitude coordinates of the incremental SfM real-time point cloud of each UAV are converted into the geocentric Cartesian coordinate system. Combined with the GPS position information of the UAV, the incremental SfM real-time point cloud of all UAVs is translated to the same spatial area by rigid body transformation. The origin of the unified local coordinate system is selected, and the three-dimensional point coordinates of the incremental SfM real-time point cloud of each UAV in the spatial area and the camera pose of each UAV are recalculated based on the origin of the unified local coordinate system.

[0085] Based on the 3D point coordinates of the incremental SfM real-time point cloud of each UAV and the camera pose of each UAV, the incremental SfM real-time point cloud of each UAV is fused, and the reprojection error term and GPS constraint term are fused to construct a globally consistent objective function. The Levenberg-Marquardt algorithm is used to optimize the fusion result of the incremental SfM real-time point cloud of each UAV, so as to obtain the reconstructed incremental SfM real-time point cloud of the UAV cluster in the study area.

[0086] The method described in this embodiment of the invention is based on aerial imagery and GPS positioning data collected by multiple UAVs to achieve real-time, high-precision sparse point cloud reconstruction, and includes the following design:

[0087] Design 1 is an image matching strategy that integrates deep learning and sliding window to balance matching accuracy, real-time performance and computational cost.

[0088] Design 2, based on a dynamic initialization image pair filtering mechanism with multiple geometric constraints and a sliding window, ensures initialization reliability and real-time startup;

[0089] Design 3: A layered bundle adjustment (BA) iterative reconstruction strategy that combines local and global approaches to accurately suppress error accumulation;

[0090] Design 4, which combines incremental SfM with GPS information, lays a unified coordinate foundation for multi-machine fusion;

[0091] Design 5, based on a multi-dimensional coordinate unification and global consistency optimization scheme, achieves accurate registration and fusion of local models of multiple UAVs.

[0092] In Design 1, a two-stage image matching method is implemented by combining SuperGlue depth feature matching and sliding window matching, balancing accuracy and real-time performance. The specific implementation is as follows:

[0093] The specific implementation of SuperGlue deep feature matching: SuperGlue first uses CNNs (such as SuperPoint) to extract keypoints and descriptors from the image, and then constructs the keypoints into graph structure nodes, with attention mechanisms connecting the nodes to build the graph. During the matching process, SuperGlue uses GNNs to iteratively update the contextual information of each point and uses a cross-attention mechanism to find possible correspondences between images.

[0094] The specific implementation of sliding window matching: 1) Window parameter settings: Based on the UAV image transmission frequency (assuming 3 seconds / frame) and overlap characteristics, set the sliding window size. That is, the window contains the 10 most recently received frames of imagery. 2) Window update mechanism: When new images... When inputting, if the current image set scale ,window Automatically shifts back, discarding the earliest received image and incorporating new images. 3) Matching range limitation: New images are only matched with historical images within the window, and no matching relationship is calculated with images outside the window, reducing unnecessary computational overhead.

[0095] Figure 3 A schematic diagram showing the matching effect comparison according to Design 1 is shown: Figure 3 (a) in the image is an example of the matching effect comparison in some scenes with large perspective changes. Figure 3 (b) in the figure is an example of the matching effect comparison in a complex scene.

[0096] In Design 2, the optimal initial image pair is selected through multiple constraints to ensure the reliability of reconstruction and initialization. The specific implementation is as follows:

[0097] Specific implementation of homography constraint monitoring. 1) Matrix estimation: For candidate matching image pairs ( The homography matrix is ​​estimated using the RANSAC algorithm. With the fundamental matrix 2) Determining the proportion of interior points: Statistical analysis Corresponding number of interior points and Corresponding number of interior points Calculate the ratio When the ratio is less than 0.25 (empirical threshold), the image pair is determined to be nearly coplanar or purely rotated and is therefore discarded.

[0098] Specific implementation of baseline constraints. 1) Pose recovery: Based on matched point pairs, the relative pose of the camera (rotation matrix) is obtained by decomposing the essential matrix E. Translation vector ), thereby determining the center of the two cameras , 1) Coordinates. 2) Angle calculation: Triangulate the matching points to obtain the three-dimensional points. According to the formula and calculate and , The included angle Statistical mean angle ,when If the baseline is less than 1.5°, the image pair is deemed too short and is discarded.

[0099] Specific implementation of takeoff image screening. 1) Calculation of forward motion ratio: Based on the camera's relative translation vector t: ( (for the three components of t), according to Calculate the forward motion ratio ,when 1) When the forward motion is dominant, the image pair is removed. 2) Pitch angle detection: The camera pitch angle data is obtained through the UAV flight control system. When the pitch angle is >45°, it is judged as an abnormal vertical takeoff image and is directly excluded from the candidate image pair.

[0100] The specific implementation of local optimum selection involves calculating the matching results between the new image and all historical images within the sliding window. and Filter image pairs that meet the threshold, and select them. The highest image pair; if the highest If the result is positive, it is determined as the initial image pair; otherwise, wait for new images to be received, move the window back, and re-filter. The threshold parameter is set as follows: , That is, when an image pair simultaneously satisfies the conditions of having ≥80 inliers and an inlier ratio ≥0.4, it is determined to be a matching image pair for reconstruction.

[0101] In Design 3, iterative reconstruction optimization is achieved through a hierarchical BA strategy to suppress error accumulation, including:

[0102] Design 3.1, Local BA Optimization: For each new image, only the pose parameters of that image and its adjacent images are optimized. Corresponding three-dimensional points We perform reprojection error minimization optimization to quickly suppress local error propagation. The objective function is:

[0103]

[0104] In the formula, This represents the observed pixel coordinates of the j-th 3D point in the i-th frame of the image; Represents the drone camera pose of the i-th frame; The projection function represents the predicted coordinates obtained by projecting 3D points onto the image plane based on the UAV pose.

[0105] Design 3.2, Global BA Optimization: Set trigger conditions (such as the cumulative number of newly added images, error threshold) to jointly optimize the poses and 3D points of all images, globally minimize the reprojection error of the entire model, and thus obtain the best geometric consistency of the entire scene.

[0106] Specifically, the implementation of Design 3 includes:

[0107] Specific implementation of local BA:

[0108] 1) Optimize object selection: For each newly added image, select that image and 7-9 adjacent images. The optimal set of local optimizations is composed of the three-dimensional points observed by these images together (the optimal ones). 2) Solving the objective function: Construct an objective function that minimizes the reprojection error, and solve it using the Levenberg-Marquardt algorithm. The maximum number of iterations is [not specified]. The convergence threshold is set to The optimization stops when the number of iterations reaches 20 or the error is less than the threshold.

[0109] Specific implementation of global BA:

[0110] When the system detects a significant expansion in the 3D reconstruction scale, or when local optimization strategies fail to effectively suppress error accumulation, a global reprojection (BA) optimization process is triggered. In this scheme, a quantitative trigger for global BA activation is defined as the number of newly added images exceeding 20. The global BA activation is triggered when the overall reprojection error decreases by less than a preset threshold of 1×10⁻⁶. -5 The global BA optimization process is terminated immediately when the iterative calculation reaches the maximum number of iterations (80). Figure 4 This is a comparison of the results after global optimization of SfM in the embodiments of the present invention.

[0111] Furthermore, based on Design 3, Design 4 improves the implementation process of local BA by incorporating GPS information, laying a unified coordinate foundation for multi-machine fusion, including:

[0112] Design 4.1, GPS Initial Pose Estimation: The image GPS coordinates (lat, lon, alt) (longitude, latitude, altitude) are transformed into a local coordinate system using the WGS84 coordinate system, which serves as the initial position of the camera center. The initial pose estimation of the camera is completed by combining the relative rotation obtained from image matching;

[0113] Design 4.2, Incorporation of GPS Soft Constraints: In local BA optimization, GPS constraint terms are introduced. : ( The location of the UAV corresponding to the i-th frame image obtained by GPS is incorporated as an additional error term into the optimization objective to suppress local reconstruction drift.

[0114] Specifically, the implementation of Design 4 includes:

[0115] GPS initial pose estimation. The specific steps are: 1) Coordinate transformation: Convert the GPS coordinates (lat, lon, alt) of the UAV imagery to the WGS84 geodetic coordinate system using the pyproj tool, and then convert them to the UTM local plane coordinate system to obtain the initial position of the camera center. 2) Initial pose estimation: Combined with the relative rotation estimated based on image matching, the initial pose is estimated.

[0116] GPS soft constraint integration: incorporating the drone camera position measured by GPS. With the reconstruction of the camera center The Euclidean distance is introduced as an additional error term in the optimization objective, and the objective function is: .

[0117] In Design 5, the steps for achieving multi-UAV model registration and fusion include:

[0118] Design 5.1, Coordinate Transformation and Unification: Use the pyproj tool to convert the latitude and longitude coordinates of each local model to the geocentric coordinate system, and then to the local plane rectangular coordinate system; perform initial model alignment based on spatial location, select a UAV takeoff point as the unified reference origin, complete coordinate normalization, and solve the problem of numerical instability;

[0119] Design 5.2, Global Consistency Optimization: Global Bundle Adjustment (GBA) is employed to jointly minimize reprojection error and GPS constraint error. The objective function is: Eliminate multi-model boundary errors, scale inconsistencies, and structural redundancy;

[0120] Specifically, the implementation of Design 5 includes:

[0121] Coordinate Transformation and Unification: 1) Geocentric Coordinate Transformation: Using pyproj, the latitude and longitude coordinates of the local models of each UAV (i.e., the incremental SfM real-time point cloud of a single UAV) are converted to the geocentric Cartesian coordinate system (ECEF) to eliminate geographical coordinate differences. 2) Initial Alignment: Based on the GPS position information of each UAV, rigid body transformation (translation + rotation) is used to translate all local models to the same spatial region, with the alignment error controlled within 5 meters. 3) Origin Normalization: The takeoff point of UAV No. 1 is selected as the origin of the unified local coordinate system. The coordinate values ​​of all models are recalculated relative to this origin, so that the coordinate value range is controlled within [-5000, 5000] meters to avoid numerical calculation instability.

[0122] Global consistency optimization: by formula ( This represents the set of three-dimensional points that the camera can observe in the i-th frame of the image. The objective function is constructed using weight coefficients, and the reprojection error term and GPS constraint term are integrated. The Levenberg-Marquardt algorithm is used to solve the nonlinear least squares problem, eliminating boundary errors and scale inconsistencies between models.

[0123] The implementation of the various embodiments of the present invention is based on programmed processing through a system with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of the present invention are encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of the present invention provide a single-area UAV swarm incremental SfM real-time point cloud reconstruction system, which is used to execute a single-area UAV swarm incremental SfM real-time point cloud reconstruction method from the above method embodiments.

[0124] See Figure 5 The system includes:

[0125] A single UAV SfM real-time point cloud reconstruction module is used to reconstruct incremental SfM real-time point clouds of each UAV within the research area.

[0126] The UAV swarm SfM real-time point cloud reconstruction module is used to register and fuse the incremental SfM real-time point clouds of each UAV in the study area based on a multi-dimensional coordinate unification and global consistency optimization scheme, so as to obtain the reconstructed incremental SfM real-time point cloud of the UAV swarm in the study area.

[0127] The incremental SfM real-time point cloud reconstruction process for a single UAV includes:

[0128] It receives images captured by a single drone in real time and combines a sliding window mechanism with deep learning to obtain the latest image matching image pair and its matching point pair;

[0129] The matching image pairs for reconstruction are selected based on multiple geometric constraints and local optimum strategies;

[0130] The incremental SfM real-time point cloud of the UAV is iteratively reconstructed based on the matching image pairs and matching point pairs corresponding to the latest image. The iterative reconstruction is optimized by combining local BA and global BA, and is registered with GPS information.

[0131] It should be noted that the system embodiments provided by the present invention are used not only to implement the methods in the above method embodiments, but also to implement the methods in other method embodiments provided by the present invention. The only difference is that corresponding functional modules are set. The principle is basically the same as that of the above system embodiments provided by the present invention. As long as those skilled in the art can improve the modules in the above system embodiments by referring to the specific technical solutions in other method embodiments and combining technical features to obtain corresponding technical means and technical solutions composed of these technical means, on the basis of the above system embodiments, and on the premise of ensuring the practicality of the technical solutions, they can obtain corresponding system-like embodiments for implementing the methods in other method-like embodiments.

[0132] The method in this embodiment of the invention is implemented using an electronic device; therefore, it is necessary to introduce the relevant electronic device. For this purpose, embodiments of the present invention provide an electronic device, such as... Figure 6 As shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus, wherein the at least one processor, the communication interface, and the at least one memory communicate with each other via the communication bus. The at least one processor invokes logical instructions stored in the at least one memory to execute all or part of the steps of the methods provided in the foregoing method embodiments.

[0133] Furthermore, when the logical instructions in at least one of the aforementioned memories are implemented as software functional units and sold or used as independent products, they are stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, is embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (a personal computer, server, or network device) to execute all or part of the steps of the methods described in the various method embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks—various media for storing program code.

[0134] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, located in one place, or distributed across multiple network units. The purpose of this embodiment is achieved by selecting some or all of the modules according to actual needs. Those skilled in the art will understand and implement this without any inventive effort.

[0135] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0137] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0138] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0139] Based on the same technical concept as the foregoing embodiments, the present invention provides a non-transitory computer-readable storage medium that stores computer instructions that cause the computer to execute a single-area UAV swarm incremental SfM real-time point cloud reconstruction method.

[0140] In summary, this invention discloses an incremental SfM real-time point cloud reconstruction method for a single-region UAV swarm, specifically designed for collaborative operation scenarios involving asynchronous multi-UAV shooting and real-time data transmission. The steps include: 1) fusing SuperGlue deep learning feature matching with a sliding window strategy to achieve two-stage image matching, balancing robustness in complex scene matching with computational efficiency; 2) selecting reliable initial image pairs through homography monitoring, baseline constraints, and other geometric constraints to ensure reconstruction startup stability; 3) employing a layered bundle adjustment (BA) strategy combining local and global methods to accurately suppress error accumulation and improve reconstruction accuracy; 4) optimizing the local model of a single UAV with GPS assistance and establishing a unified reference basis through coordinate transformation, paving the way for multi-UAV fusion; 5) achieving accurate registration and fusion of local models from multiple UAVs based on a multi-dimensional coordinate unification and global consistency optimization scheme, eliminating problems such as scale inconsistency and structural redundancy. This invention achieves high real-time, high-precision 3D point cloud reconstruction, applicable to multiple fields such as surveying and exploration, emergency rescue, and environmental monitoring.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method for incremental SfM real-time point cloud reconstruction in a single-region UAV swarm, characterized in that, include: Reconstruct incremental SfM real-time point clouds for each UAV within the study area; Based on the multi-dimensional coordinate unification and global consistency optimization scheme, the incremental SfM real-time point clouds of each UAV in the study area are registered and fused to obtain the reconstructed incremental SfM real-time point cloud of the UAV cluster in the study area. The incremental SfM real-time point cloud reconstruction process for a single UAV includes: It receives images captured by a single drone in real time and combines a sliding window mechanism with deep learning to obtain the latest image matching image pair and its matching point pair; The matching image pairs for reconstruction are selected based on multiple geometric constraints and local optimum strategies; The incremental SfM real-time point cloud of the UAV is iteratively reconstructed based on the matching image pairs and matching point pairs corresponding to the latest image. The iterative reconstruction is optimized by combining local BA and global BA, and is registered with GPS information.

2. The method for incremental SfM real-time point cloud reconstruction of a single-area UAV swarm as described in claim 1, characterized in that, The steps for obtaining the latest image matching image pair and its matching point pair by integrating the sliding window mechanism and deep learning include: According to the shooting time sequence, the images captured by the drone are received frame by frame using a sliding window. When the number of images in the sliding window reaches the preset maximum number of images in the sliding window, whenever the latest image is received, the sliding window moves one frame and the earliest image in the sliding window is removed. Within the sliding window, whenever a new image is received, it is paired with other images in the sliding window one by one to obtain multiple matching image pairs. SuperGlue feature matching is performed on each matching image pair to extract the matching point pairs of the matching image pairs, and all matching image pairs and their matching point pairs corresponding to the latest image in the sliding window are output in real time.

3. The method for incremental SfM real-time point cloud reconstruction of a single-area UAV swarm as described in claim 1, characterized in that, The steps for selecting the latest image matching pair for reconstruction based on multiple geometric constraints and local optimum strategies include: Based on all matching image pairs and their matching point pairs within the sliding window, perform multi-constraint filtering to select reliable image pairs from all matching image pairs within the sliding window; Within a sliding window, calculate the total number of matching points and the number of geometrically verified inliers for a reliable image pair. Calculate the inlier ratio based on the total number of matching points and the number of geometrically verified inliers. Select reliable image pairs that simultaneously meet the criteria of having a geometrically verified inlier number not less than a preset threshold and an inlier ratio not less than a preset threshold. When a reliable image pair that meets the selection requirements exists, select the reliable image pair with the highest inlier ratio as the matching image pair for the reconstruction of the latest image; otherwise, no matching image pair for reconstruction exists for the latest image.

4. The method for incremental SfM real-time point cloud reconstruction of a single-area UAV swarm as described in claim 3, characterized in that, The multiple constraint screening includes homography monitoring constraints, baseline constraints, and takeoff image screening constraints.

5. The method for incremental SfM real-time point cloud reconstruction of a single-area UAV swarm as described in claim 1, characterized in that, Iterative reconstruction, through the optimization process of combining local and global business intelligence (BA), includes: Select a preset number of historical images captured at adjacent times to the latest image, and based on the matching image pairs and matching point pairs corresponding to the latest image, obtain the three-dimensional points jointly observed by the latest image and the selected historical images to form a local three-dimensional point set; Construct an objective function to minimize reprojection error, and use the Levenberg-Marquardt algorithm to optimize the local 3D point set. Optimization stops when the preset maximum number of iterations is reached or the error is less than the preset convergence threshold. The local optimal 3D point set after local BA optimization is obtained, and the incremental SfM real-time point cloud of the UAV is updated using the local optimal 3D point set. The cumulative total number of the latest images and the average reprojection error during the statistical reconstruction process are calculated. When the drift of the reprojection error of the latest images or the local optimal 3D point set relative to the average reprojection error exceeds a preset level for each newly added preset number of images, the global 3D point set of the current UAV's incremental SfM real-time point cloud is obtained. The Levenberg-Marquardt algorithm is then used to optimize the global 3D point set according to the objective function of minimizing the reprojection error. The global BA optimization is terminated when the decrease in the average reprojection error is less than a preset threshold or the maximum number of iterations is reached. The global BA-optimized global 3D point set is then obtained, and the UAV's incremental SfM real-time point cloud is updated using the global 3D point set.

6. The method for incremental SfM real-time point cloud reconstruction of a single-area UAV swarm as described in claim 5, characterized in that, The steps of iterative reconstruction combined with GPS information registration include: The GPS coordinates of each image in the reconstructed matching image pair are converted into local coordinates and used as the initial position of the camera center. The initial pose estimation of the camera for each image is obtained by combining the image matching. In the local BA optimization process, the GPS soft constraint term is introduced into the objective function of minimizing the reprojection error. Based on the initial pose estimation of the camera in each image, the Levenberg-Marquardt algorithm is used to optimize the local 3D point set, and the local optimal 3D point set after local BA optimization combined with GPS information registration is obtained.

7. The method for incremental SfM real-time point cloud reconstruction of a single-area UAV swarm as described in claim 6, characterized in that, The steps for reconstructing the incremental SfM real-time point cloud of the UAV swarm within the study area include: The latitude and longitude coordinates of the incremental SfM real-time point cloud of each UAV are converted into the geocentric Cartesian coordinate system. Combined with the GPS position information of the UAV, the incremental SfM real-time point cloud of all UAVs is translated to the same spatial area by rigid body transformation. The origin of the unified local coordinate system is selected, and the three-dimensional point coordinates of the incremental SfM real-time point cloud of each UAV in the spatial area and the camera pose of each UAV are recalculated based on the origin of the unified local coordinate system. Based on the 3D point coordinates of the incremental SfM real-time point cloud of each UAV and the camera pose of each UAV, the incremental SfM real-time point cloud of each UAV is fused, and the reprojection error term and GPS constraint term are fused to construct a globally consistent objective function. The Levenberg-Marquardt algorithm is used to optimize the fusion result of the incremental SfM real-time point cloud of each UAV, so as to obtain the reconstructed incremental SfM real-time point cloud of the UAV cluster in the study area.

8. A single-area UAV swarm incremental SfM real-time point cloud reconstruction system, characterized in that, include: A single UAV SfM real-time point cloud reconstruction module is used to reconstruct incremental SfM real-time point clouds of each UAV within the research area. The UAV swarm SfM real-time point cloud reconstruction module is used to register and fuse the incremental SfM real-time point clouds of each UAV in the study area based on a multi-dimensional coordinate unification and global consistency optimization scheme, so as to obtain the reconstructed incremental SfM real-time point cloud of the UAV swarm in the study area. The incremental SfM real-time point cloud reconstruction process for a single UAV includes: It receives images captured by a single drone in real time and combines a sliding window mechanism with deep learning to obtain the latest image matching image pair and its matching point pair; The matching image pairs for reconstruction are selected based on multiple geometric constraints and local optimum strategies; The incremental SfM real-time point cloud of the UAV is iteratively reconstructed based on the matching image pairs and matching point pairs corresponding to the latest image. The iterative reconstruction is optimized by combining local BA and global BA, and is registered with GPS information.

9. An electronic device, characterized in that, The method includes a memory and a processor, the memory storing program instructions that are executed by the processor, the processor invoking the program instructions to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the method described in any one of claims 1 to 7.

Images