Unmanned aerial vehicle target displacement and settlement monitoring method and system based on SFM

Abstract

The invention provides an SFM-based unmanned aerial vehicle target displacement and settlement monitoring method and system, relates to the technical field of computer vision, and solves the technical problem that automatic, high-precision and traceable time sequence three-dimensional coordinate monitoring cannot be performed on a single target in the prior art. The method comprises the steps of obtaining an unmanned aerial vehicle image sequence; performing feature point extraction and matching on the unmanned aerial vehicle image sequence through a feature extraction algorithm to obtain homonymy feature point pairs; performing three-dimensional reconstruction and geographical registration on the homonymous feature point pairs through an SFM algorithm to obtain an image-geographical coordinate mapping relation; performing target identification and segmentation on the unmanned aerial vehicle image sequence through an image segmentation algorithm to obtain target pixel coordinates; performing coordinate conversion on the target pixel coordinates based on the image-geographic coordinate mapping relation to obtain target geographic coordinates; and comparing and analyzing the geographic coordinates of the target to obtain the displacement amount and the settlement amount of the target. The slope deformation monitoring device is used in the slope deformation monitoring process.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and system for monitoring the displacement and settlement of unmanned aerial vehicle targets based on SFM. Background Technology

[0002] In the field of slope stability and deformation monitoring, existing technologies mainly rely on UAV photogrammetry and 3D reconstruction methods. This technology typically uses UAVs to collect regional images, generates a holistic 3D point cloud or model using feature matching and structured bundle dynamics (SFM), and analyzes terrain deformation trends through overall comparison across different time phases. However, this approach has significant limitations: First, its monitoring focuses on macroscopic terrain, making it difficult to achieve high-precision, target-level displacement and settlement extraction for individual targets. Second, the processing flow lacks a stable and traceable mapping relationship from image pixels to geographic coordinates, resulting in insufficient reliability of image-based fine measurements. Third, target identification largely relies on manual interpretation, leading to low efficiency and poor consistency. Fourth, due to the lack of a continuous and automated coordinate extraction and alignment mechanism for the same target, it is difficult to support long-term temporal quantitative analysis of target 3D coordinate changes, especially when dealing with centimeter-level small-scale deformations, where its accuracy and reliability fail to meet the precise requirements of engineering monitoring. Summary of the Invention

[0003] This application provides a method and system for monitoring the displacement and settlement of UAV targets based on SFM, which solves the technical problem that the existing technology focuses on the macroscopic deformation analysis of the overall terrain and cannot perform automated, high-precision, and traceable time-series three-dimensional coordinate monitoring of individual targets.

[0004] To achieve the above objectives, this application adopts the following technical solution: Firstly, a method for monitoring the displacement and settlement of UAV targets based on SFM is provided, including: By using drones to collect multi-view images of the monitored area, a drone image sequence is obtained; Feature points are extracted and matched from UAV image sequences using feature extraction algorithms to obtain pairs of feature points with the same name. The SFM algorithm is used to perform 3D reconstruction and georegistration on pairs of identical feature points to obtain the image-geographic coordinate mapping relationship. Target identification and segmentation of UAV image sequences are performed using image segmentation algorithms to obtain target pixel coordinates. Based on the image-geographic coordinate mapping relationship, the target pixel coordinates are transformed to obtain the target geographic coordinates; By comparing and analyzing the geographic coordinates of the target, the displacement and settlement of the target can be obtained.

[0005] Based on the above technical solution, in the UAV target displacement and settlement monitoring method based on SFM provided in this application, by establishing a stable and traceable image-geographic coordinate mapping relationship, the target automatic identification results based on deep learning are directly converted into accurate three-dimensional geographic coordinates, thereby overcoming the technical bottleneck of traditional methods that rely on manual labor and are difficult to associate pixels with geographic space; finally, by automatically aligning and continuously tracking the three-dimensional coordinates of the same target in multiple time phases, quantitative and automated analysis of the target's horizontal displacement and vertical settlement is realized, significantly improving the accuracy, efficiency and reliability of deformation monitoring in engineering scenarios such as slopes.

[0006] In conjunction with the first aspect above, in one possible implementation, the step of extracting and matching feature points from the UAV image sequence using a feature extraction algorithm includes: Feature points are obtained by detecting feature points in UAV image sequences using the DSP-SIFT algorithm; The UAV image sequence is divided into multiple grid cells, and the image feature points in the grid cells are subjected to non-maximum suppression algorithm to obtain high response feature points. The FLANN algorithm is used to match high-response feature points to obtain initial feature point pairs. The GMS algorithm is used to remove mismatches from the initial feature point pairs to obtain feature point pairs with the same name.

[0007] In conjunction with the first aspect above, in one possible implementation, the feature point detection of the UAV image sequence using the DSP-SIFT algorithm includes: Constructing a Gaussian multi-scale space based on UAV image sequences; Extremum point detection is performed in the Gaussian multi-scale space by neighborhood comparison to obtain candidate key points; A three-dimensional quadratic function is established, and the position and scale of candidate key points are calculated using the three-dimensional quadratic function to obtain the interpolation contrast. The interpolated contrast is compared with a preset contrast threshold to obtain stable key points; Based on the gradient direction histogram of stable key points calculated from the neighborhood pixels, oriented key points are obtained. Image feature points are obtained by calculating high-dimensional original gradient feature vectors based on directional key points and then performing dimensionality reduction.

[0008] In conjunction with the first aspect above, in one possible implementation, the feature point matching of high-response feature points using the FLANN algorithm includes: Construct a FLANN search index based on high-response feature points; Based on the FLANN search index, K-nearest neighbor search is performed on high-response feature points to obtain candidate matching points; The distance between high-response feature points and candidate matching points is calculated using a distance metric. The minimum and second-minimum distance values ​​are then divided to obtain the distance ratio. The distance ratio is compared with a preset ratio threshold. When the distance ratio is less than the ratio threshold, the candidate matching point corresponding to the minimum distance is marked as the final matching point; High-response feature points and final matching points are labeled as initial feature point pairs.

[0009] In conjunction with the first aspect above, in one possible implementation, the step of performing 3D reconstruction and georegistration of homonymous feature point pairs using the SFM algorithm includes: The camera pose estimation and 3D point triangulation of the same feature point pairs are performed by the SFM algorithm to generate sparse 3D point cloud and camera parameters. Geographic registration data was obtained by performing geographic coordinate registration on sparse 3D point clouds and camera parameters using joint bundle adjustment. The image-geographic coordinate mapping relationship is constructed based on the georegistration data.

[0010] In conjunction with the first aspect above, in one possible implementation, the step of performing camera pose estimation and 3D point triangulation on pairs of corresponding feature points using the SFM algorithm includes: The number of matching points and geometric quality indicators are calculated based on pairs of feature points with the same name, and then weighted and summed to obtain a comprehensive evaluation score. Select the corresponding feature point pairs and UAV image pairs with the maximum comprehensive evaluation score, and calculate the fundamental matrix and essential matrix to obtain the camera parameters; Triangulation of feature point pairs with the same name based on camera parameters yields a sparse 3D point cloud.

[0011] In conjunction with the first aspect above, in one possible implementation, the triangulation of corresponding feature point pairs based on camera parameters includes: Construct a multi-view camera projection matrix based on camera parameters; Construct an overdetermined linear equation system based on the projection matrix of a multi-view camera as the same feature point pair; The initial three-dimensional coordinates are obtained by solving the overdetermined linear equations using the singular value decomposition method. The initial 3D coordinates are optimized by maximum likelihood estimation to obtain a sparse 3D point cloud.

[0012] In conjunction with the first aspect above, in one possible implementation, the target identification and segmentation of the UAV image sequence using an image segmentation algorithm includes: The UAV image sequence is subjected to illumination normalization and contrast enhancement to obtain preprocessed images; The preprocessed image is input into the semantic segmentation model to obtain the target segmentation mask; wherein, the semantic segmentation model is the SAM2 model based on the Transformer architecture; Noise removal and hole filling are performed on the target segmentation mask to obtain a post-processed mask image; Calculate the geometric center point of the post-processed mask image, and mark the pixel coordinates corresponding to the geometric center point as the target pixel coordinates.

[0013] In conjunction with the first aspect above, in one possible implementation, the comparative analysis of the target's geographic coordinates includes: The target geographic coordinates are sorted in chronological order and outliers are removed to obtain the target coordinate sequence. Establish a unified coordinate system, transform the target coordinate sequence to the unified coordinate system, and mark the earliest time coordinate in the target coordinate sequence as the reference coordinate; The horizontal displacement vector and vertical settlement of the target coordinate sequence are calculated based on the reference coordinates to obtain the displacement and settlement of the target.

[0014] Secondly, a UAV target displacement and settlement monitoring system based on SFM is provided, including an image acquisition module, a three-dimensional reconstruction and mapping module, a target identification and positioning module, and a time-series coordinate analysis module. The image acquisition module is used to acquire multi-view images of the monitoring area using a drone to obtain a drone image sequence; The 3D reconstruction and mapping module is used to extract and match feature points in UAV image sequences using a feature extraction algorithm to obtain pairs of feature points with the same name; and to perform 3D reconstruction and georegistration on the pairs of feature points with the same name using the SFM algorithm to obtain the image-geographic coordinate mapping relationship. The target recognition and localization module is used to identify and segment the UAV image sequence using an image segmentation algorithm to obtain the target pixel coordinates; and to perform coordinate transformation on the target pixel coordinates based on the image-geographic coordinate mapping relationship to obtain the target geographic coordinates. The time-series coordinate analysis module is used to compare and analyze the geographic coordinates of the target to obtain the displacement and settlement of the target.

[0015] This application provides a method and system for monitoring the displacement and settlement of unmanned aerial vehicles (UAVs) targets based on SFM (Search Engine Function). Significant technological advancements have been achieved through the deep integration of SFM 3D reconstruction, georegistration, and deep learning target recognition technologies. First, by establishing a precise and stable mapping relationship between image pixel coordinates and geographic coordinates, sub-pixel-level 3D geographic coordinate reconstruction of individual targets directly from images is realized, overcoming the technical challenge of precise target positioning. Second, an automatic target recognition and segmentation mechanism based on an advanced semantic segmentation model is introduced, completely replacing inefficient and subjective manual interpretation, ensuring automation and consistency throughout the entire process. Finally, relying on the aforementioned automatic recognition and precise mapping capabilities, the system can automatically align and continuously track the 3D coordinates of the same target at different time phases, thereby achieving reliable and quantitative time-series analysis of horizontal displacement and vertical settlement. This solution significantly improves the accuracy, automation level, and engineering practical value of small-scale deformation monitoring, providing a superior technical means for slope safety monitoring.

[0016] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0017] Figure 1 A system architecture diagram of an UAV target displacement and settlement monitoring system based on SFM is provided for embodiments of this application; Figure 2 A flowchart illustrating a method for monitoring the displacement and settlement of a UAV target based on SFM, provided in an embodiment of this application; Figure 3 A flowchart illustrating another SFM-based method for monitoring the displacement and settlement of a UAV target, provided in an embodiment of this application; Figure 4 This is a flowchart illustrating another SFM-based method for monitoring the displacement and settlement of a UAV target, provided in an embodiment of this application. Detailed Implementation

[0018] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0019] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0020] The SFM-based UAV target displacement and settlement monitoring method provided in this application can be applied to, for example... Figure 1 In the SFM-based UAV target displacement and settlement monitoring system 100 shown, as Figure 1 As shown, the system includes: an image acquisition module 10, a 3D reconstruction and mapping module 20, a target recognition and localization module 30, and a time-series coordinate analysis module 40; Among them, the image acquisition module 10 is used to acquire multi-view images of the monitoring area through a drone to obtain a drone image sequence; The 3D reconstruction and mapping module 20 is used to extract and match feature points in UAV image sequences using feature extraction algorithms to obtain pairs of corresponding feature points; and to perform 3D reconstruction and georegistration on the pairs of corresponding feature points using the SFM algorithm to obtain the image-geographic coordinate mapping relationship. The target recognition and localization module 30 is used to perform target recognition and segmentation on the UAV image sequence through image segmentation algorithm to obtain target pixel coordinates; and to perform coordinate transformation on the target pixel coordinates based on the image-geographic coordinate mapping relationship to obtain the target geographic coordinates. The time-series coordinate analysis module 40 is used to compare and analyze the geographic coordinates of the target to obtain the displacement and settlement of the target.

[0021] To address the technical problem that existing technologies focus on macroscopic deformation analysis of the overall terrain, and cannot perform automated, high-precision, and traceable time-series three-dimensional coordinate monitoring of individual targets, this application provides a UAV target displacement and settlement monitoring method based on SFM. The method includes: By using drones to collect multi-view images of the monitored area, a drone image sequence is obtained; Feature points are extracted and matched from UAV image sequences using feature extraction algorithms to obtain pairs of feature points with the same name. The SFM algorithm is used to perform 3D reconstruction and georegistration on pairs of identical feature points to obtain the image-geographic coordinate mapping relationship. Target identification and segmentation of UAV image sequences are performed using image segmentation algorithms to obtain target pixel coordinates. Based on the image-geographic coordinate mapping relationship, the target pixel coordinates are transformed to obtain the target geographic coordinates; By comparing and analyzing the geographic coordinates of the target, the displacement and settlement of the target can be obtained.

[0022] Based on this, the technical problem of existing technologies focusing on macroscopic deformation analysis of the overall terrain, which cannot perform automated, high-precision, and traceable temporal three-dimensional coordinate monitoring of individual targets, has been solved.

[0023] like Figure 2 As shown in the embodiments of this application, the UAV target displacement and settlement monitoring method based on SFM includes: S201. The monitoring area is captured by a drone from multiple perspectives to obtain a drone image sequence.

[0024] It should be noted that, by using a visible light camera mounted on a drone, multi-view and multi-flight image acquisition is carried out on the slope area where artificial targets are set up, and a drone image sequence containing target information is obtained. The drone image sequence covers different positions and viewpoints of the same slope area to meet the requirements of image overlap for subsequent 3D reconstruction.

[0025] For example, a DJI Matrice M300 RTK multi-rotor drone is used, equipped with a Hasson L1D-20c visible light camera (20 million effective pixels). It performs automated flight at a relative altitude of about 100 meters above the monitoring area, with a preset forward overlap of 80% and a lateral overlap of 70%, thereby acquiring a high-definition orthophoto and oblique drone image sequence with a ground resolution of about 2.5 cm.

[0026] S202. Feature points are extracted and matched from the UAV image sequence using a feature extraction algorithm to obtain pairs of feature points with the same name.

[0027] In some implementations, the feature extraction algorithm is used to extract and match feature points in the UAV image sequence, such as... Figure 3 As shown, it includes: S21. Feature point detection is performed on the UAV image sequence using the DSP-SIFT algorithm to obtain image feature points; S22. Divide the UAV image sequence into multiple grid cells, and use a non-maximum suppression algorithm to suppress the image feature points in the grid cells to obtain high-response feature points; S23. Use the FLANN algorithm to perform feature point matching on high-response feature points to obtain initial feature point pairs; S24. The initial feature point pairs are removed by using the GMS algorithm to eliminate mismatches, resulting in feature point pairs with the same name.

[0028] It should be noted that the UAV imagery is divided into multiple regular grid cells of the same size. Within each grid cell, non-maximum suppression is applied to the image feature points, retaining only the top N feature points with the highest response values, where N is a positive integer. Alternatively, SIFT, SURF, ORB, or their improved algorithms can be used for feature point extraction, combined with brute-force matching, graph matching, or other approximate nearest neighbor matching methods to complete feature point matching.

[0029] For example, firstly, the DSP-SIFT algorithm is used to detect initial feature points from each image. Typically, about 10,000 to 15,000 feature points can be extracted from a single image. Then, each image is uniformly divided into grid cells of size 128 pixels by 128 pixels, and non-maximum suppression is applied within each cell, retaining only the top 50 feature points with the highest response values ​​as high-response feature points. Next, the FLANN algorithm is used to perform a fast approximate nearest neighbor search, and a ratio test method is used, setting the distance ratio threshold to 0.8 to select initial matching pairs. Finally, the GMS algorithm is applied to evaluate the motion consistency of matching point pairs in local regions, eliminating mismatches that obviously do not conform to geometric constraints, and finally obtaining a robust set of corresponding feature point pairs for each pair of overlapping images.

[0030] In this embodiment, by introducing a grid-based adaptive density control mechanism, the shortcomings of traditional methods, such as excessively dense feature points in textured areas and excessively sparse feature points in smooth areas, are overcome, ensuring that the subsequently generated 3D point cloud is uniformly distributed and providing consistent support accuracy for the entire monitoring area.

[0031] In some implementations, the feature point detection of the UAV image sequence using the DSP-SIFT algorithm includes: Constructing a Gaussian multi-scale space based on UAV image sequences; Extremum point detection is performed in the Gaussian multi-scale space by neighborhood comparison to obtain candidate key points; A three-dimensional quadratic function is established, and the position and scale of candidate key points are calculated using the three-dimensional quadratic function to obtain the interpolation contrast. The interpolated contrast is compared with a preset contrast threshold to obtain stable key points; Based on the gradient direction histogram of stable key points calculated from the neighborhood pixels, oriented key points are obtained. Image feature points are obtained by calculating high-dimensional original gradient feature vectors based on directional key points and then performing dimensionality reduction.

[0032] It should be noted that for the input UAV imagery, multiple octave scale spaces are constructed through Gaussian blurring and downsampling to simulate images at different observation scales; when the interpolation contrast is greater than the contrast threshold, the candidate keypoints corresponding to the interpolation contrast are marked as stable keypoints.

[0033] For example, firstly, a Gaussian multi-scale space containing four octaves is constructed for each UAV image, with each octave containing five scale levels. Different observation scales are simulated through continuous Gaussian blurring and downsampling. Next, at each scale, extreme points are detected by comparing with its 26 spatial neighbors and adjacent scale neighbors, generating candidate keypoints. Then, for each candidate keypoint, a three-dimensional quadratic function is locally fitted to its scale space to accurately interpolate its sub-pixel position and scale, and the interpolation contrast is calculated. Low-contrast points (typically sensitive to noise) with contrast below a preset threshold of 0.03 are removed, thus obtaining... Stable keypoints are identified. Subsequently, on the scale image where the keypoint is located, the gradient directions of the pixels in its neighborhood are statistically analyzed to form a 36-dimensional gradient direction histogram. The main peak direction (e.g., the direction exceeding 80% of the peak value is also retained) is assigned to the keypoint to complete the orientation. Finally, around each oriented keypoint, a 16x16 neighborhood region is divided in its scale space, and an 8-direction gradient histogram is calculated for each 4x4 sub-region to form a 128-dimensional original feature description vector. This vector is further reduced to a more discriminative 64-dimensional vector using principal component analysis (PCA) technology, and the final output is the image feature points used for subsequent matching.

[0034] In this embodiment, the DSP-SIFT feature extraction algorithm is used to extract image feature points that are scale- and rotation-invariant. Compared with conventional feature extraction methods in the prior art, this step improves the stability and robustness of feature point extraction under complex slope texture conditions.

[0035] In some implementations, the feature point matching of high-response feature points using the FLANN algorithm includes: Construct a FLANN search index based on high-response feature points; Based on the FLANN search index, K-nearest neighbor search is performed on high-response feature points to obtain candidate matching points; The distance between high-response feature points and candidate matching points is calculated using a distance metric. The minimum and second-minimum distance values ​​are then divided to obtain the distance ratio. The distance ratio is compared with a preset ratio threshold. When the distance ratio is less than the ratio threshold, the candidate matching point corresponding to the minimum distance is marked as the final matching point; High-response feature points and final matching points are labeled as initial feature point pairs.

[0036] For example, firstly, a fast search index structure based on a KD-Tree is constructed using a 64-dimensional descriptor set of high-response feature points. Then, for each feature point in the query image, the index is used to search for its K nearest neighbor candidate matching points in the reference image, where K is typically set to 2. Next, the Euclidean distance between the query point descriptor and these two candidate point descriptors is calculated to obtain the minimum and second-minimum distances, and their ratio is calculated. This distance ratio is compared with an empirical threshold, typically set to 0.8. If the distance ratio is less than this threshold, the match is considered to have sufficient discriminative power, and the candidate point corresponding to the minimum distance is accepted as the final correct matching point, and this point pair is marked as a reliable initial feature point pair. This method significantly improves the matching speed between massive feature points while ensuring matching accuracy, providing crucial geometric correspondences for subsequent 3D reconstruction.

[0037] S203. The SFM algorithm is used to perform 3D reconstruction and georegistration of the same feature point pairs to obtain the image-geographic coordinate mapping relationship.

[0038] In some implementations, the SFM algorithm is used to perform 3D reconstruction and georegistration of pairs of feature points with the same name, such as... Figure 4 As shown, it includes: S51. The camera pose estimation and 3D point triangulation of the same feature point pairs are performed by the SFM algorithm to generate sparse 3D point cloud and camera parameters. S52. Geographic coordinate registration of sparse 3D point cloud and camera parameters is performed by joint bundle adjustment to obtain geographic registration data; S53. Based on georegistration data, construct the image-geographic coordinate mapping relationship.

[0039] It should be noted that the 3D reconstruction can also generate dense point clouds through multi-view image matching (MVS) technology, or adopt other image-based 3D reconstruction methods.

[0040] For example, firstly, using the selected pairs of identical feature points, the high-precision camera exterior orientation elements of each image are estimated using the SFM algorithm, and the matching points are subjected to multi-view forward intersection to generate a sparse point cloud containing approximately 50,000 three-dimensional spatial points. Subsequently, four ground control points with known WGS84 coordinates are introduced, with a planar positioning accuracy better than 0.02 meters and an elevation accuracy better than 0.03 meters. The entire sparse point cloud and all camera parameters are optimized using constrained joint bundle adjustment to make the reconstructed model consistent with the real geographic coordinate system. Finally, based on the precise interior and exterior orientation elements of each image after adjustment optimization, a rigorous mapping relationship is constructed from the back projection of any pixel coordinate of the image to the three-dimensional geographic space. This relationship is usually stored in the form of a projection matrix or lookup table, providing a mathematical basis for subsequent target coordinate transformation.

[0041] In some implementations, the step of performing camera pose estimation and 3D point triangulation on pairs of corresponding feature points using the SFM algorithm includes: The number of matching points and geometric quality indicators are calculated based on pairs of feature points with the same name, and then weighted and summed to obtain a comprehensive evaluation score. Select the corresponding feature point pairs and UAV image pairs with the maximum comprehensive evaluation score, and calculate the fundamental matrix and essential matrix to obtain the camera parameters; Triangulation of feature point pairs with the same name based on camera parameters yields a sparse 3D point cloud.

[0042] It should be noted that the geometric quality indicators include relative pose, baseline length and disparity angle, and triangulation uncertainty; the relative pose is estimated based on the corresponding point pairs between the two current images by solving the essential matrix or the fundamental matrix; the triangulation uncertainty is based on the estimated relative pose, which can be used to triangulate the matching points and evaluate their uncertainty.

[0043] For example, firstly, all UAV image pairs with sufficient overlap are traversed. For each image pair, its comprehensive evaluation score is calculated using a weighted formula. For example, this score can be determined by the number of matching points (accounting for approximately 60% weight) and geometric quality indicators (such as the calculated baseline length and disparity angle, accounting for approximately 40% weight). Then, the image pair with the highest score is selected as the initialization seed, for example, a pair of images with more than 1500 high-quality matching points and a moderate baseline length. Its fundamental matrix is ​​calculated using the five-point method or the eight-point method, and further decomposed to obtain the initial two camera pose parameters. Finally, using this set of initial camera parameters, forward intersection based on linear triangulation is performed on all the corresponding matching points between the image pairs, successfully reconstructing thousands of three-dimensional spatial points, thus forming an initial sparse three-dimensional point cloud as a reliable basis for subsequent incremental reconstruction.

[0044] In some implementations, the triangulation of pairs of corresponding feature points based on camera parameters includes: Construct a multi-view camera projection matrix based on camera parameters; Construct an overdetermined linear equation system based on the projection matrix of a multi-view camera as the same feature point pair; The initial three-dimensional coordinates are obtained by solving the overdetermined linear equations using the singular value decomposition method. The initial 3D coordinates are optimized by maximum likelihood estimation to obtain a sparse 3D point cloud.

[0045] For example, firstly, based on the calibrated camera intrinsic parameters and the estimated exterior orientation elements, a specific 3x4 camera projection matrix is ​​constructed for each image participating in the reconstruction. For a set of corresponding feature points that are successfully matched on at least three images, a homogeneous linear equation system with six or more equations can be constructed using their corresponding pixel coordinates and their respective projection matrices. Subsequently, the overdetermined equation system is solved using the standard singular value decomposition method to obtain the initial 3D coordinates of the spatial point in the target coordinate system. Finally, this initial solution is used as the initial value for iteration and input into a maximum likelihood estimation optimizer based on the Levenberg-Marquardt algorithm. With the goal of minimizing the forward reprojection error, the 3D point coordinates and possible camera parameters are jointly refined after 10 to 15 iterations, and finally, a high-precision optimized 3D point coordinate as a component of the sparse 3D point cloud is output.

[0046] S204. Target recognition and segmentation are performed on the UAV image sequence using an image segmentation algorithm to obtain the target pixel coordinates.

[0047] In some implementations, the target identification and segmentation of the UAV image sequence using an image segmentation algorithm includes: The UAV image sequence is subjected to illumination normalization and contrast enhancement to obtain preprocessed images; The preprocessed image is input into the semantic segmentation model to obtain the target segmentation mask; wherein, the semantic segmentation model is the SAM2 model based on the Transformer architecture; Noise removal and hole filling are performed on the target segmentation mask to obtain a post-processed mask image; Calculate the geometric center point of the post-processed mask image, and mark the pixel coordinates corresponding to the geometric center point as the target pixel coordinates.

[0048] It should be noted that U-Net, DeepLab series, or other deep learning semantic segmentation models can also be used to automatically segment targets in images.

[0049] For example, firstly, histogram equalization and contrast-limited adaptive histogram equalization algorithms are applied to the input UAV imagery to perform illumination normalization and contrast enhancement, generating a preprocessed image. Then, the preprocessed image is input into a SAM2 semantic segmentation model based on the Transformer architecture and pre-trained on numerous visual tasks. This model outputs the probability that each pixel belongs to the "target" category, generating an initial binary segmentation mask by setting a probability threshold of 0.5. Next, morphological post-processing is performed on this initial mask, including opening operations using circular structuring elements with a radius of 3 pixels to remove minor noise and hole filling to obtain a connected and complete post-processed mask image. Finally, for each independent connected region in the mask image, the arithmetic mean of all its pixel coordinates is calculated. This two-dimensional coordinate point is taken as the geometric center of the target and recorded as the target pixel coordinates for subsequent coordinate transformations. For example, the center pixel coordinates of a typical circular target mask can be obtained after calculation (1245.3, 867.1).

[0050] In this embodiment, the SAM2 semantic segmentation model is used to process the target region in the image and automatically segment it to obtain the pixel-level segmentation result of each target, thereby avoiding the errors and inconsistencies caused by manual interpretation.

[0051] S205. Based on the image-geographic coordinate mapping relationship, the target pixel coordinates are transformed to obtain the target geographic coordinates.

[0052] For example, based on the precise interior and exterior orientation elements of each image obtained after bundle adjustment, the pixel coordinates of the target, such as column 1245.3 and row 867.1 on the image, are first converted into approximate coordinates in the local 3D space using the corresponding camera projection matrix. Then, combined with the precise exterior orientation elements of the image, the local spatial coordinates are transformed to a world coordinate system with a uniform scale and orientation using collinearity equations. Finally, the 3D coordinates in the world coordinate system are transformed to the target geographic coordinate system, such as CGCS2000 or WGS84, using a pre-determined seven-parameter transformation model. The final output is the geographic coordinates of the target, such as longitude 112.123456 degrees, latitude 23.456789 degrees, and elevation 125.34 meters. The plane transformation accuracy can reach a level better than 0.02 meters when high-precision control points are introduced.

[0053] S206. Compare and analyze the geographic coordinates of the target to obtain the displacement and settlement of the target.

[0054] In some implementations, the comparative analysis of the target's geographic coordinates includes: The target geographic coordinates are sorted in chronological order and outliers are removed to obtain the target coordinate sequence. Establish a unified coordinate system, transform the target coordinate sequence to the unified coordinate system, and mark the earliest time coordinate in the target coordinate sequence as the reference coordinate; The horizontal displacement vector and vertical settlement of the target coordinate sequence are calculated based on the reference coordinates to obtain the displacement and settlement of the target.

[0055] For example, firstly, the geographic coordinates of the same target acquired over multiple monitoring periods (e.g., 12 consecutive months) are sorted by acquisition time, and the 3σ criterion is used to remove abnormal observations that significantly deviate from the main cluster, forming a stable target coordinate sequence. Subsequently, the coordinates of all periods are unified to the same local engineering coordinate system using a known seven-parameter transformation model. Next, the coordinates of the earliest period in the sequence (e.g., January 2023) (e.g., north coordinates 3321546.123 meters, east coordinates 512345.678 meters, elevation 125.345 meters) are set as the reference coordinates for displacement calculation. Finally, the time-series deformation of the target is output by sequentially calculating the Euclidean distance between the coordinates of each subsequent period and the reference coordinates on the horizontal plane as the displacement, and calculating the elevation difference as the settlement. For example, the cumulative horizontal displacement measured up to December 2023 is 0.032 meters, and the cumulative settlement is -0.019 meters.

[0056] Based on the above technical solutions, the UAV target displacement and settlement monitoring method based on SFM provided in this application has achieved significant technical progress by deeply integrating SFM 3D reconstruction, georegistration, and deep learning target recognition technologies. First, by establishing a precise and stable mapping relationship between image pixel coordinates and geographic coordinates, sub-pixel-level 3D geographic coordinate reconstruction of a single target directly from the image is achieved, overcoming the technical challenge of precise target positioning. Second, an automatic target recognition and segmentation mechanism based on an advanced semantic segmentation model is introduced, completely replacing inefficient and subjective manual interpretation, ensuring automation and consistency of results throughout the entire process. Finally, relying on the aforementioned automatic recognition and precise mapping capabilities, the system can automatically align and continuously track the 3D coordinates of the same target at different time phases, thereby achieving reliable and quantitative time-series analysis of horizontal displacement and vertical settlement. This solution significantly improves the accuracy, automation level, and engineering practical value of small-scale deformation monitoring, providing a superior technical means for slope safety monitoring.

[0057] In one possible implementation, this application embodiment also provides a UAV target displacement and settlement monitoring system based on SFM, including an image acquisition module, a three-dimensional reconstruction and mapping module, a target identification and positioning module, and a time-series coordinate analysis module; The image acquisition module is used to acquire multi-view images of the monitoring area using a drone to obtain a drone image sequence; The 3D reconstruction and mapping module is used to extract and match feature points in UAV image sequences using a feature extraction algorithm to obtain pairs of feature points with the same name; and to perform 3D reconstruction and georegistration on the pairs of feature points with the same name using the SFM algorithm to obtain the image-geographic coordinate mapping relationship. The target recognition and localization module is used to identify and segment the UAV image sequence using an image segmentation algorithm to obtain the target pixel coordinates; and to perform coordinate transformation on the target pixel coordinates based on the image-geographic coordinate mapping relationship to obtain the target geographic coordinates. The time-series coordinate analysis module is used to compare and analyze the geographic coordinates of the target to obtain the displacement and settlement of the target.

[0058] For example, the image acquisition module uses a DJI M300 RTK drone and a Hasselblad L1D-20c camera to automatically acquire images at a flight altitude of 100 meters with a heading and lateral overlap of no less than 80% and 70%, respectively. The 3D reconstruction and mapping module integrates the OpenMVG library to perform DSP-SIFT feature extraction, FLANN matching, and incremental SFM reconstruction on the acquired image sequences. It also performs bundle adjustment using four ground control points with a planar accuracy better than 0.02 meters to establish a rigorous mapping relationship from pixels to the WGS84 coordinate system. The target recognition and localization module calls the SAM2 segmentation model based on PyTorch to automatically segment targets and extract center points from the images. It then converts the target pixel coordinates into 3D geographic coordinates through the aforementioned mapping relationship, achieving centimeter-level single-point localization accuracy. The time-series coordinate analysis module manages multi-period coordinates based on an SQLite database. Through coordinate alignment and differential calculation, it outputs the target's horizontal displacement and vertical settlement in millimeters, forming a complete automated monitoring result.

[0059] Based on the above technical solution, a high-precision traceable mapping relationship between image pixels and real geographic coordinates is established through the collaboration of image acquisition and 3D reconstruction mapping modules, fundamentally ensuring the spatial reliability of monitoring data sources. Combined with the deep learning automatic recognition capabilities of the target identification and positioning module, accurate extraction and coordinate transformation of specific monitoring targets are achieved, overcoming the subjectivity and inefficiency of manual interpretation. Finally, the time-series coordinate analysis module, relying on the standardized and alignable multi-period coordinate data provided in the aforementioned steps, can automatically and continuously quantify the displacement and settlement changes of targets. By integrating SFM reconstruction, semantic segmentation, and coordinate analysis technologies, this system significantly improves the automation level, accuracy, and long-term time-series analysis capabilities of slope and other engineering deformation monitoring.

[0060] Compared with existing technologies, this invention introduces a target-level processing mechanism and semantic segmentation technology into the UAV image 3D reconstruction process, achieving significant technical effects in terms of target displacement and settlement monitoring accuracy, automation level, and time series analysis capabilities. Specifically, these effects are reflected in the following aspects: (1) Target-level three-dimensional precise positioning was achieved, improving the spatial accuracy of displacement and settlement monitoring. This invention generates sparse 3D point clouds using the Structured Bundle Method (SFM) and triangulation reconstruction. After georegistration, it establishes a one-to-one correspondence between image pixel coordinates and latitude, longitude, and elevation coordinates, enabling any pixel location in the image to be mapped to its corresponding 3D geographic coordinates. Compared to existing methods that rely solely on comparative analysis of the overall point cloud or model, this invention directly obtains the 3D spatial coordinates of a single target center point, effectively avoiding interference from overall point cloud noise on local target displacement analysis and improving the accuracy of target displacement and settlement monitoring.

[0061] (2) Automatic target identification is achieved through semantic segmentation, which significantly improves the automation and consistency of the monitoring process.

[0062] This invention employs the SAM2 semantic segmentation model to automatically segment targets in UAV imagery, avoiding the subjective errors and inconsistencies inherent in existing technologies that rely on manual interpretation or simple threshold segmentation. By extracting the target segmentation region at the pixel level and calculating its center point, the method for acquiring the target position remains consistent across different monitoring phases, thereby improving the stability and repeatability of long-term continuous monitoring data.

[0063] (3) A stable mapping relationship from pixel coordinates to geographic coordinates was established, which enhanced the traceability of image measurement results.

[0064] This invention preserves the pixel coordinates of feature points in the original image during sparse point cloud reconstruction and georegistration, establishing a stable and clear correspondence between image pixels, 3D point cloud, and geographic coordinates. This technique enables image-based measurement results to have a clear spatial source and transformation path, overcoming the problem in existing technologies where image location and geographic coordinates are difficult to directly correlate, thus improving the reliability and traceability of measurement results.

[0065] (4) Supports time-series statistical analysis of target three-dimensional coordinates, suitable for long-term monitoring and trend judgment.

[0066] By uniformly managing and statistically analyzing the latitude, longitude, and elevation coordinates of the same target acquired at different time phases, this invention can directly calculate the horizontal displacement and elevation changes of the target over time, thereby achieving a quantitative assessment of target displacement and settlement. Compared to existing technologies that mostly remain at the qualitative or macroscopic analysis level, this invention can provide continuous and quantifiable time-series data support for engineering monitoring and safety assessment.

[0067] (5) Improved reliability and engineering applicability in small-scale displacement and settlement monitoring scenarios.

[0068] Because this invention uses the three-dimensional coordinate changes of the target center point as the monitoring object and ensures the consistency of each processing step through automated algorithms, it has higher stability and reliability in centimeter-level and smaller-scale displacement and settlement monitoring scenarios, and can better meet the accuracy and continuity requirements of engineering applications such as slope stability monitoring and geological disaster early warning.

[0069] In implementation, each step of the method provided in this embodiment can be completed by integrated logic circuits in the processor or by instructions in software form. The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0070] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0071] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0072] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A method for monitoring the displacement and settlement of a UAV target based on SFM, characterized in that, include: By using drones to collect multi-view images of the monitored area, a drone image sequence is obtained; Feature points are extracted and matched from UAV image sequences using feature extraction algorithms to obtain pairs of feature points with the same name. The SFM algorithm is used to perform 3D reconstruction and georegistration on pairs of identical feature points to obtain the image-geographic coordinate mapping relationship. Target identification and segmentation of UAV image sequences are performed using image segmentation algorithms to obtain target pixel coordinates. Based on the image-geographic coordinate mapping relationship, the target pixel coordinates are transformed to obtain the target geographic coordinates; By comparing and analyzing the geographic coordinates of the target, the displacement and settlement of the target can be obtained.

2. The method for monitoring the displacement and settlement of UAV targets based on SFM according to claim 1, characterized in that, The step of extracting and matching feature points from UAV image sequences using a feature extraction algorithm includes: Feature points are obtained by detecting feature points in UAV image sequences using the DSP-SIFT algorithm; The UAV image sequence is divided into multiple grid cells, and the image feature points in the grid cells are subjected to non-maximum suppression algorithm to obtain high response feature points. The FLANN algorithm is used to match high-response feature points to obtain initial feature point pairs. The GMS algorithm is used to remove mismatches from the initial feature point pairs to obtain feature point pairs with the same name.

3. The method for monitoring the displacement and settlement of UAV targets based on SFM according to claim 2, characterized in that, The feature point detection of the UAV image sequence using the DSP-SIFT algorithm includes: Constructing a Gaussian multi-scale space based on UAV image sequences; Extremum point detection is performed in the Gaussian multi-scale space by neighborhood comparison to obtain candidate key points; A three-dimensional quadratic function is established, and the position and scale of candidate key points are calculated using the three-dimensional quadratic function to obtain the interpolation contrast. The interpolated contrast is compared with a preset contrast threshold to obtain stable key points; Based on the gradient direction histogram of stable key points calculated from the neighborhood pixels, oriented key points are obtained. Image feature points are obtained by calculating high-dimensional original gradient feature vectors based on directional key points and then performing dimensionality reduction.

4. The method for monitoring the displacement and settlement of UAV targets based on SFM according to claim 2, characterized in that, The feature point matching of high-response feature points using the FLANN algorithm includes: Construct a FLANN search index based on high-response feature points; Based on the FLANN search index, K-nearest neighbor search is performed on high-response feature points to obtain candidate matching points; The distance between high-response feature points and candidate matching points is calculated using a distance metric. The minimum and second-minimum distance values ​​are then divided to obtain the distance ratio. The distance ratio is compared with a preset ratio threshold. When the distance ratio is less than the ratio threshold, the candidate matching point corresponding to the minimum distance is marked as the final matching point; High-response feature points and final matching points are labeled as initial feature point pairs.

5. The method for monitoring the displacement and settlement of UAV targets based on SFM according to claim 1, characterized in that, The step of performing 3D reconstruction and georegistration of identical feature point pairs using the SFM algorithm includes: The camera pose estimation and 3D point triangulation of the same feature point pairs are performed by the SFM algorithm to generate sparse 3D point cloud and camera parameters. Geographic registration data was obtained by performing geographic coordinate registration on sparse 3D point clouds and camera parameters using joint bundle adjustment. The image-geographic coordinate mapping relationship is constructed based on the georegistration data.

6. The method for monitoring the displacement and settlement of UAV targets based on SFM according to claim 5, characterized in that, The step of performing camera pose estimation and 3D point triangulation on pairs of identical feature points using the SFM algorithm includes: The number of matching points and geometric quality indicators are calculated based on pairs of feature points with the same name, and then weighted and summed to obtain a comprehensive evaluation score. Select the corresponding feature point pairs and UAV image pairs with the maximum comprehensive evaluation score, and calculate the fundamental matrix and essential matrix to obtain the camera parameters; Triangulation of feature point pairs with the same name based on camera parameters yields a sparse 3D point cloud.

7. The method for monitoring the displacement and settlement of UAV targets based on SFM according to claim 6, characterized in that, The triangulation of corresponding feature point pairs based on camera parameters includes: Construct a multi-view camera projection matrix based on camera parameters; Construct an overdetermined linear equation system based on the projection matrix of a multi-view camera as the same feature point pair; The initial three-dimensional coordinates are obtained by solving the overdetermined linear equations using the singular value decomposition method. The initial 3D coordinates are optimized by maximum likelihood estimation to obtain a sparse 3D point cloud.

8. The method for monitoring the displacement and settlement of UAV targets based on SFM according to claim 1, characterized in that, The process of target identification and segmentation of UAV image sequences using image segmentation algorithms includes: The UAV image sequence is subjected to illumination normalization and contrast enhancement to obtain preprocessed images; The preprocessed image is input into the semantic segmentation model to obtain the target segmentation mask; wherein, the semantic segmentation model is the SAM2 model based on the Transformer architecture; Noise removal and hole filling are performed on the target segmentation mask to obtain a post-processed mask image; Calculate the geometric center point of the post-processed mask image, and mark the pixel coordinates corresponding to the geometric center point as the target pixel coordinates.

9. The method for monitoring the displacement and settlement of UAV targets based on SFM according to claim 1, characterized in that, The comparative analysis of the target's geographic coordinates includes: The target geographic coordinates are sorted in chronological order and outliers are removed to obtain the target coordinate sequence. Establish a unified coordinate system, transform the target coordinate sequence to the unified coordinate system, and mark the earliest time coordinate in the target coordinate sequence as the reference coordinate; The horizontal displacement vector and vertical settlement of the target coordinate sequence are calculated based on the reference coordinates to obtain the displacement and settlement of the target.

10. A UAV target displacement and settlement monitoring system based on SFM according to claim 1, characterized in that, It includes an image acquisition module, a 3D reconstruction and mapping module, a target recognition and localization module, and a time-series coordinate analysis module; The image acquisition module is used to acquire multi-view images of the monitoring area using a drone to obtain a drone image sequence; The 3D reconstruction and mapping module is used to extract and match feature points in UAV image sequences using a feature extraction algorithm to obtain pairs of feature points with the same name; and to perform 3D reconstruction and georegistration on the pairs of feature points with the same name using the SFM algorithm to obtain the image-geographic coordinate mapping relationship. The target recognition and localization module is used to identify and segment the UAV image sequence using an image segmentation algorithm to obtain the target pixel coordinates; and to perform coordinate transformation on the target pixel coordinates based on the image-geographic coordinate mapping relationship to obtain the target geographic coordinates. The time-series coordinate analysis module is used to compare and analyze the geographic coordinates of the target to obtain the displacement and settlement of the target.

Images