LiDAR and unmanned aerial vehicle image fusion processing method applied to complex mountain land

Abstract

The invention relates to the technical field of geographic space information acquisition, and discloses a LiDAR and unmanned aerial vehicle image fusion processing method applied to a complex mountain land, which comprises the following steps: extracting stable optical flow feature points through an unmanned aerial vehicle image sequence to construct a dynamic skeleton, establishing a dynamic anchoring relation by combining LiDAR point cloud sparse three-dimensional feature points, and obtaining a dynamic optical flow image; and constructing a time-varying affine transformation model based on the local neighborhood feature points to realize LiDAR point cloud dynamic reprojection. According to the method, LiDAR point cloud local self-adaptive fusion is guided through image optical flow continuity, the problems of sensor dynamic errors and texture missing in a complex mountain environment are effectively solved, the geometric consistency of multi-source data in a weak texture area and under the dynamic interference condition is remarkably improved, and a high-reliability fusion data basis is provided for mountain three-dimensional perception.

Description

Technical Field

[0001] The present invention relates to a method for fusing LiDAR and UAV images applied to complex mountainous areas, and belongs to the technical field of geospatial information acquisition. Background Art

[0002] Current mainstream technologies usually rely on a high-precision positioning and orientation system (POS) to obtain the pose of the UAV, and based on this, globally rigid registration of LiDAR point clouds and images is achieved through iterative optimization algorithms (such as variants of ICP). Such methods have a certain effectiveness in static scenes, but in mountainous environments with frequent airflow disturbances and significant vegetation dynamics, their inherent limitations are becoming increasingly prominent: when the UAV performs monitoring tasks in complex mountainous areas, airflow turbulence causes high-frequency attitude jitter of the platform, and the local movement of the vegetation canopy caused by wind disturbance further destroys the spatio-temporal consistency of the images and point clouds.

[0003] Existing technologies attempt to bridge this dynamic difference using a globally rigid transformation model, but actually fall into the fundamental contradiction of fitting a dynamic scene with a static model. The core bottlenecks are as follows: 1. Low-precision POS data cannot accurately reflect the instantaneous pose changes of the sensors, and forced global registration leads to meter-level misalignments in local areas, such as forest edges and steep slopes, destroying the geometric authenticity of the fused data; 2. Weak-texture areas such as bare rock walls and snow-covered slopes lack stable image features, and the geometric features of point clouds are prone to spatial ambiguity in two-dimensional projections, significantly increasing the failure probability of traditional matching methods in such areas; 3. To compensate for the above defects, existing solutions either rely on high-precision POS hardware or introduce complex computational models, greatly increasing the data processing threshold and making it difficult to meet the lightweight operation requirements such as geological disaster emergency monitoring.

[0004] The industry has tried to alleviate the matching problem by improving feature descriptors or introducing deep learning models, but has not addressed the core contradiction of adaptability to dynamic environments, and instead increases the computational burden due to algorithm complexity. Therefore, how to construct a multi-source data fusion mechanism with inherent adaptability to mountain dynamic environments under low-cost sensor conditions and achieve high-confidence spatio-temporal alignment at the local scale has become a key technical challenge for improving the reliability of three-dimensional perception in complex scenes. Summary of the Invention

[0005] The present invention provides a method for fusing LiDAR and UAV images applied to complex mountainous areas, and its main purpose is to solve the problems of fusion misalignment and weak-texture area matching failure caused by insufficient self-adaptability of dynamic environments under low-cost sensor conditions.

[0006] To achieve the above object, a method for fusing LiDAR and UAV images applied to complex mountainous areas provided by the present invention includes the following steps: Step a: Obtain the continuous image sequence collected by the drone in the target area, and track the motion trajectories of recognizable feature points between consecutive image frames through the sparse optical flow algorithm; based on the dominant motion parameters between image frames, screen out the optical flow feature points that can be stably tracked in multiple consecutive frames and whose local motion vectors deviate from the motion trend characterized by the dominant motion parameters by less than a preset deviation threshold to form an image dynamic skeleton; Step b: Obtain the LiDAR point cloud data collected in the same period as the image sequence and the data of the low-precision positioning and orientation system for preliminary positioning; use the data of the low-precision positioning and orientation system to georeference the LiDAR point cloud data, and perform voxel downsampling on the LiDAR point cloud data to extract sparse three-dimensional feature points based on the change rate of the normal vector or the local elevation extreme value to form a LiDAR approximate skeleton; Step c: Based on the camera pose provided by the low-precision positioning and orientation system data, back-project the sparse three-dimensional feature points in the LiDAR approximate skeleton onto the image frames in the image sequence to form the projection area of the sparse three-dimensional feature points on the image; for the projection area, search for the optical flow feature point sequence with the shortest distance from the sparse three-dimensional feature points in the three-dimensional space and whose optical flow motion trajectory stability meets the preset stability standard, and establish a dynamic anchoring relationship between the sparse three-dimensional feature points and the optical flow feature point sequence; Step d: Based on the dynamic anchoring relationship, for the sparse three-dimensional feature points and their anchored optical flow feature point sequences, combine three to five pairs of the nearest neighbor feature point pairs with established dynamic anchoring relationships around them to construct a locally affine transformation model that changes with time; use the locally affine transformation model to dynamically reproject the original point cloud corresponding to the neighborhood of the sparse three-dimensional feature points in the LiDAR point cloud data onto the image frames corresponding to the dynamic anchoring relationship to realize the dynamic fusion of the LiDAR point cloud data and the drone images.

[0007] Preferably, the method for screening optical flow feature points in step a is as follows: after tracking the motion trajectories of recognizable feature points, estimate the dominant inter-frame motion parameters caused by the self-motion of the drone between consecutive image frames; for each tracked feature point, calculate the vector difference between its local motion vector and the motion vector expected at the position of this feature point based on the dominant inter-frame motion parameters; when the magnitude is greater than the preset deviation threshold then identify this feature point as a motion inconsistent point and exclude it when screening stable optical flow feature points or establishing a dynamic anchoring relationship.

[0008] Preferably, the method for tracking the motion trajectories of recognizable feature points in step a is to use the KLT optical flow algorithm.

[0009] Preferably, before step a, it further includes: based on the LiDAR point cloud data, performing three-dimensional geometric structure saliency analysis on each point or its local neighborhood in the LiDAR point cloud data, calculating the normal vector change rate, curvature or elevation standard deviation of the point cloud; and identifying the geometrically significant regions according to the condition that the three-dimensional geometric structure saliency metric value is higher than a specific threshold.

[0010] Preferably, the geometrically significant regions are back-projected onto the image frames in the continuous image sequence using the low-precision positioning and orientation system data to calibrate the corresponding virtual anchor potential regions within the image frames; and when performing step a to extract the optical flow feature points or step c to establish the dynamic anchoring relationship, for the image regions that fall within the virtual anchor potential regions, referring to the local three-dimensional geometric structure saliency information of the LiDAR point cloud to guide or enhance the extraction of the optical flow feature points or the establishment of the dynamic anchoring relationship.

[0011] Preferably, the method for establishing the dynamic anchoring relationship in step c is: for each sparse three-dimensional feature point, within the projection region where it is back-projected onto the image frame, search for and select the optical flow feature point that has the shortest distance from it in three-dimensional space and whose optical flow motion trajectory satisfies the preset stability criterion between consecutive frames to form the dynamic anchoring relationship.

[0012] Preferably, the continuous image sequence is a video stream of five to ten frames per second or high-frequency burst photos.

[0013] Preferably, the method for extracting sparse three-dimensional feature points from the LiDAR point cloud data includes normal vector mutation detection or local elevation extreme value extraction based on the LiDAR point cloud.

[0014] Preferably, after step d, it further includes mapping the elevation or intensity information carried by the dynamically re-projected LiDAR point cloud data to the matching pixels on the corresponding image frame; or overlaying the spectral or texture information of the image frame onto its corresponding original three-dimensional point cloud.

[0015] Preferably, in step b, the voxel downsampling is performed based on the determined voxel size to ensure the stability and repeatability of the sparsification degree.

[0016] Compared with the problems in the background art, the beneficial effects of the present invention are: 1. By constructing a dynamic skeleton using the continuity of the optical flow trajectories in the image sequence, replacing the traditional rigid registration mode that relies on a high-precision positioning system, this mechanism enables the reprojection process of the LiDAR point cloud to adaptively follow the true motion trend of the ground surface on the imaging plane, effectively absorbing the high-frequency disturbances of the UAV platform in complex airflows. Thus, the system can still maintain the geometric-spectral consistency of the local area under the condition of low-cost sensors, avoiding the fusion misalignment caused by the mismatch between the global model and local dynamics in traditional methods. At the same time, the image feature points and LiDAR sparse three-dimensional feature points selected based on the stability of the optical flow motion form a spatio-temporal coupling through a dynamic anchoring relationship. This relationship further combines the spatio-temporal distribution of local neighborhood feature point pairs to construct an affine transformation model that evolves over time, making the local reprojection of the LiDAR point cloud always synchronized with the image dynamics. This closed-loop logic of extracting a benchmark from the dynamics and guiding the fusion with the benchmark significantly weakens the feature matching ambiguity in the mountain texture repetition area, providing a geometric and semantic consistent data basis for subsequent intelligent analysis.

[0017] 2. Reverse-map the geometric significance features such as the sudden change of the normal vector and the elevation extreme value of the LiDAR point cloud to the image space to form a potential area of virtual anchor points. When the texture features of the image itself are insufficient, this area activates the generation ability of the optical flow trajectory by reducing the feature extraction threshold or implanting geometrically guided virtual tracking points. This operation converts the three-dimensional structure information into a guiding signal for two-dimensional interpretation, avoiding the fusion blind spots of traditional methods in weak texture areas such as snow and shadows, and ensuring the continuity of the full-scene data. In addition, introduce the dominant motion parameters as the reference benchmark for screening the optical flow feature points. Through the difference analysis between the local motion vector and the dominant trend, actively identify and exclude the interference trajectories of independent moving objects such as birds and floating objects in the scene. This mechanism and the dynamic anchoring relationship form a collaborative defense: the former purifies the input dynamic skeleton, and the latter ensures that the anchoring relationship is only associated with the real ground surface motion. The two replace complex object recognition models with extremely simple vector operations, enabling the system to achieve anti-interference fusion in the edge computing unit.

[0018] 3. Based on the elevation-spectral information interaction of the local dynamic anchoring relationship, the geometric attributes of the LiDAR point cloud and the texture attributes of the image are naturally fused in the spatio-temporal alignment state. This process does not require global point cloud registration or pixel-level alignment, but through the local reprojection guided by sparse feature points, realizes the accurate mapping of three-dimensional geometric details to two-dimensional images, and the semantic attachment of two-dimensional spectral information to three-dimensional point clouds, forming a geospatial information expression with bidirectional enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flowchart of the multi-source data fusion process of the present invention; Figure 2 is a comparison chart of the anchoring success rates of the dynamic anchoring relationship construction under various scenarios of the present invention; Figure 3 This is the timing diagram of the dynamic fusion of LiDAR and UAV images with multi-module collaboration in the present invention.

[0020] The implementation, functional features and advantages of the present invention will be further described with reference to the embodiments and the accompanying drawings. Detailed implementation manners

[0021] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0022] The present invention proposes a method for fusing LiDAR and UAV images applied to complex mountainous areas, avoiding the limitation of the traditional rigid registration model's dependence on high-precision positioning. By constructing a dynamic anchoring relationship and a local time-varying affine model, high-precision dynamic data fusion under low-cost sensor conditions is achieved; in the specific application process, for concepts such as the geometrically significant region, dynamic anchoring relationship, and dominant inter-frame motion parameters in the present invention, the system combines a large amount of field test experience in typical mountainous scenarios, and based on the observed point cloud structure change characteristics, the dominant motion trend between image frames, and the stability performance of optical flow trajectories, a set of robust and reusable determination strategies are formed. For example, the identification of the geometrically significant region essentially relies on the phenomenon of sudden height difference or rapid direction change presented by the point cloud within a local range, which often appears in areas such as rock wall turns, vegetation edges, or mountain fracture zones, showing stable spatial structure identification characteristics; the selection of the dominant inter-frame motion parameters is based on the main displacement component presented in the overall motion trend of the image, which can be naturally summarized through the aggregation analysis of the overall motion field of consecutive image frames; as for the establishment of the dynamic anchoring relationship, it mainly depends on the continuity and motion consistency of the point cloud features in the corresponding regions of the image, and the system is determined based on the dual constraints of a relatively strict optical flow trajectory stability criterion and spatial proximity.

[0023] The entire solution first based on the image sequence collected by the UAV continuously flying in the target area, uses the sparse optical flow algorithm to track the recognizable feature points between frames, and extracts the optical flow feature points that remain stable and have the same motion trend between multiple frames to construct an image dynamic skeleton reflecting the true motion trend of the ground surface. Here, the so-called stability means that the modulus of the deviation vector between the motion vector direction of the feature points in consecutive frames and the overall dominant motion direction of the image is less than the set threshold , usually set between 0.3 and 0.5 pixel frame - to - frame movement, which can be moderately adjusted according to the flight speed of the drone and the sampling frequency. The acquisition method of the dominant motion parameter is to perform principal component analysis on the full - frame motion vector field and extract the direction of the maximum eigenvalue as the overall dominant motion direction. Synchronous with the image acquisition, the acquisition of LiDAR point cloud data is also carried out, supplemented by the initial pose information provided by a low - precision POS (Position and Orientation System). In the original point cloud, the present invention does not directly use the full amount of data, but introduces a sparsification mechanism based on local geometric significance indicators: after voxel grid down - sampling, the rate of change of the normal vector of each point or its neighborhood, that is, the rotation distance on the unit sphere, local elevation extreme value, or point cloud curvature, is calculated respectively. If the above - mentioned geometric significance indicators are higher than the set threshold, such as local elevation difference or curvature , then it is marked as a sparse three - dimensional feature point to form a LiDAR approximate skeleton. This strategy can effectively enhance the distinguishability of the projection area of three - dimensional features on the image and reduce the mis - matching in the blurred area.

[0024] Next, based on the rough pose information provided in the POS data, the sparse three - dimensional feature points are back - projected onto the image frame through internal and external parameters. For each projection area, search for the optical flow feature point that is the closest in spatial distance to the original three - dimensional feature point within the area and whose optical flow trajectory meets the requirements of continuity and stability, and establish a one - to - one dynamic anchoring relationship between them. The stability criteria of this relationship include that during continuous tracking of the optical flow point for more than 5 frames, the variance of the inter - frame motion vector , and the deviation of the direction angle from the dominant direction does not exceed 10°. This anchoring mechanism ensures that the selected optical flow points truly reflect the continuous movement of surface physical entities, rather than independent disturbing objects such as birds or floating branches. Once the anchoring relationship is established, the system uses this relationship as a benchmark and combines 3 - 5 groups of known stable anchor point pairs selected within the neighborhood of each pair of anchor points to jointly participate in constructing a local affine transformation model , which takes time as a variable to capture the local deformation trend of the fusion area between image frames. Specifically, the affine model is fitted by the least - squares method in the following form: , where is the projection of the original point cloud coordinates, is the corresponding position in the target image frame, is the 2×2 local linear transformation matrix, is the translation vector. By dynamically updating this model, the precise matching of the point cloud to the corresponding image position at different times can be achieved.

[0025] After that, all the original points in the original point cloud that belong to the neighborhood of the sparse three - dimensional feature points (usually within a radius of 1.5 - 3m) are dynamically re - projected, that is, based on the above The model adjusts its position in the imaging plane to complete the temporal synchronous fusion of the point cloud and the image. Different from the traditional method of performing rigid transformation on the entire point cloud, the present invention relies on a local affine model to achieve small-range non-rigid geometric correction, which is especially suitable for the cases where the vegetation canopy has dynamic undulations or the platform attitude has frequent micro-perturbations in mountainous scenes. At the same time, for weak texture regions, such as snow slopes and rock wall shadows, due to the scarcity of optical flow points, the present invention further introduces a LiDAR point cloud geometric saliency back-projection mechanism, projects the high-saliency three-dimensional structure in the image to form a potential area of virtual anchor points, and combines image histogram equalization and gradient enhancement processing, supplemented by a feature point extraction strategy with a reduced threshold, to artificially activate the ability to generate optical flow points, so that the texture-sparse area can still effectively participate in dynamic anchoring. After the core fusion is completed, to achieve further unification of multi-source data at the semantic level, the present invention supports a two-way information projection mechanism: on the one hand, map the elevation (z value) or intensity value of the re-projected point cloud to the corresponding pixels of the image to construct a digital surface model (DSM) layer; on the other hand, the texture information of the image (such as RGB values, edge features, texture directionality, etc.) can also be superimposed on the corresponding three-dimensional point cloud for subsequent surface semantic segmentation or intelligent recognition. In addition, to improve the stability and repeatability of data processing, the present invention adopts a fixed grid resolution (such as 0.5m × 0.5m × 0.5m) in the voxel down-sampling stage to avoid the drift of feature points caused by fluctuations in point density. At the same time, all parameter selections are set in combination with the test results of the actual mountain scene to ensure wide adaptability in different seasons and different geomorphic types. At the same time, to ensure the spatial accuracy and robustness of the dynamic anchoring relationship in the multi-source data fusion process, after establishing a one-to-one correspondence between sparse three-dimensional feature points and optical flow feature points, the system further introduces an anchoring offset statistical mechanism, evaluates the mean and range of the projection trajectory offsets of the already matched point pairs in multiple consecutive image frames. If the maximum projection offset exceeds the reasonable image movement range set by the system, the anchoring pair is automatically removed, and a candidate feature point re-selection strategy for adjacent regions is triggered to ensure that the geometric constraint accuracy of the anchoring relationship is not affected by instantaneous abnormal perturbations.In addition, for cases where it is difficult to extract optical flow feature points or establish anchoring relationships fails in weak texture regions, the system provides a regional-based local fallback mechanism. That is, based on the anchoring trend in the same region of historical frames or the established adjacent stable anchoring relationships, interpolation is used to infer the motion state of this region, and the system automatically generates temporary auxiliary anchor points as supplements to avoid local fusion interruption caused by individual anchoring missing, ensuring the continuity and stability of the overall fusion process. And during the construction of time-varying affine transformation, to improve its adaptability under multi-angle observation conditions in mountainous scenes, the system adopts a sliding modeling mechanism relying on time series, ensuring that the model is driven by multiple pairs of anchor points with good spatial distribution characteristics at each stage. This modeling method can dynamically adapt to the change of the sensor's perspective, and through frame-by-frame update, ensure the smooth transition of affine parameters in the time series dimension, avoiding instantaneous geometric distortion caused by single-frame perturbation. In the practical process, if the fitting residuals in multiple consecutive image frames exceed the set range, the system will trigger the anchor point re-selection process, and preferentially select new anchor point pairs with uniform distribution and relatively stability for model reconstruction, thereby further improving the convergence stability of the local affine model and the overall fusion consistency; and the deviation threshold in this embodiment. It is used to judge the consistency between the local motion trend of feature points and the dominant inter-frame motion trend of images, and belongs to the core parameter of the stable optical flow feature point screening mechanism. Specifically, when the degree of deviation of the motion trajectory of feature points between consecutive image frames from the dominant motion direction exceeds this threshold, that is, the inter-frame displacement deviation corresponding to the included angle difference is greater than , the system will identify it as a point with inconsistent motion trend and eliminate it. In typical experiments, is set to 0.4 pixels, determined based on the principal component analysis results of the full-frame dominant motion vector field and the statistical experience of the continuous tracking deviation distribution of feature points. This threshold takes into account both the strictness of motion trend consistency and the sufficiency of the number of feature points. Under different geomorphic types or flight parameters, can be appropriately adjusted according to the actual data performance to ensure that the extracted optical flow feature points are both stable and spatially representative, which all belong to the extended implementation methods known to those of ordinary skill in the art.

[0026] Embodiment 1: In a complex mountain environment, to achieve high-precision dynamic fusion of LiDAR point cloud and UAV image data under low-cost sensor conditions, this embodiment details a specific implementation method of fusion processing. Its core mechanism includes constructing a dynamic skeleton through stable optical flow features in the image, establishing a dynamic anchoring relationship by combining sparse three-dimensional feature points, and realizing the reprojection of the point cloud based on a time-varying local affine model, thereby forming multi-source fusion data with highly consistent geometry and spectrum. First, deploy a UAV platform with continuous image acquisition capabilities in the specified target area. This platform acquires image sequences at a frequency of about six frames per second to ensure sufficient temporal resolution to capture high-frequency dynamic changes. Through a sparse optical flow algorithm based on corner detection, the recognizable feature points in consecutive image frames are tracked frame by frame, and the overall dominant motion direction is extracted using the principal component analysis of the full-frame motion vector field. Subsequently, calculate the angular difference between the motion vector of each feature point and the dominant direction, and filter out the stable feature points with an inter-frame displacement deviation less than 0.5 pixels for constructing the dynamic skeleton. Secondly, synchronously acquire LiDAR point cloud data corresponding to the image sequence period, and introduce the pose information provided by a low-precision positioning and orientation system for preliminary georeferencing. To improve the efficiency and accuracy of fusion processing, first perform voxel downsampling on the original point cloud, and select points with significant normal vector change rates or local elevation extremes in each voxel to construct a sparse three-dimensional feature point set. Then, back-project this feature point set to the corresponding image frame through the internal and external parameters of the camera. In each projection area, search for optical flow feature points and establish a one-to-one dynamic anchoring relationship with the three-dimensional feature points based on their stability shown in continuous tracking for more than five frames.

[0027] Subsequently, three to five groups of confirmed stable anchor point pairs are selected in the neighborhood of each pair of dynamic anchor points to jointly construct a local time-varying affine model. The model is dynamically generated by least squares fitting to reflect the spatial transformation characteristics of the point cloud neighborhood between different image frames. Based on the established affine model, the original point cloud data with a radius of about two meters in the neighborhood of the sparse three-dimensional feature points is dynamically reprojected to the corresponding image frame to achieve local non-rigid matching; for problem areas with texture loss (such as snow-covered areas, shadow areas or rock walls), since traditional optical flow algorithms are difficult to generate stable feature trajectories, this embodiment back-projects the geometrically significant areas of the LiDAR point cloud at the initial stage of construction, and performs local histogram equalization and gradient enhancement on the corresponding image areas to improve feature extraction. The feasibility of taking is ensured. At the same time, the feature detection threshold is appropriately lowered in such areas to increase the probability of generating feature points and ensure that they can participate in the subsequent dynamic anchoring process. After the fusion is completed, the elevation or reflection intensity information carried by the reprojected point cloud is mapped to the image frame to construct an elevation layer or a multispectral fusion layer. The texture information of the image can also be mapped to the corresponding three-dimensional points to achieve semantic enhancement expression of the point cloud. In all processing links, the parameter settings used, including voxel size, stability judgment standard, affine neighborhood radius, etc., can be optimized based on the mountain real-life test data in historical data to ensure that the method has good versatility and operational robustness under different seasons and different landform types, which are all extended implementation methods known to ordinary technicians in this field.

[0028] Embodiment 2: For example, an area with typical complex mountain features in the southwest region is used as a test site. The area has steep terrain, dense vegetation, and weak texture areas such as exposed rock walls and seasonal snow cover. The area is affected by valley winds all year round and has frequent air disturbances. During the test, a customized and improved drone-mounted LiDAR-imaging integration system was used. The drone platform uses a six-rotor drone with high load capacity, and its flight control system is optimized to enhance attitude anti-disturbance performance. The LiDAR sensor uses a laser radar with medium range and high-frequency scanning capabilities (ranging accuracy can reach plus or minus 2 cm), and the image sensor uses an industrial The high-resolution digital camera is of commercial grade, and its high-frequency continuous shooting capability can reach 10 frames per second to ensure the acquisition of continuous image sequences with high correlation between frames. When acquiring data, the UAV strictly follows the preset route and flies at a speed of about 6 meters per second, while recording LiDAR point cloud data, continuous image sequences, and low-precision positioning and orientation system data for preliminary geographic reference. The low-precision positioning and orientation system provides horizontal positioning accuracy within the range of about 1 meter to 3 meters and attitude accuracy of about 0.5 degrees, which simulates the typical performance of the system under low-cost sensor conditions. The image acquisition frequency is set to 7 frames per second. The selection of this frequency is intended to balance the time resolution and the efficiency of subsequent data processing.

[0029] Imaging dynamic skeleton construction experiment: The purpose of this stage of the experiment is to verify the technical effectiveness of accurately extracting stable optical flow feature points from a continuous image sequence through the sparse optical flow algorithm and constructing an imaging dynamic skeleton. The experimental method is to use the KLT (Kanade-Lucas-Tomasi) optical flow algorithm to track the motion trajectories of recognizable feature points between consecutive image frames. To evaluate the guiding role of dominant motion parameters in the screening of stable optical flow feature points, this experiment designed two screening strategies for comparative evaluation: Strategy 1 (control group), this strategy only relies on the continuity of optical flow tracking and the stability of feature points in multiple consecutive frames, that is, the proportion of successfully tracked frames must be higher than 90% for screening, without considering the dominant motion parameters; Strategy 2 (the strategy of the present invention), on the basis of Strategy 1, the dominant inter-frame motion parameters caused by the movement of the UAV itself between consecutive image frames are estimated. The way to obtain this parameter is to perform principal component analysis on the full-frame motion vector field and determine the direction of the maximum eigenvalue as the overall dominant motion direction. Subsequently, calculate the vector difference between the local motion vector of each tracked feature point and the expected motion vector at the position of this feature point based on this dominant motion parameter. The preset deviation threshold is set to 0.4 pixels. When the magnitude of the vector difference is greater than this threshold, this feature point is identified as a motion inconsistent point and excluded; Experimental results and analysis: In an image sequence covering typical airflow disturbances and vegetation swaying, 1000 initial feature points were randomly selected for tracking and screening, and the results are shown in the following table:

[0030] Although Strategy 1 obtained a relatively large number of stable tracking points, the proportion of interference points of independent moving objects among them was significantly high, reaching 12.3%. The motion trajectories of these interference points, such as birds, floating objects, or violently swaying branches, are different from the dominant motion trend of the UAV platform. If such points are incorporated into the dynamic skeleton, it will inevitably introduce deviations, thus affecting the subsequent fusion accuracy; In contrast, after the strict screening of the dominant motion parameters in Strategy 2 (the strategy of the present invention), although the number of stable optical flow feature points decreased slightly, the proportion of interference points of independent moving objects among them dropped significantly to 1.8%, and the consistency between the motion trend of the dynamic skeleton points and the dominant trend increased significantly to 0.95. This result clearly shows that the present invention effectively filters the interference of independent moving objects in the scene by introducing the dominant motion parameter as a reference benchmark, ensuring that the constructed imaging dynamic skeleton can truly and accurately reflect the overall motion trend of the ground surface on the imaging plane. This mechanism provides a highly reliable motion reference for the subsequent dynamic reprojection of LiDAR point clouds, fundamentally avoiding the registration error introduced by independent moving objects in the scene.

[0031] Experiment on Extracting Sparse 3D Feature Points from LiDAR Point Cloud and Constructing Dynamic Anchoring Relationship: The purpose of this stage of experiment is to verify the extraction effect of sparse 3D feature points from LiDAR point cloud and, on this basis, the effectiveness of establishing a dynamic anchoring relationship with image optical flow feature points, especially focusing on the anchoring ability in weak texture areas; The experimental method is as follows: Extraction of sparse 3D feature points from LiDAR point cloud: First, the original LiDAR point cloud is subjected to voxel downsampling, and the voxel size is set to 0.5 meters by 0.5 meters by 0.5 meters. Subsequently, the change rate of the normal vector or the local elevation extreme value of the points or their local neighborhoods in each voxel is calculated. The change rate of the normal vector is defined here as the rotation distance between adjacent normal vectors on the unit sphere. In this experiment, the threshold of the change rate of the normal vector is set to be greater than 0.05 radians, or the local elevation difference (defined as the difference between the maximum elevation value and the minimum elevation value of the point cloud in the voxel) is greater than 1.5 meters. The points that meet either condition are marked as sparse 3D feature points, thus forming a LiDAR approximate skeleton; Construction of dynamic anchoring relationship: Using the camera rough pose information provided by the low-precision positioning and orientation system data, the sparse 3D feature points in the LiDAR approximate skeleton are back-projected onto the corresponding image frames, forming the projection area of the sparse 3D feature points on the image. For each projection area, the optical flow feature points that are the shortest in distance from the sparse 3D feature points in 3D space and whose optical flow motion trajectory stability meets the preset criteria are searched. The optical flow motion trajectory stability criteria include that in the tracking of more than 5 consecutive frames of the optical flow points, the variance of the inter-frame motion vector needs to be less than 0.02 pixel squares, and the deviation of the motion direction angle from the dominant direction does not exceed 15 degrees. A dynamic anchoring relationship can be established between the optical flow feature points that meet the above conditions and the sparse 3D feature points. To evaluate the adaptability of the present invention in weak texture areas, the experiment further introduces the LiDAR point cloud geometric significance back-projection mechanism as an auxiliary means. For weak texture areas such as snow-covered areas and bare rock walls, the geometric significance area of the LiDAR point cloud is back-projected onto the image frame, forming a virtual anchor point potential area. In this potential area, the detection threshold of the optical flow feature points is appropriately reduced, and local image histogram equalization and gradient enhancement processing are performed to activate the generation ability of the optical flow feature points; Experimental results and analysis: In multiple typical mountain scenes including bare rock walls (weak texture), vegetation canopies (rich texture), and snow slopes (weak texture), 200 LiDAR sparse 3D feature points were randomly selected for anchoring experiments, and the results are shown in the following table:

[0032] In texture-rich areas such as vegetation canopies, the success rate of dynamic anchoring is as high as 98.5%, and the average anchoring error remains below 1 pixel. This indicates that the method of the present invention can achieve high-precision matching under conditions of sufficient texture features. In weak-texture areas such as bare rock walls and snow slopes, the initial anchoring success rates without LiDAR geometric guidance are 65.2% and 58.7% respectively, and the average anchoring error increases, which reflects the inherent limitations faced by traditional methods in weak-texture areas due to sparse or ambiguous feature points. However, after introducing the LiDAR point cloud geometric saliency back-projection mechanism and combining image processing optimization, the anchoring success rates of bare rock walls and snow slopes have been significantly improved by 28.5 and 35.1 percentage points respectively, reaching high levels of 93.7% and 93.8%. This result clearly demonstrates that by converting the three-dimensional structure information of LiDAR into a guiding signal in the two-dimensional image space, the present invention can effectively make up for the lack of image features in weak-texture areas, significantly improve the dynamic anchoring ability in these challenging scenarios, and ensure the continuity and reliability of full-scene data fusion.

[0033] LiDAR Point Cloud Dynamic Reprojection and Fusion Geometric Consistency Evaluation: The purpose of this stage of the experiment is to evaluate the effectiveness of the present invention in realizing the dynamic reprojection of LiDAR point clouds based on the local time-varying affine transformation model and ultimately improving the geometric consistency of multi-source data. The experimental method is that after successfully establishing the dynamic anchoring relationship, for each sparse three-dimensional feature point and its sequence of anchored optical flow feature points, combined with three to five pairs of nearest-neighbor feature point pairs with established dynamic anchoring relationships around it, a local affine transformation model that changes with time is constructed. This model is fitted using the least squares method and realizes the precise matching of the point cloud to the corresponding image position at different times through dynamic updates. Subsequently, all the original point clouds in the original LiDAR point cloud corresponding to the neighborhood of the sparse three-dimensional feature point (the neighborhood radius is set to 2 meters) are dynamically reprojected onto the image frame corresponding to the dynamic anchoring relationship, thus completing the dynamic fusion of the LiDAR point cloud and the UAV image. To quantitatively evaluate the fusion effect, the following indicators are selected in this experiment: Geometric Consistency Error of Fusion Data (Root Mean Square Error RMSE). Randomly select the fused LiDAR point cloud data and the homologous points that can be clearly identified in the image, calculate the projection position deviation on the two-dimensional image, and perform root mean square error statistics in pixels. The comparison strategy uses the global rigid registration method based on low-precision positioning and orientation system data as the control group to evaluate its geometric consistency performance in dynamic mountain scenarios. Experimental Results and Analysis: In a complex mountain scenario containing strong airflow disturbances, vegetation dynamics, and weak-texture areas, multiple repeated experiments were carried out on the two fusion methods, and the average geometric consistency error was statistically analyzed. The results are shown in the following table:

[0034] Significant improvement in overall geometric consistency: The root mean square error of the overall geometric consistency of the control group (global rigid registration method) was as high as 2.87 pixels, and even reached 4.51 pixels in weakly textured areas, which confirmed the inherent limitations of relying solely on a low-precision positioning and orientation system for global rigid registration in a complex mountain dynamic environment. Due to the high-frequency jitter and local dynamic deformation of the UAV platform, the forced global model is difficult to accurately fit the real motion of the local area, resulting in obvious misalignment. In contrast, the root mean square error of the overall geometric consistency of the method of the present invention was significantly reduced to 0.72 pixels, and the error in weakly textured areas was only 0.98 pixels. This result clearly demonstrated that the strategy of using the continuity of image optical flow to guide the local adaptive fusion of LiDAR point clouds proposed by the present invention can effectively absorb the high-frequency disturbances of the UAV platform in complex airflows and accurately capture local deformations through a time-varying affine model, thus achieving high-precision geometric alignment between LiDAR point clouds and images at the local scale. Especially in weakly textured areas, the improvement in geometric consistency error of the method of the present invention compared to the control group was more significant. This advantage benefited from the effective guidance of the LiDAR geometric saliency back-projection mechanism for the extraction of optical flow feature points, avoiding the matching blind spot problem of traditional methods in such areas.

[0035] The experimental data examples in this embodiment were systematically verified in a complex mountain environment, strongly confirming that a LiDAR and UAV image fusion processing method applied to complex mountains disclosed by the present invention has the following remarkable technical advantages and beneficial effects: high-precision dynamic adaptability. By using the continuity of image optical flow to construct a dynamic skeleton and combining dominant motion parameters to screen optical flow feature points, the present invention can effectively filter the interference of independent moving objects, ensuring the purity of the motion reference. This enables the reprojection process of LiDAR point clouds to adaptively follow the real motion trend of the ground surface on the imaging plane, significantly absorbing the dynamic errors of the UAV platform under high-frequency airflow disturbances, and thus achieving high-precision fusion under low-cost sensor conditions; enhanced robustness in weakly textured areas. By introducing the LiDAR point cloud geometric saliency back-projection mechanism to convert three-dimensional structure information into a guiding signal for two-dimensional images, this mechanism effectively activates the ability to extract optical flow feature points in weakly textured areas (such as bare rock walls, snow slopes), fundamentally solving the problem of matching failure caused by texture loss in traditional methods in such scenarios and ensuring the continuity and reliability of full-scene data fusion; excellent geometric consistency. Based on the dynamic anchoring relationship and local time-varying affine transformation model, the present invention realizes the local adaptive fusion of LiDAR point clouds and UAV images, significantly improving the geometric consistency between multi-source data. This enables the fused three-dimensional data to still maintain high reliability in weakly textured areas and dynamic interference conditions, providing a solid data foundation for three-dimensional perception of mountains.

[0036] Example 3: This example combinesFigures 1 to 3 , an implementation description of the LiDAR and UAV image fusion processing method applied to complex mountainous areas is provided. As Figure 1 shown, the process starts with the acquisition of UAV image sequences, which is carried out by high-frequency continuous acquisition. Image data of 5-10 frames per second is obtained. Sparse optical flow tracking is applied to extract stable optical flow feature points to construct a dynamic skeleton, and the spatio-temporal consistency of feature points is improved by combining dominant motion parameters. Motion-inconsistent points are excluded through stability verification (anti-dynamic interference) to ensure the accuracy of dynamic features. At the same time, LiDAR point cloud data is collected, georeferenced with the assistance of low-precision POS, and sparsified by voxel downsampling. Subsequently, geometric saliency analysis is carried out, including normal vector mutation detection and elevation extreme value extraction operations, to extract geometrically significant regions for constructing a LiDAR approximate skeleton. At the same time, it is back-projected and matched to achieve dynamic matching with the image sequence. Based on this, a dynamic anchoring relationship is established, which includes back-projection of 3D feature points and verification of optical flow trajectories. Neighborhood feature point modeling is completed based on 3-5 anchor point pairs in the matching area, and time-varying parameter updates are realized. Finally, a local affine transformation is formed, and non-rigid correction is carried out on this basis to complete <f1>Objectives of multi-source data fusion, dynamic reprojection, and bidirectional information mapping.

[0037] As Figure 2 shown, the scene types include snow-covered slopes, texture-rich areas, and bare rock walls, representing weak-texture and strong-texture environments respectively. The state is classified into three categories: the first is the state before optimization, represented by hollow circles in the figure; the second is the state after optimization, represented by solid circles; the third is the baseline performance, represented by diamonds. The specific data is as follows: in the snow-covered slope scene, the anchoring success rate in the state before optimization (circle marker) is 58.7%, and it is increased to 93.8% in the state after optimization (solid circle marker); in the bare rock wall scene, the state before optimization is 65.2%, and it is increased to 93.7% in the state after optimization; in the texture-rich area, as the baseline performance, the anchoring success rate reaches 98.5% (diamond marker).

[0038] As Figure 3 shown, first, in the UAV image processing module, the system acquires and preprocesses a continuous image sequence, then tracks the trajectories of recognizable feature points through an optical flow algorithm, and further extracts the dynamic skeleton and transmits the image dynamic skeleton to the fusion module. At the same time, the LiDAR point cloud processing module is responsible for acquiring LiDAR point cloud data, and performs preliminary georeferencing and sparsification in combination with low-precision POS information to identify sparse feature points to form a skeleton, and then transmits the LiDAR approximate skeleton to the fusion core module; the fusion core module first back-projects sparse three-dimensional feature points based on the low-precision pose, compares them with the optical flow features in the image space, and then establishes a dynamic anchoring relationship. On this basis, a time-varying affine transformation model is constructed. The fusion core module then requests local data of the original point cloud, and the LiDAR module responds to return the corresponding original point cloud, enabling the fusion module to perform dynamic reprojection of the LiDAR point cloud, and finally completing the dynamic fusion of the LiDAR point cloud and the UAV image.

[0039] Example 4: In a typical high-risk early warning response mission for landslide disasters, the local emergency management department dispatched and deployed a ground-air-algorithm integrated emergency remote sensing platform constructed based on the technical solution of the present invention. The deployment location was along a highway section near a cliff in a mountainous area of Yunnan. The terrain slope in this area exceeded 45 degrees, the landform fluctuated violently, and there were significant occlusion and complex shadow areas. Conventional optical remote sensing technology was limited by imaging distortion and texture loss, making it difficult to obtain continuous and reliable three-dimensional model data. Especially under continuous rainy conditions for several days, the traditional three-dimensional modeling method based on single-image difference had large errors and was difficult to meet the urgent need for high-precision and high-timeliness landform information in emergency rescue. In this application scenario, the system carried out data acquisition and fusion processing operations according to the following process: First, the unmanned aerial vehicle obtained high-resolution image data at an interval of 0.5 seconds according to the preset flight path, and simultaneously triggered the LiDAR module carried to collect point cloud data. Both the image and point cloud data recorded the absolute time stamps through the high-precision time synchronization module to ensure that the subsequent registration accuracy was controlled within ±5 milliseconds. This synchronization accuracy requirement was set based on the time delay evaluation of multiple groups of typical tasks and the statistical results of the actual response time of the system, taking into account the performance upper limit of the embedded timing module and the average fluctuation range of image buffer processing.

[0040] In the initial modeling stage, to solve the problem of incomplete image information caused by cliffs, occluded depressions, etc. in the mountain area, the system introduced a terrain adaptive anchoring mechanism. Specifically, the system first extracted the spatial point set with a normal vector mutation rate exceeding 12% from the collected LiDAR point cloud as the candidate set of suspected boundary points. This numerical threshold was derived from the analysis results of the geometric feature evolution process in typical landslide precursor areas and was dynamically weighted in combination with the local elevation gradient in the current task scenario to effectively avoid misjudgments caused by interference points such as vegetation and vehicle bodies. The extracted boundary points were back-projected onto the corresponding synchronized image frames after being double-weighted and screened by local curvature and point density, forming an anchor candidate projection area. In this area, the system enhanced the effectiveness of subsequent feature point extraction through operations such as brightness histogram broadening and image detail enhancement. Subsequently, the Harris corner detection method with adjusted thresholds (the threshold was set to 70% of the standard value) was used to extract initial feature points from the enhanced image, and the optical flow trajectories were calculated in combination with the images of the previous and subsequent frames. If a certain feature point successfully completed optical flow tracking in three consecutive frames of images, and its change in motion direction was less than 15 degrees and the change in modulus length did not exceed 10%, the system confirmed that this area met the conditions for constructing a dynamic anchoring relationship. After establishing the anchoring relationship, the system constructed a local affine transformation model based on the selected anchor point pairs on each frame of the image, and selected anchor point pairs with a spatial angle greater than 20 degrees to avoid numerical ill-conditioning of the affine matrix, aiming to avoid numerical ill-conditioning caused by linear correlation and ensure that the model solution had sufficient physical separation.

[0041] To address the problem of missing texture information caused by the coexistence of shadow areas and bright spot areas in this scenario, the system introduces a geometric prior guidance mechanism. The specific operation is as follows: When the number of feature points that can be extracted in a certain area is less than one-thousandth of the total number of pixels (i.e., there are less than 100 valid corner points in every 100,000 pixels), the system automatically calls the corresponding 3D point cloud information in this area for auxiliary positioning and guides the feature recovery. Specifically, it includes selecting points in the point cloud with an elevation range difference exceeding 1.5 meters and a change in the normal vector angle greater than 8 degrees as the basis for auxiliary anchor points, and constructing the area to be enhanced based on this; after the above-mentioned auxiliary anchor points are back-projected and mapped to the image space, a 25-pixel × 25-pixel rectangular region of interest (ROI) is constructed with the pixel where it is located as the center, and optical flow initialization and the stability verification of its motion trajectory are performed therein. When the offset of the trajectory between consecutive frames is less than 2 pixels and the angle between its motion direction and the normal vector of this point is less than 30 degrees, the system includes this anchor point in the set of valid anchor points; after completing the anchoring and registration of all image frames and point cloud data, the system performs dynamic image projection and texture assignment processing on the original point cloud based on the above affine transformation model. In this process, the system not only projects each 3D point to the pixel position of the corresponding image frame, but also constructs a fused texture layer based on its laser intensity value, spatial elevation, and image grayscale to achieve pseudo-color rendering, so as to enhance the visual recognition ability of structures such as landform boundaries, landslide accumulation bodies, and cracks. At the same time, for some point cloud areas that cannot be projected due to occlusion or high reflection, the system completes the texture information through the interpolation mechanism between adjacent three frames of images, ensuring that the coherence of the final fused image in the time series is not less than 90%. To verify the technical feasibility and effect advantages of the above fusion modeling process, the system synchronously collects the results of traditional SFM modeling + texture mapping under the same flight line at the above deployment site for comparative analysis. The test results show that the feature recovery rate in the area with missing texture information has increased by more than 60%; and the overall time from data collection to modeling completion is controlled within 2 hours, significantly better than the processing cycle of more than 6 hours generally for existing mainstream methods.

[0042] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for fusing and processing LiDAR and UAV images applied to complex mountainous areas, characterized in that The method includes the following steps: Step a: Obtain a continuous image sequence collected by a drone in a target area, and track the motion trajectories of recognizable feature points between consecutive image frames through a sparse optical flow algorithm; Based on the dominant motion parameters between image frames, screen out the optical flow feature points that can be stably tracked in multiple consecutive frames and whose local motion vectors have a deviation from the motion trend characterized by the dominant motion parameters less than a preset deviation threshold, and form an image dynamic skeleton; Step b: Obtain LiDAR point cloud data collected during the same period as the image sequence and data of a low-precision positioning and orientation system for preliminary positioning; Use the data of the low-precision positioning and orientation system to perform georeferencing on the LiDAR point cloud data, and perform voxel downsampling on the LiDAR point cloud data to extract sparse three-dimensional feature points based on the rate of change of normal vectors or local elevation extrema, and form a LiDAR rough skeleton; Step c: Based on the camera pose provided by the data of the low-precision positioning and orientation system, back-project the sparse three-dimensional feature points in the LiDAR rough skeleton onto the image frames in the image sequence to form a projection area of the sparse three-dimensional feature points on the image; For the projection area, search for a sequence of optical flow feature points in it that have the shortest distance from the sparse three-dimensional feature points in three-dimensional space and whose optical flow motion trajectory stability meets a preset stability standard, and establish a dynamic anchoring relationship between the sparse three-dimensional feature points and the sequence of optical flow feature points; Step d: Based on the dynamic anchoring relationship, for the sparse three-dimensional feature points and their anchored sequence of optical flow feature points, combine three to five pairs of nearest neighbor feature point pairs with established dynamic anchoring relationships around them to construct a locally affine transformation model that changes over time; Use the locally affine transformation model to dynamically reproject the original point cloud corresponding to the neighborhood of the sparse three-dimensional feature points in the LiDAR point cloud data onto the image frame corresponding to the dynamic anchoring relationship.

2. A LiDAR and UAV image fusion processing method applied to complex mountains according to claim 1, characterized in that The method for screening optical flow feature points in step a is as follows: after tracking the motion trajectories of recognizable feature points, estimate the dominant inter-frame motion parameters caused by the motion of the UAV itself between consecutive image frames; for each tracked feature point, calculate the vector difference between its local motion vector and the motion vector expected at the position of this feature point based on the dominant inter-frame motion parameters; when the magnitude of the vector difference is greater than the preset deviation threshold then identify this feature point as a motion inconsistent point and exclude it when screening stable optical flow feature points or establishing dynamic anchoring relationships.

3. A LiDAR and UAV image fusion processing method applied to complex mountains according to claim 1 or 2, characterized in that The method for tracking the motion trajectories of recognizable feature points in step a is to use the KLT optical flow algorithm.

4. A LiDAR and UAV image fusion processing method applied to complex mountainous areas according to claim 1, wherein, Before step a, it further includes: Based on the LiDAR point cloud data, perform three-dimensional geometric structure significance analysis on each point or its local neighborhood in the LiDAR point cloud data, and calculate the rate of change of normal vectors, curvature, or elevation standard deviation of the point cloud; And according to the condition that the three-dimensional geometric structure significance measurement value is higher than a specific threshold, identify the geometric structure significant area.

5. A LiDAR and UAV image fusion processing method applied to complex mountains according to claim 4, characterized in that, The geometric structure significant area is back-projected onto the image frames in the continuous image sequence using the data of the low-precision positioning and orientation system to calibrate the corresponding virtual anchor point potential area within the image frames; And when extracting optical flow feature points in step a or establishing a dynamic anchoring relationship in step c, for the image area falling within the virtual anchor point potential area, refer to the local three-dimensional geometric structure significance information of the LiDAR point cloud to guide or strengthen the extraction of optical flow feature points or the establishment of a dynamic anchoring relationship.

6. A LiDAR and UAV image fusion processing method applied to complex mountains according to claim 1, characterized in that, The method for establishing a dynamic anchoring relationship in step c is: For each sparse three-dimensional feature point, search and select in the projection area of its back-projection onto the image frame an optical flow feature point that has the shortest distance from it in three-dimensional space and whose optical flow motion trajectory has stability between consecutive frames meeting a preset stability standard, and form a dynamic anchoring relationship.

7. A LiDAR and UAV image fusion processing method applied to complex mountains according to claim 1, characterized in that, The continuous image sequence is a video stream or high-frequency burst photos at five to ten frames per second.

8. A method for fusing and processing LiDAR and UAV images applied to complex mountains according to claim 1, characterized in that, A method for extracting sparse three-dimensional feature points from LiDAR point cloud data, including normal vector mutation detection or local elevation extreme value extraction based on LiDAR point cloud.

9. A LiDAR and UAV image fusion processing method applied to complex mountainous areas according to claim 1, wherein, After step d, it further includes mapping the elevation or intensity information carried by the LiDAR point cloud data after dynamic reprojection to the matching pixels on the corresponding image frame; or, superimposing the spectral or texture information of the image frame onto its corresponding original three-dimensional point cloud.

Images