LiDAR and UAV image fusion processing method applied to complex mountainous areas

Abstract

The present invention relates to the field of geospatial information acquisition technology and discloses a method for fusion processing of LiDAR and drone imagery for use in complex mountainous areas. The method comprises extracting stable optical flow feature points from drone image sequences to construct a dynamic skeleton, establishing dynamic anchoring relationships based on sparse three-dimensional feature points in the LiDAR point cloud, and constructing a time-varying affine transformation model based on local neighborhood feature point pairs to achieve dynamic reprojection of the LiDAR point cloud. The method utilizes the continuity of image optical flow to guide the local adaptive fusion of LiDAR point clouds, effectively overcoming the problems of sensor dynamic error and texture loss in complex mountainous environments. It significantly improves the geometric consistency of multi-source data in weakly textured areas and under dynamic interference conditions, providing a highly reliable fusion data foundation for three-dimensional perception in mountainous areas.

Description

Technical Field

[0001] The present invention relates to a LiDAR and UAV image fusion processing method applied to complex mountainous areas, belonging to the technical field of geospatial information acquisition. Background Art

[0002] Current mainstream technologies usually rely on high-precision positioning and orientation systems (POS) to obtain the drone's position and posture, and based on this, use iterative optimization algorithms (such as ICP variants) to achieve global rigid registration of LiDAR point clouds and images. This type of method has certain effectiveness in static scenes, but in mountainous environments with frequent airflow disturbances and significant vegetation dynamics, its inherent limitations are becoming increasingly prominent: when drones perform monitoring tasks in complex mountainous areas, airflow turbulence causes high-frequency attitude jitter of the platform, and the local movement of the vegetation canopy caused by wind disturbance further destroys the spatiotemporal consistency between the image and the point cloud.

[0003] Existing technologies use global rigid transformation models to attempt to bridge this dynamic difference, but in fact they fall into the fundamental contradiction of fitting static models to dynamic scenes. The core bottlenecks are: 1. Low-precision POS data cannot accurately reflect the instantaneous pose changes of the sensor. Forced global alignment leads to meter-level misalignment in local areas such as forest edges and steep slopes, destroying the geometric authenticity of the fused data; 2. Weakly textured areas such as exposed rock walls and snow-covered slopes lack stable image features, and point cloud geometric features are prone to spatial ambiguity in two-dimensional projections. The probability of failure of traditional matching methods in such areas is significantly increased; 3. To compensate for the above defects, existing solutions either rely on high-precision POS hardware or introduce complex computational models, which greatly raise the data processing threshold and make it difficult to meet the needs of lightweight operations such as geological disaster emergency monitoring.

[0004] The industry has attempted to alleviate the matching problem by improving feature descriptors or introducing deep learning models, but these efforts have failed to address the core challenge of adaptability to dynamic environments. Instead, algorithmic complexity exacerbates the computational burden. Therefore, building a multi-source data fusion mechanism that is inherently adaptable to dynamic mountain environments, while using low-cost sensors, and achieving high-confidence spatiotemporal alignment at the local scale, has become a key technical challenge in improving the reliability of 3D perception in complex scenes. Summary of the Invention

[0005] The present invention provides a LiDAR and UAV image fusion processing method applied to complex mountainous areas. Its main purpose is to solve the problems of fusion dislocation and weak texture area matching failure caused by insufficient dynamic environment adaptability under low-cost sensor conditions.

[0006] To achieve the above objectives, the present invention provides a method for fusion processing of LiDAR and UAV images applied to complex mountainous areas, the method comprising the following steps:

[0007] Step a: Acquire a sequence of continuous images captured by the drone in the target area and track the motion trajectories of identifiable feature points between consecutive image frames using a sparse optical flow algorithm. Based on the dominant motion parameters between the image frames, select optical flow feature points that can be stably tracked in multiple consecutive frames and whose deviation between the local motion vector and the motion trend represented by the dominant motion parameter is less than a preset deviation threshold, thereby forming an image dynamic skeleton.

[0008] Step b: obtaining LiDAR point cloud data collected contemporaneously with the image sequence and low-precision positioning and orientation system data for preliminary positioning; georeferencing the LiDAR point cloud data using the low-precision positioning and orientation system data, and performing voxel downsampling on the LiDAR point cloud data to extract sparse three-dimensional feature points based on normal vector change rates or local elevation extremes to form a LiDAR skeleton;

[0009] Step c: Based on the camera pose provided by the low-precision positioning and orientation system data, the sparse three-dimensional feature points in the LiDAR skeleton are reversely projected onto the image frames in the image sequence to form a projection area of the sparse three-dimensional feature points on the image; within the projection area, a sequence of optical flow feature points is searched for that has the shortest distance to the sparse three-dimensional feature points in three-dimensional space and whose optical flow motion trajectory stability meets a preset stability standard, and a dynamic anchoring relationship is established between the sparse three-dimensional feature points and the sequence of optical flow feature points;

[0010] In step d, based on the dynamic anchoring relationship, for the sparse three-dimensional feature points and their anchored optical flow feature point sequences, a local affine transformation model that changes with time is constructed in combination with the feature point pairs of the three to five nearest neighbors with established dynamic anchoring relationships. The local affine transformation model is used to dynamically reproject the original point cloud corresponding to the sparse three-dimensional feature point neighborhood in the LiDAR point cloud data onto the image frame corresponding to the dynamic anchoring relationship, thereby realizing the dynamic fusion of LiDAR point cloud data and drone images.

[0011] Preferably, the method for selecting optical flow feature points in step a is as follows: after tracking the motion trajectory of the identifiable feature points, estimating the dominant inter-frame motion parameters caused by the drone's own motion between consecutive image frames; for each tracked feature point, calculating the vector difference between its local motion vector and the motion vector expected at the feature point position based on the dominant inter-frame motion parameters; when the modulus of the vector difference is Greater than the preset deviation threshold , the feature point is identified as a motion inconsistent point and is excluded when screening stable optical flow feature points or establishing dynamic anchoring relationships.

[0012] Preferably, the method for tracking the motion trajectory of the identifiable feature points in step a is to use a KLT optical flow algorithm.

[0013] Preferably, before step a, the method further includes: performing a three-dimensional geometric structure significance analysis on each point in the LiDAR point cloud data or its local neighborhood based on the LiDAR point cloud data, calculating the normal vector change rate, curvature or elevation standard deviation of the point cloud; and identifying geometric structure significant areas based on the condition that the three-dimensional geometric structure significance metric value is higher than a specific threshold.

[0014] Preferably, the geometrically salient area is back-projected onto the image frames in the continuous image sequence using the low-precision positioning and orientation system data to calibrate the corresponding virtual anchor potential area within the image frame; and when executing step a to extract the optical flow feature points or executing step c to establish the dynamic anchor relationship, for the image area falling within the virtual anchor potential area, the local three-dimensional geometric structure saliency information of the LiDAR point cloud is referenced to guide or strengthen the extraction of optical flow feature points or the establishment of the dynamic anchor relationship.

[0015] Preferably, the method for establishing a dynamic anchoring relationship in step c is: for each sparse three-dimensional feature point, within the projection area of its reverse projection onto the image frame, search and select the optical flow feature point with the shortest distance to it in the three-dimensional space and whose optical flow motion trajectory stability between consecutive frames meets the preset stability standard to form a dynamic anchoring relationship.

[0016] Preferably, the continuous image sequence is a video stream of five to ten frames per second or high-frequency continuous shooting photos.

[0017] Preferably, the method for extracting sparse three-dimensional feature points from LiDAR point cloud data includes normal vector mutation detection or local elevation extreme value extraction based on LiDAR point cloud.

[0018] Preferably, after step d, the method further includes mapping the elevation or intensity information carried by the dynamically reprojected LiDAR point cloud data to matching pixels on the corresponding image frame; or, superimposing the spectrum or texture information of the image frame to its corresponding original three-dimensional point cloud.

[0019] Preferably, in step b, voxel downsampling is performed based on the determined voxel size to ensure stability and repeatability of the sparsification degree.

[0020] Compared with the background technology problems, the beneficial effects of the present invention are:

[0021] 1. By leveraging the continuity of optical flow trajectories in image sequences to construct a dynamic skeleton, the system replaces the traditional rigid registration model that relies on high-precision positioning systems. This mechanism enables the reprojection process of the LiDAR point cloud to adaptively follow the actual motion of the ground surface on the imaging plane, effectively absorbing the high-frequency disturbances of the UAV platform in complex airflow. As a result, the system can maintain geometric-spectral consistency in local areas even with low-cost sensors, avoiding the fusion misalignment caused by the mismatch between the global model and local dynamics in traditional methods. Furthermore, image feature points selected based on optical flow motion stability are spatiotemporally coupled with LiDAR sparse 3D feature points through a dynamic anchoring relationship. This relationship is further combined with the spatiotemporal distribution of local neighborhood feature point pairs to construct an affine transformation model that evolves over time, ensuring that the local reprojection of the LiDAR point cloud remains synchronized with the image dynamics. This closed-loop logic, which extracts benchmarks from dynamics and uses them to guide fusion, significantly reduces feature matching ambiguity in areas with repetitive mountain textures, providing a geometrically and semantically consistent data foundation for subsequent intelligent analysis.

[0022] 2. Geometrically significant features of the LiDAR point cloud, such as normal vector mutations and elevation extremes, are reverse-mapped to the image space to form a potential area for virtual anchor points. When the image's own texture features are insufficient, this area is activated by lowering the feature extraction threshold or implanting geometrically guided virtual tracking points to generate optical flow trajectories. This operation converts 3D structural information into guiding signals for 2D interpretation, avoiding the fusion blind spots of traditional methods in weakly textured areas such as snow and shadows, and ensuring data continuity across the entire scene. Furthermore, dominant motion parameters are introduced as a reference for screening optical flow feature points. By analyzing the differences between local motion vectors and dominant trends, interfering trajectories of independent moving objects within the scene, such as flying birds and floating objects, are actively identified and eliminated. This mechanism forms a synergistic defense with dynamic anchoring relationships: the former purifies the input dynamic skeleton, while the latter ensures that the anchoring relationship only relates to real surface motion. Both replace complex object recognition models with minimalist vector operations, enabling the system to achieve interference-resistant fusion within the edge computing unit.

[0023] 3. The interaction of elevation and spectral information based on local dynamic anchoring relationships enables the natural fusion of the geometric properties of the LiDAR point cloud and the texture properties of the image in a spatiotemporal alignment state. This process does not require global point cloud registration or pixel-level alignment. Instead, it achieves precise mapping of 3D geometric details to 2D images and semantic attachment of 2D spectral information to 3D point clouds through local reprojection guided by sparse feature points, forming a bidirectionally enhanced geospatial information expression. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flowchart of the multi-source data fusion process of the present invention;

[0025] Figure 2 A comparison chart of anchoring success rates for dynamic anchoring relationships in various scenarios of the present invention;

[0026] Figure 3 This is a timing diagram of the dynamic fusion of multi-module collaborative LiDAR and drone images in the present invention.

[0027] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0028] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0029] The present invention proposes a LiDAR and UAV image fusion processing method for complex mountainous areas, which avoids the limitation of traditional rigid registration models on high-precision positioning. By constructing dynamic anchoring relationships and local time-varying affine models, high-precision dynamic data fusion is achieved under low-cost sensor conditions. In the specific application process, the present invention combines the concepts of geometrically significant areas, dynamic anchoring relationships, and dominant inter-frame motion parameters, and combines the field test experience of a large number of typical mountain scenes. Based on the observed point cloud structure change characteristics, the dominant motion trend between image frames, and the stability of optical flow trajectories, a robust and reusable judgment strategy is formed, such as The identification of geometrically significant areas is essentially based on the sudden change of height difference or rapid change of direction presented by the point cloud in a local range, which often appears in areas such as rock wall turning points, vegetation edges or mountain fault zones, showing stable spatial structure identification characteristics; the selection of the dominant inter-frame motion parameters is based on the main displacement component presented in the overall motion trend of the image, which can be naturally summarized through the aggregation analysis of the overall motion field of continuous frames of the image; as for the establishment of dynamic anchoring relationships, it mainly depends on the continuity and motion consistency of the point cloud features in the corresponding area of the image. The system is determined based on the dual constraints of relatively strict optical flow trajectory stability standards and spatial proximity.

[0030] The whole scheme is based on the image sequence collected by the UAV in the target area continuously. The sparse optical flow algorithm is used to track the identifiable feature points between frames, and the optical flow feature points that remain stable and have consistent motion trends between multiple frames are extracted to construct the image dynamic skeleton that reflects the real motion trend of the surface. The so-called stability here refers to the deviation vector model between the motion vector direction of the feature point in consecutive frames and the overall dominant motion direction of the image. Less than the set threshold , usually set between 0.3 and 0.5 pixels of inter-frame movement, and can be adjusted appropriately according to the flight speed and sampling frequency of the drone. The dominant motion parameters are obtained by performing principal component analysis on the full-frame motion vector field and extracting the direction of the maximum eigenvalue as the overall dominant motion direction; the acquisition of LiDAR point cloud data is carried out simultaneously with image acquisition, and is supplemented by the initial posture information provided by the low-precision POS (Position and Orientation System). In the original point cloud, the present invention does not directly use the full amount of data, but introduces a sparsification mechanism based on local geometric significance indicators: after downsampling through the voxel grid, the normal vector change rate of each point or its neighborhood, that is, the rotation distance on the unit sphere, the local elevation extreme value or the point cloud curvature, is calculated respectively. If the above-mentioned geometric significance indicators are higher than the set threshold, such as the local elevation difference or curvature , it is marked as a sparse three-dimensional feature point to form a LiDAR skeleton. This strategy can effectively enhance the recognition of the projection area of 3D features on the image and reduce mismatching in fuzzy areas.

[0031] Next, based on the coarse pose information provided in the POS data, the sparse three-dimensional feature points are back-projected onto the image frame through internal and external parameters. For each projection area, the optical flow feature points in the area that are closest to the original three-dimensional feature points in spatial distance and whose optical flow trajectories meet the continuity and stability requirements are searched, and a one-to-one dynamic anchoring relationship is established between them. The stability standard of this relationship includes the inter-frame motion vector variance of the optical flow point in continuous tracking of more than 5 frames. , and the direction angle deviates from the dominant direction by no more than 10°. This anchoring mechanism ensures that the selected optical flow points truly reflect the continuous motion of physical entities on the surface, rather than independent disturbances such as flying birds or floating branches and leaves. Once the anchoring relationship is established, the system uses this relationship as a benchmark and combines 3 to 5 groups of known stable anchor point pairs selected in the neighborhood of each pair of anchor points to jointly participate in the construction of a local affine transformation model. , the model is based on time is a variable that captures the local deformation trend of the fusion area between image frames. Specifically, the affine model uses the least squares method to fit the following form: ,in, is the original point cloud coordinate projection, is the corresponding position of the target image frame, is a 2×2 local linear transformation matrix, The translation vector is dynamically updated to achieve accurate matching of the point cloud with the corresponding image position at different times.

[0032] After that, all original points in the original point cloud that belong to the sparse 3D feature point neighborhood (usually within a radius of 1.5 to 3 meters) are dynamically reprojected, that is, according to the above The model adjusts its position in the imaging plane to complete the time-synchronous fusion of the point cloud and the image. Unlike the traditional method of implementing rigid transformation on the entire point cloud, the present invention relies on the local affine model to achieve small-scale non-rigid geometric correction, which is particularly suitable for mountain scenes with dynamic fluctuations of vegetation canopies or frequent platform posture perturbations; at the same time, for weak texture areas, such as snow-covered slopes and rock shadows, due to the scarcity of optical flow points, the present invention further introduces the LiDAR point cloud geometric saliency back-projection mechanism, projects the high-saliency three-dimensional structure in the image to form a virtual anchor potential area, and combines image histogram equalization and gradient enhancement processing, supplemented by the threshold-reducing feature point extraction strategy, to artificially activate the optical flow The point generation capability enables sparse texture areas to still effectively participate in dynamic anchoring; after the core fusion is completed, in order to achieve further unification of multi-source data at the semantic level, the present invention supports a two-way information projection mechanism: on the one hand, the elevation (z value) or intensity value of the reprojected point cloud is mapped to the corresponding pixel of the image to construct a digital surface model (DSM) layer; on the other hand, the texture information of the image (such as RGB value, edge features, texture directionality, etc.) can also be superimposed on the corresponding three-dimensional point cloud for subsequent surface semantic segmentation or intelligent recognition; in addition, in order to improve the stability and repeatability of data processing, the present invention adopts a fixed grid resolution (such as 0.5m × 0.5m × 0.5m), to avoid feature point drift caused by point density fluctuations; at the same time, all parameter selections are combined with the actual mountain scene test results to ensure wide adaptability in different seasons and different landform types; at the same time, to ensure the spatial accuracy and robustness of the dynamic anchoring relationship in the multi-source data fusion process, after establishing a one-to-one correspondence between sparse 3D feature points and optical flow feature points, the system further introduces an anchor offset statistical mechanism. By evaluating the mean and range of the projection trajectory offsets of the matched point pairs in multiple consecutive image frames, if its maximum projection offset exceeds the reasonable image movement range set by the system, the anchor pair is automatically eliminated, and the candidate feature point reselection strategy in the adjacent area is triggered to ensure that the geometric constraint accuracy of the anchoring relationship is not affected by instantaneous abnormal disturbances.In addition, for situations where optical flow feature points are difficult to extract or anchoring relationships fail to be established in weak texture areas, the system provides a regional local fallback mechanism, that is, based on the anchoring trend of the same area in the historical frame or the established adjacent stable anchoring relationship, the motion state of the area is inferred by interpolation, and the system automatically generates temporary auxiliary anchor points as a supplement to avoid local fusion interruption due to the lack of individual anchors, ensuring the continuity and stability of the overall fusion process. In the process of constructing time-varying affine transformation, in order to improve its adaptability under multi-angle observation conditions of mountain scenes, the system adopts a sliding method based on time sequence. The modeling mechanism ensures that the model is jointly driven by multiple anchor point pairs with good spatial distribution characteristics at each stage. This modeling method can dynamically adapt to changes in the sensor's perspective, and ensure the smooth transition of affine parameters in the time series dimension through frame-by-frame updates, avoiding instantaneous geometric distortion caused by single-frame disturbances. In practice, if the fitting residuals in multiple consecutive image frames exceed the set range, the system will trigger the anchor point re-screening process, and give priority to new anchor point pairs with uniform distribution and relatively stable distribution for model reconstruction, thereby further improving the convergence stability and overall fusion consistency of the local affine model; as well as the deviation threshold in this embodiment. It is used to judge the consistency between the local motion trend of the feature point and the dominant inter-frame motion trend of the image. It is a core parameter of the stable optical flow feature point screening mechanism. Specifically, when the degree of deviation of the motion trajectory of the feature point between consecutive image frames from the dominant motion direction exceeds the threshold, that is, the inter-frame displacement deviation corresponding to the angle difference is greater than , the system identifies it as a point with inconsistent motion trend and removes it. In a typical experiment, The value of is set to 0.4 pixels, which is determined based on the principal component analysis results of the dominant motion vector field of the full frame and the statistical experience of the distribution of the continuous tracking deviation of the feature points. This threshold takes into account the strictness of the consistency of the motion trend and the sufficiency of the number of feature points. Under different landform types or flight parameters, Appropriate adjustments can be made based on actual data performance to ensure that the extracted optical flow feature points are both stable and spatially representative, which are all extended implementation methods known to those skilled in the art.

[0033] Example 1: In order to achieve high-precision dynamic fusion of LiDAR point cloud and drone image data under low-cost sensor conditions in complex mountainous environments, this example details a specific implementation method of fusion processing. Its core mechanism includes constructing a dynamic skeleton through stable optical flow features in the image, establishing a dynamic anchoring relationship in combination with sparse three-dimensional feature points, and reprojecting the point cloud based on a time-varying local affine model, thereby forming multi-source fusion data with high geometric and spectral consistency. First, a drone platform with continuous image acquisition capabilities is deployed in the designated target area. The platform acquires image sequences at a frequency of approximately six frames per second to ensure sufficient temporal resolution to capture high-frequency dynamic changes. The identifiable feature points in the continuous image frames are tracked frame by frame through a sparse optical flow algorithm based on corner detection, and the overall dominant motion direction is extracted using principal component analysis of the full-frame motion vector field. Subsequently, the angle difference between the motion vector of each feature point and the dominant direction is calculated, and stable feature points with a deviation of less than 0.5 pixels between frame displacement are screened out for constructing a dynamic skeleton; secondly, the image is acquired synchronously with the image. The LiDAR point cloud data corresponding to the sequential time period is used, and the pose information provided by the low-precision positioning and orientation system is introduced for preliminary geographic reference. To improve the efficiency and accuracy of the fusion processing, the original point cloud is first voxelized and down-sampled, and points with significant normal vector change rate or local elevation extreme value are selected in each voxel to construct a sparse 3D feature point set. Then, the feature point set is back-projected to the corresponding image frame through the internal and external parameters of the camera. In each projection area, optical flow feature points are searched, and a one-to-one dynamic anchoring relationship is established between them and the 3D feature points based on their stability in continuous tracking for more than five frames.

[0034] Subsequently, three to five groups of confirmed stable anchor point pairs are selected in the neighborhood of each pair of dynamic anchor points to jointly construct a local time-varying affine model. The model is dynamically generated by least squares fitting to reflect the spatial transformation characteristics of the point cloud neighborhood between different image frames. Based on the established affine model, the original point cloud data with a radius of about two meters in the neighborhood of the sparse three-dimensional feature points is dynamically reprojected to the corresponding image frame to achieve local non-rigid matching; for problem areas with texture loss (such as snow-covered areas, shadow areas or rock walls), since traditional optical flow algorithms are difficult to generate stable feature trajectories, this embodiment back-projects the geometrically significant areas of the LiDAR point cloud at the initial stage of construction, and performs local histogram equalization and gradient enhancement on the corresponding image areas to improve feature extraction. At the same time, the feature detection threshold is appropriately lowered in such areas to increase the probability of generating feature points and ensure that they can participate in the subsequent dynamic anchoring process; after the fusion is completed, the elevation or reflection intensity information carried by the reprojected point cloud is mapped to the image frame to construct an elevation layer or a multispectral fusion layer; the texture information of the image can also be mapped to the corresponding three-dimensional points to achieve semantic enhancement expression of the point cloud. In all processing links, the parameter settings adopted, including voxel size, stability judgment standard, affine neighborhood radius, etc., can be optimized based on the mountain real-life test data in historical data to ensure that the method has good versatility and operational robustness under different seasons and different landform types. They are all extended implementation methods that can be known to ordinary technicians in this field.

[0035] Example 2: For example, in an area with typical complex mountain features in the southwest region, the test site is used. The area has steep terrain, dense vegetation, and covers areas with weak textures such as exposed rock walls and seasonal snow cover. The area is affected by valley winds all year round and has frequent airflow disturbances. During the test, a customized and improved drone-mounted LiDAR-imaging integration system was used. The drone platform uses a six-rotor drone with high payload capacity, and its flight control system is optimized to enhance attitude anti-disturbance performance. The LiDAR sensor uses a laser radar with medium range and high-frequency scanning capabilities (ranging accuracy can reach plus or minus 2 cm), and the image sensor uses an industrial The commercial-grade high-resolution digital camera has a high-frequency continuous shooting capability of up to 10 frames per second, ensuring the acquisition of continuous image sequences with high inter-frame correlation. When acquiring data, the drone strictly follows the preset route and flies at a speed of approximately 6 meters per second, while recording LiDAR point cloud data, continuous image sequences, and low-precision positioning and orientation system data for preliminary georeferencing. This low-precision positioning and orientation system provides horizontal positioning accuracy within the range of approximately 1 to 3 meters and attitude accuracy of approximately 0.5 degrees, which simulates the typical performance of the system under low-cost sensor conditions. The image acquisition frequency is set to 7 frames per second. This frequency is selected to balance temporal resolution and the efficiency of subsequent data processing.

[0036] Image dynamic skeleton construction experiment: The experiment in this stage aims to verify the technical effectiveness of accurately extracting stable optical flow feature points from continuous image sequences through the sparse optical flow algorithm and constructing an image dynamic skeleton. The experimental method is to use the KLT (Kanade-Lucas-Tomasi) optical flow algorithm to track the motion trajectory of identifiable feature points between continuous image frames. In order to evaluate the guiding role of dominant motion parameters in the screening of stable optical flow feature points, this experiment designed two screening strategies for comparative evaluation: Strategy 1 (control group), this strategy is based only on the continuity of optical flow tracking and the stability of feature points in multiple consecutive frames, that is, the proportion of successful tracking frames must be higher than 90% for screening, and does not introduce the consideration of dominant motion parameters; Strategy 2 (the strategy of the present invention), in Based on strategy one, the dominant inter-frame motion parameters caused by the drone's own motion between consecutive image frames were estimated. This parameter was obtained by performing principal component analysis on the full-frame motion vector field and determining the direction of the maximum eigenvalue as the overall dominant motion direction. Subsequently, the vector difference between the local motion vector of each tracked feature point and the motion vector expected at the feature point position based on the dominant motion parameter was calculated. The preset deviation threshold was set to 0.4 pixels. When the modulus of the vector difference was greater than this threshold, the feature point was identified as a motion inconsistent point and was excluded. Experimental results and analysis: In an image sequence covering typical airflow disturbances and swaying vegetation, 1000 initial feature points were randomly selected for tracking and screening. The results are shown in the following table:

[0037]

[0038] Although Strategy 1 obtains a large number of stable tracking points, it contains a significantly higher proportion of interference points from independent moving objects, reaching 12.3%. These interference points, such as flying birds, floating objects, or violently swaying branches, have motion trajectories that differ from the dominant motion trend of the UAV platform. Incorporating such points into the dynamic skeleton will inevitably introduce bias, thus affecting subsequent fusion accuracy. In contrast, Strategy 2 (the present invention's strategy), after rigorous screening of the dominant motion parameters, slightly reduces the number of stable optical flow feature points, but significantly reduces the proportion of interference points from independent moving objects to 1.8%. The consistency between the motion trend of the dynamic skeleton points and the dominant trend is significantly improved to 0.95. This result clearly demonstrates that by introducing the dominant motion parameters as a reference, the present invention effectively filters out the interference of independent moving objects in the scene, ensuring that the constructed image dynamic skeleton can truly and accurately reflect the overall motion trend of the surface on the imaging plane. This mechanism provides a highly reliable motion reference for the subsequent dynamic reprojection of the LiDAR point cloud, fundamentally avoiding the registration errors introduced by independent moving objects within the scene.

[0039] Experiment on extracting sparse 3D feature points from LiDAR point clouds and establishing dynamic anchoring relationships: This phase of the experiment aims to verify the extraction effect of sparse 3D feature points from LiDAR point clouds, and on this basis, to establish the effectiveness of dynamic anchoring relationships with image optical flow feature points, with particular attention paid to the anchoring ability in weak texture areas; The experimental method involves extracting sparse 3D feature points from LiDAR point clouds: First, the original LiDAR point cloud is voxel-downsampled, and the voxel size is set to 0.5 meters by 0.5 meters by 0.5 meters. Subsequently, the normal vector change rate or local elevation extreme value of each voxel midpoint or its local neighborhood is calculated. The normal vector change rate is defined here as the rotation distance between adjacent normal vectors on the unit sphere. This experiment sets the normal vector change rate threshold to be greater than 0.05 radians, or the local elevation difference (defined as the difference between the maximum elevation value and the minimum elevation value of the point cloud within the voxel) is greater than 1 .5 meters, and the points that meet any condition are marked as sparse three-dimensional feature points, thus forming a LiDAR rough skeleton; dynamic anchor relationship construction: using the camera coarse pose information provided by the low-precision positioning and orientation system data, the sparse three-dimensional feature points in the LiDAR rough skeleton are reversely projected onto the corresponding image frame to form a projection area of the sparse three-dimensional feature points on the image. For each projection area, search for the optical flow feature point with the shortest distance to the sparse three-dimensional feature point in the three-dimensional space and whose optical flow motion trajectory stability meets the preset standard. The optical flow motion trajectory stability standard includes that the inter-frame motion vector variance of the optical flow point must be less than 0.02 pixels squared in the tracking of more than 5 consecutive frames, and the deviation of its motion direction angle from the dominant direction does not exceed 15 degrees. A dynamic anchor relationship can be established between the optical flow feature point that meets the above conditions and the sparse three-dimensional feature point. To evaluate the adaptability of the present invention in weakly textured areas, the experiment further introduced a LiDAR point cloud geometric saliency back-projection mechanism as an auxiliary means. For weakly textured areas such as snow-covered areas and exposed rock walls, the geometrically salient areas of the LiDAR point cloud were back-projected onto the image frame to form a virtual anchor point potential area. Within this potential area, the optical flow feature point detection threshold was appropriately lowered, and local image histogram equalization and gradient enhancement processing were performed to activate the ability to generate optical flow feature points. Experimental results and analysis: In multiple typical mountain scenes containing exposed rock walls (weak texture), vegetation canopies (rich texture), and snow-covered slopes (weak texture), 200 LiDAR sparse 3D feature points were randomly selected for anchoring experiments. The results are shown in the following table:

[0040]

[0041] In texture-rich areas, such as vegetation canopies, the dynamic anchoring success rate reached as high as 98.5%, and the average anchoring error remained below 1 pixel, indicating that the method of the present invention can achieve high-precision matching under conditions of sufficient texture features. In weakly textured areas such as exposed rock walls and snow-covered slopes, the initial anchoring success rates without LiDAR geometric guidance were 65.2% and 58.7%, respectively, and the average anchoring error increased, reflecting the inherent limitations of traditional methods in weakly textured areas due to sparse or ambiguous feature points. However, after introducing the LiDAR point cloud geometric saliency backprojection mechanism and combining it with image processing optimization, the anchoring success rates for exposed rock walls and snow-covered slopes increased significantly by 28.5 percentage points and 35.1 percentage points, respectively, reaching high levels of 93.7% and 93.8%, respectively. This result clearly demonstrates that by converting LiDAR's three-dimensional structural information into guidance signals in two-dimensional image space, the present method can effectively compensate for the lack of image features in weakly textured areas, significantly improving the dynamic anchoring capability in these challenging scenarios and ensuring the continuity and reliability of full-scene data fusion.

[0042] LiDAR point cloud dynamic reprojection and fusion geometric consistency evaluation: The experiment in this stage aims to evaluate the performance of the present invention in realizing dynamic reprojection of LiDAR point cloud based on the local time-varying affine transformation model, and ultimately improve the performance of geometric consistency of multi-source data. The experimental method is that after successfully establishing a dynamic anchoring relationship, for each sparse three-dimensional feature point and its anchored optical flow feature point sequence, a local affine transformation model that changes with time is constructed in combination with the three to five pairs of nearest neighboring feature points with established dynamic anchoring relationships. The model is fitted using the least squares method and dynamically updated to achieve accurate matching of point clouds to corresponding image positions at different times. Subsequently, all original point clouds in the original LiDAR point cloud corresponding to the sparse three-dimensional feature point neighborhood (the neighborhood radius is set to 2 meters) are dynamically reprojected to the local affine transformation model according to their corresponding local affine transformation models. The dynamic anchoring relationship is based on the image frame corresponding to the dynamic anchoring relationship, thereby completing the dynamic fusion of LiDAR point cloud and UAV imagery. To quantitatively evaluate the fusion effect, this experiment selected the following indicators: Geometric consistency error of fusion data (root mean square error RMSE). The fused LiDAR point cloud data and the clearly identifiable points in the image are randomly selected, and their projection position deviation on the two-dimensional image is calculated, and the root mean square error is statistically analyzed in pixels. The comparison strategy uses a global rigid registration method based on low-precision positioning and orientation system data as a control group to evaluate its geometric consistency performance in dynamic mountain scenes. Experimental results and analysis: In a composite mountain scene containing strong airflow disturbances, vegetation dynamics, and weak texture areas, the two fusion methods were repeatedly tested, and the average geometric consistency error was statistically analyzed. The results are shown in the following table:

[0043]

[0044] The overall geometric consistency is significantly improved: the root mean square error of the overall geometric consistency error of the control group (global rigid registration method) is as high as 2.87 pixels, and it reaches 4.51 pixels in the weak texture area. This confirms the inherent limitations of relying solely on low-precision positioning and orientation systems for global rigid registration in complex mountain dynamic environments. Due to the high-frequency jitter and local dynamic deformation of the UAV platform, the forced global model is difficult to accurately fit the real movement of the local area, resulting in obvious misalignment; in contrast, the root mean square error of the overall geometric consistency error of the method of the present invention is greatly reduced to 0.72 pixels, and the error in the weak texture area is only 0.9 8 pixels. This result clearly proves that the strategy of using the continuity of image optical flow to guide the local adaptive fusion of LiDAR point clouds proposed in the present invention can effectively absorb the high-frequency disturbances of the UAV platform in complex airflow, and accurately capture local deformation through the time-varying affine model, thereby achieving high-precision geometric alignment of LiDAR point clouds and images at a local scale; especially in weak texture areas, the improvement in geometric consistency error of the method of the present invention is more significant than that of the control group. This advantage is due to the effective guidance of the LiDAR geometric saliency back projection mechanism on the extraction of optical flow feature points, which avoids the matching blind spot problem of traditional methods in such areas.

[0045] The experimental data embodiment has been systematically verified in a complex mountain environment, which strongly confirms that the LiDAR and UAV image fusion processing method disclosed by the present invention for complex mountainous areas has the following significant technical advantages and beneficial effects: high-precision dynamic adaptability, by using the continuity of image optical flow to construct a dynamic skeleton, and combining the dominant motion parameters to screen optical flow feature points, the present invention can effectively filter out the interference of independent moving bodies and ensure the purity of the motion reference, which enables the reprojection process of the LiDAR point cloud to adaptively follow the real motion trend of the surface on the imaging surface, and significantly absorb the dynamic error of the UAV platform under high-frequency airflow disturbance, thereby achieving high-precision fusion under low-cost sensor conditions; the robustness of weak texture areas is enhanced, and the present invention By introducing the geometric saliency back-projection mechanism of LiDAR point clouds, three-dimensional structural information is converted into guiding signals for two-dimensional images. This mechanism effectively activates the ability to extract optical flow feature points in weak-texture areas (such as exposed rock walls and snow-covered slopes), fundamentally solving the matching failure problem caused by texture loss in traditional methods in such scenes, and ensuring the continuity and reliability of full-scene data fusion; excellent geometric consistency, based on dynamic anchoring relationships and local time-varying affine transformation models, the present invention realizes local adaptive fusion of LiDAR point clouds and UAV images, significantly improving the geometric consistency between multi-source data, which enables the fused three-dimensional data to maintain high reliability in weak-texture areas and dynamic interference conditions, providing a solid data foundation for mountain three-dimensional perception.

[0046] Example 3: This example combines Figures 1 to 3 , and describes the implementation of the LiDAR and UAV image fusion processing method applied to complex mountainous areas. Figure 1 As shown, the process starts with the acquisition of drone image sequences, which is carried out through high-frequency continuous acquisition, acquiring image data of 5-10 frames per second. Sparse optical flow tracking is applied to extract stable optical flow feature points to construct a dynamic skeleton, and the temporal and spatial consistency of the feature points is improved by combining the dominant motion parameter screening. The motion inconsistency points are eliminated through stability verification (anti-dynamic interference) to ensure the accuracy of dynamic features. At the same time, LiDAR point cloud data is collected, combined with low-precision POS assistance for geo-reference, and voxel downsampling is used for sparse processing. Then, geometric significance analysis is carried out, including normal vector mutation detection and elevation extreme value extraction operations. Geometrically significant areas are extracted to construct the LiDAR skeleton, and reverse projection matching is performed on them to achieve dynamic matching with the image sequence. Based on this, a dynamic anchor relationship is established, which includes three-dimensional feature point back-projection and optical flow trajectory verification. Neighborhood feature point modeling is completed based on 3-5 anchor point pairs in the matching area, and time-varying parameter updates are implemented. Finally, a local affine transformation is formed, and non-rigid correction is performed on this basis to complete the local affine transformation. <f1>The goals of multi-source data fusion | dynamic reprojection | bidirectional information mapping.

[0047] like Figure 2 As shown, the scene types include snow slopes, texture-rich areas and exposed rock walls, representing weak texture and strong texture environments respectively. The states are classified into three types: the first category is the state before optimization, represented by a hollow circle in the figure; the second category is the state after optimization, represented by a solid circle; the third category is the benchmark performance, represented by a diamond. The specific data are as follows: in the snow slope scene, the anchoring success rate in the state before optimization (circle mark) is 58.7%, and the state after optimization (solid circle mark) is increased to 93.8%; in the exposed rock wall scene, the state before optimization is 65.2%, and the state after optimization is increased to 93.7%; in the texture-rich area, as the benchmark performance, the anchoring success rate reaches 98.5% (diamond mark).

[0048] like Figure 3 As shown in the figure, first, in the UAV image processing module, the system acquires and preprocesses the continuous image sequence, and then tracks the identifiable feature point trajectory through the optical flow algorithm, and then extracts the dynamic skeleton and transmits the image dynamic skeleton to the fusion module. At the same time, the LiDAR point cloud processing module is responsible for acquiring the LiDAR point cloud data, and combines the low-precision POS information for preliminary geo-reference and sparseness to identify the sparse feature points to form the skeleton, and then transmits the LiDAR general skeleton to the fusion core module; the fusion core module first back-projects the sparse three-dimensional feature points based on the low-precision pose, compares them with the optical flow features in the image space, and then establishes a dynamic anchoring relationship. On this basis, a time-varying affine transformation model is constructed. The fusion core module then requests the original point cloud local data, and the LiDAR module responds and returns the corresponding original point cloud, so that the fusion module can perform dynamic reprojection of the LiDAR point cloud, and finally completes the dynamic fusion of the LiDAR point cloud and the UAV image.

[0049] Example 4: In a typical high-risk early warning response mission for landslide disasters, the local emergency management department dispatched and deployed a ground-air-algorithm integrated emergency remote sensing platform based on the technical solution of the present invention. The deployment location was along a cliff section of a mountainous road in Yunnan. The terrain slope in this area exceeded 45 degrees, the landform was undulating, and there were significant obstructions and complex shadow areas. Conventional optical remote sensing technology was limited by imaging distortion and texture loss, making it difficult to obtain continuous and reliable three-dimensional model data. Especially under continuous rainy conditions, the traditional three-dimensional modeling method based on single image difference had large errors and could not meet the requirements of high precision and high timeliness in emergency rescue. Urgent demand for terrain information; in this application scenario, the system carries out data collection and fusion processing operations according to the following process: First, the UAV acquires high-resolution image data at a time interval of 0.5 seconds according to the preset route, and simultaneously triggers the onboard LiDAR module to collect point cloud data. The image and point cloud data are recorded with absolute timestamps through the high-precision time synchronization module to ensure that the subsequent alignment accuracy is controlled within ±5 milliseconds. The synchronization accuracy requirement is set based on the delay evaluation of multiple groups of typical tasks in the early stage and the actual response time statistics of the system, taking into account the performance upper limit of the embedded timing module and the average fluctuation range of image cache processing.

[0050] In the initial modeling stage, in order to solve the problem of incomplete image information caused by cliffs, obstructing depressions, etc. in the mountain area, the system introduces a terrain adaptive anchoring mechanism. Specifically, the system first extracts a set of spatial points with a normal vector mutation rate of more than 12% from the collected LiDAR point cloud as a candidate set of suspected boundary points. The numerical threshold is derived from the analysis results of the evolution process of the geometric characteristics of the typical landslide precursor area, and is dynamically weighted in combination with the local elevation gradient in the current mission scene to effectively avoid misjudgment caused by interference points such as vegetation and vehicle bodies; the extracted boundary points are double-weighted screened by local curvature and point density. Back-projection is performed onto the corresponding synchronized image frame to form an anchor candidate projection area. Within this area, the system improves the effectiveness of subsequent feature point extraction by performing brightness histogram widening and image detail enhancement operations. The system then uses a threshold-adjusted Harris corner detection method (the threshold is set to 70% of the standard value) to extract initial feature points from the enhanced image and calculate their optical flow trajectory in combination with the previous and next frame images. If a feature point successfully completes optical flow tracking in three consecutive frames, and its motion direction changes by less than 15 degrees and its modulus changes by no more than 10%, the system confirms that the area has the conditions for establishing a dynamic anchor relationship. After establishing the anchor relationship, the system constructs a local affine transformation model based on the selected anchor point pairs on each frame, selecting anchor point pairs with a spatial angle greater than 20 degrees to avoid numerical ill-conditioning of the affine matrix, aiming to avoid numerical ill-conditioning caused by linear correlation and ensure that the model solution has sufficient physical separation.

[0051] In order to solve the problem of texture information loss caused by the coexistence of shadow areas and bright spots in the scene, the system introduces a geometric prior guidance mechanism. The specific operation is: when the number of extractable feature points in a certain area is less than one thousandth of the total number of pixels (that is, there are less than 100 valid corner points in every 100,000 pixels), the system automatically calls the 3D point cloud information corresponding to the area for auxiliary positioning and guides feature recovery. Specifically, it selects points in the point cloud with an elevation range of more than 1.5 meters and a normal vector angle change of more than 8 degrees as auxiliary anchor points, and constructs the area to be enhanced based on them; after the auxiliary anchor points are back-projected into the image space, a 25 pixel × 25 pixel rectangular region of interest (ROI) is constructed with the pixel where they are located as the center, and optical flow initialization and motion trajectory stability verification are performed in it. When the trajectory is connected When the offset between consecutive frames is less than 2 pixels and the angle between its movement direction and the normal vector of the point is less than 30 degrees, the system will include the anchor point in the valid anchor point set; after completing the anchoring and registration of all image frames and point cloud data, the system performs dynamic image projection and texture assignment processing on the original point cloud based on the above-mentioned affine transformation model. In this process, the system not only projects each 3D point to the pixel position of the corresponding image frame, but also constructs a fusion texture layer based on its laser intensity value, spatial elevation and image grayscale to achieve pseudo-color rendering to enhance the visual recognition ability of landform boundaries, landslide deposits, cracks and other structures. At the same time, for some point cloud areas that cannot be projected due to occlusion or high reflection, the system supplements the texture information through the interpolation mechanism between three adjacent frames of images to ensure that the coherence of the final fused image in the time series is not less than 90%. To verify the technical feasibility and effectiveness of the above-mentioned fusion modeling process, the system simultaneously collected traditional SFM modeling + texture mapping results under the same route at the above-mentioned deployment site for comparative analysis. The test results show that the feature recovery rate in areas where texture information is missing is improved by more than 60%; and the overall data collection to modeling completion time is controlled within 2 hours, which is significantly better than the existing mainstream methods that generally have a processing cycle of more than 6 hours.

[0052] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A LiDAR and UAV image fusion processing method applied to complex mountainous areas, characterized by: The method comprises the following steps: Step a: Acquire a sequence of continuous images captured by the drone in the target area and track the motion trajectories of identifiable feature points between consecutive image frames using a sparse optical flow algorithm. Based on the dominant motion parameters between the image frames, select optical flow feature points that can be stably tracked in multiple consecutive frames and whose deviation between the local motion vector and the motion trend represented by the dominant motion parameter is less than a preset deviation threshold, thereby forming an image dynamic skeleton. Step b: obtaining LiDAR point cloud data collected contemporaneously with the image sequence and low-precision positioning and orientation system data for preliminary positioning; georeferencing the LiDAR point cloud data using the low-precision positioning and orientation system data, and performing voxel downsampling on the LiDAR point cloud data to extract sparse three-dimensional feature points based on normal vector change rates or local elevation extremes to form a LiDAR skeleton; Step c: Based on the camera pose provided by the low-precision positioning and orientation system data, the sparse three-dimensional feature points in the LiDAR skeleton are reversely projected onto the image frames in the image sequence to form a projection area of the sparse three-dimensional feature points on the image; within the projection area, a sequence of optical flow feature points is searched for that has the shortest distance to the sparse three-dimensional feature points in three-dimensional space and whose optical flow motion trajectory stability meets a preset stability standard, and a dynamic anchoring relationship is established between the sparse three-dimensional feature points and the sequence of optical flow feature points; Step d, based on the dynamic anchoring relationship, for the sparse three-dimensional feature points and their anchored optical flow feature point sequences, combined with the three to five pairs of feature point pairs with established dynamic anchoring relationships around them, a local affine transformation model that changes with time is constructed; using the local affine transformation model, the original point cloud corresponding to the sparse three-dimensional feature point neighborhood in the LiDAR point cloud data is dynamically reprojected onto the image frame corresponding to the dynamic anchoring relationship.

2. The method for fusion processing of LiDAR and UAV images applied to complex mountainous areas according to claim 1, characterized in that: The method for selecting optical flow feature points in step a is as follows: after tracking the motion trajectory of the identifiable feature points, the dominant inter-frame motion parameters caused by the drone's own motion between consecutive image frames are estimated; for each tracked feature point, the vector difference between its local motion vector and the motion vector expected at the feature point position based on the dominant inter-frame motion parameters is calculated; when the modulus of the vector difference is Greater than the preset deviation threshold , the feature point is identified as a motion inconsistent point and is excluded when screening stable optical flow feature points or establishing dynamic anchoring relationships.

3. The method for fusion processing of LiDAR and UAV images applied to complex mountainous areas according to claim 1 or 2, characterized in that: The method for tracking the motion trajectory of the identifiable feature points in step a is to use the KLT optical flow algorithm.

4. The method for fusion processing of LiDAR and UAV images applied to complex mountainous areas according to claim 1, characterized in that: Prior to step a, the method further includes: performing a three-dimensional geometric structure significance analysis on each point in the LiDAR point cloud data or its local neighborhood based on the LiDAR point cloud data, calculating the normal vector change rate, curvature or elevation standard deviation of the point cloud; and identifying geometric structure significant areas based on the condition that the three-dimensional geometric structure significance metric value is higher than a specific threshold.

5. The method for fusion processing of LiDAR and UAV images applied to complex mountainous areas according to claim 4, characterized in that: The geometrically salient areas are back-projected onto the image frames in the continuous image sequence using the low-precision positioning and orientation system data to calibrate the corresponding virtual anchor potential areas within the image frames; and when executing step a to extract optical flow feature points or executing step c to establish a dynamic anchor relationship, for the image area falling within the virtual anchor potential area, the local three-dimensional geometric structure saliency information of the LiDAR point cloud is referenced to guide or strengthen the extraction of optical flow feature points or the establishment of a dynamic anchor relationship.

6. The method for fusion processing of LiDAR and UAV images applied to complex mountainous areas according to claim 1, characterized in that: The method for establishing a dynamic anchoring relationship in step c is: for each sparse three-dimensional feature point, within the projection area of its reverse projection onto the image frame, search and select the optical flow feature point with the shortest distance to it in the three-dimensional space and whose optical flow motion trajectory stability between consecutive frames meets the preset stability standard to form a dynamic anchoring relationship.

7. The method for fusion processing of LiDAR and UAV images applied to complex mountainous areas according to claim 1, characterized in that: The continuous image sequence is a video stream of five to ten frames per second or high-frequency continuous shooting photos.

8. The method for fusion processing of LiDAR and UAV images applied to complex mountainous areas according to claim 1, characterized in that: The invention relates to a method for extracting sparse three-dimensional feature points from LiDAR point cloud data, including normal vector mutation detection or local elevation extreme value extraction based on LiDAR point cloud.

9. The method for fusion processing of LiDAR and UAV images applied to complex mountainous areas according to claim 1, characterized in that: After step d, it further includes mapping the elevation or intensity information carried by the dynamically reprojected LiDAR point cloud data to the matching pixels on the corresponding image frame; or, superimposing the spectrum or texture information of the image frame to its corresponding original three-dimensional point cloud.

Images