Rapid three-dimensional reconstruction method for unmanned aerial vehicle mine inspection scene

Abstract

The application discloses a rapid three-dimensional reconstruction method for unmanned aerial vehicle mine inspection scene, and particularly relates to the technical field of image processing and three-dimensional modeling, and comprises four steps of environment perception collection, inclination adaptive path planning, image quality optimization and three-dimensional model construction; the shooting strategy is dynamically adjusted through sensor perception of information such as illumination, wind speed and slope, and a normal line supplementary shooting path is generated in a slope mutation area; image enhancement and feature extraction are adopted to improve the image quality, and a confidence score model is constructed to screen high-quality images for modeling; the modeling result is subjected to confidence evaluation, a supplementary shooting mechanism is triggered, and the model precision and integrity are improved; the application improves the feature extraction stability of images under the condition of dramatic change of illumination, enhances the image coverage integrity of high and steep slope areas, and realizes stable control of image collection and fusion in a wind disturbance environment, so that the precision, continuity and robustness of three-dimensional modeling are effectively improved.

Description

Technical Field

[0001] This invention relates to the fields of image processing and 3D modeling technology, and more specifically, to a rapid 3D reconstruction method for unmanned aerial vehicle (UAV) mine inspection scenarios. Background Technology

[0002] In mine inspections and slope stability analyses, using drones to acquire images of target areas and generating 3D models through image reconstruction algorithms has become an important data acquisition method. In the area of ​​image processing and generation, 3D modeling methods mainly include image data processing sub-steps such as image registration, feature extraction, image fusion, depth estimation, sparse or dense point cloud generation, mesh reconstruction, and texture mapping.

[0003] However, in applications targeting steep slopes in open-pit mines, existing 3D reconstruction methods face several technical bottlenecks:

[0004] Illumination interference affects the consistency of image reconstruction: The terrain of mine slopes is complex, with a large number of areas with direct sunlight, backlight, strong reflection and alternating shadows. This results in significant differences in the brightness distribution of the same area at different times, which significantly reduces the accuracy of feature point-based registration and image fusion, and seriously affects the structural restoration and texture consistency of the subsequent 3D model.

[0005] Abrupt changes in slope structure lead to blind spots in image coverage: When the slope angle changes drastically or there are structural features such as steps, faults, or vertical rock surfaces, UAVs are often unable to obtain a viewpoint image of the slope normal angle due to flight attitude control limitations and fixed flight path planning. This results in missing images of key areas, structural voids or deformations in the 3D model, making it difficult to meet the requirements of safety assessment.

[0006] Wind disturbances cause dynamic instability between image frames: Because the inspection area is located in an open mountainous wind gap, frequent wind speed fluctuations can easily cause slight jitter in the drone's flight attitude, leading to misalignment, ghosting, or slight blurring in the acquired images between frames. Traditional 3D modeling processes struggle to automatically identify such unstable images, resulting in misregistration and artifact propagation problems. Therefore, this invention proposes a rapid 3D reconstruction method for drone-based mine inspection scenarios to meet the 3D modeling needs of image processing tasks in complex mine slope environments. Summary of the Invention

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A rapid 3D reconstruction method for unmanned aerial vehicle (UAV) mine inspection scenarios includes the following steps:

[0009] Environmental perception and data acquisition steps: The drone collects information on light intensity, wind speed, and slope in the current inspection path area using multiple sensors, and dynamically adjusts the shooting rhythm and shooting locations based on light stability and wind disturbance level.

[0010] Tilt-adaptive path planning steps: The shooting path is generated in real time based on the perceived slope tilt angle. When the tilt angle exceeds the set angle threshold, the path reconstruction mechanism is triggered to automatically adjust the flight attitude and generate a tilt reshoot path in the normal direction to improve the image coverage of steep slope areas.

[0011] Image quality optimization steps: The acquired image is subjected to multi-scale texture enhancement and brightness normalization, robust feature points are extracted, and low-stability frames caused by wind disturbance are removed by combining a multi-frame fusion strategy to form an image confidence scoring model.

[0012] In a preferred embodiment, the three-dimensional model construction steps are as follows: based on the evaluation results of the image confidence scoring model, structural reconstruction and image registration are performed on images that are higher than the modeling confidence threshold, and a reconstruction confidence judgment mechanism is introduced during the model construction process to obtain the modeling confidence value of the local area of ​​the model. When it is lower than the acceptable reconstruction threshold, re-photographing is performed to complete the generation of the three-dimensional model of the entire area.

[0013] In a preferred embodiment, an image frame-level stability detection mechanism is further introduced in the environmental perception acquisition step. The stability index Stab is calculated by the feature point displacement vector between consecutive images. This index is generated based on the average consistency of the optical flow field in the image sequence. When the value of the stability index Stab is lower than the preset dynamic stability discrimination threshold, the corresponding image frame is suspended from participating in subsequent modeling.

[0014] In a preferred embodiment, the stability index Stab is obtained by normalizing the optical flow consistency estimated using the Lucas-Kanade method, or by using a method based on bidirectional optical flow reprojection error calculation.

[0015] In a preferred embodiment, during the tilt angle adaptive path planning step, when the slope tilt angle change is detected to exceed the adaptive judgment angle threshold, the supplementary shooting area prediction calculation is automatically performed. The optimal shooting tilt angle is derived in the local fitting plane using a digital elevation model combined with the slope geometric normal fitting method. Based on the principle of minimum overlap blind zone, a slope-fitting path is generated. This path selects a flight trajectory perpendicular to the normal at the location of the slope change.

[0016] In a preferred embodiment, during the generation of the flight path that is attached to the slope path, the shooting angle satisfies the optimization objective of having an deviation angle of less than 5 degrees from the slope's central axis normal.

[0017] In a preferred embodiment, an illumination-robust feature extraction mechanism is introduced in the image quality optimization step. This mechanism uses local feature operators with illumination invariance when performing feature extraction algorithms after multi-scale image enhancement. The algorithm used is any one of AKAZE, ORB, or SuperPoint.

[0018] In a preferred embodiment, before structural reconstruction and image registration, images are screened by constructing an image confidence scoring model. The image confidence level is derived by considering factors such as the number of feature points, the spatial distribution balance of feature points, and image stability. If the score result is higher than the image confidence judgment threshold, the image is classified as a high-confidence image. Specifically, the scoring model uses a preset normalized feature point density index, a structural similarity index, and an image stability index for weighted fusion, and is ultimately used to determine the confidence level that the image should be assigned in 3D reconstruction. The confidence levels are high-confidence image and low-confidence image.

[0019] In a preferred embodiment, during the 3D model construction step, a region modeling confidence map is constructed based on the point cloud density gradient change, matching residual, and feature point redundancy in the reconstruction results. This map is used to color-code the modeling confidence level of local regions of the model. If the modeling confidence value of a local region is lower than the acceptable reconstruction threshold, the region is marked as a region that needs to have its image resampled.

[0020] In a preferred embodiment, during the environmental perception acquisition step, a wind disturbance stability factor is calculated by combining the slope openness, the current wind speed vector, and the UAV flight attitude. The wind disturbance stability factor is calculated based on the extended Kalman filter method. If the factor is higher than the wind disturbance intensity threshold, the shooting interval during flight is adjusted to the longest interval time so that the minimum inter-frame displacement is maintained when acquiring images in high wind areas.

[0021] In a preferred embodiment, in the image quality optimization step, a sequence of images under a preset time window is acquired, and a consistency vector is constructed using the mean square error of the gray-level difference rate between adjacent frames and the matching repetition rate of feature matching. The Euclidean distance between the consistency vector and the preset standard vector is used as the fusion score. If the fusion score is higher than the fusion threshold, the sequence of images is subjected to multi-frame average fusion; otherwise, the highest quality frame in the sequence of images is selected for modeling according to a preset image selection strategy.

[0022] The technical effects and advantages of this invention are as follows:

[0023] This invention introduces an illumination-robust feature extraction mechanism, combining multi-scale image enhancement with illumination-invariant local feature operators (such as AKAZE, ORB, and SuperPoint) in the image quality optimization step. This significantly improves the feature extraction and description capabilities of images in environments with alternating strong light and shadow. This mechanism avoids feature matching failures caused by uneven brightness distribution, effectively improving the registration accuracy between images and the stability of continuous modeling. Therefore, it overcomes the problem of unstable modeling accuracy in complex lighting scenarios in mines, a problem inherent in traditional methods.

[0024] In the tilt-adaptive path planning step, this invention automatically triggers a reshoot area prediction mechanism based on changes in the slope tilt angle. It then combines a digital elevation model (DEM) with a normal fitting algorithm to generate a "slope-fitting path" aligned with the slope geometry. This ensures the camera's shooting angle is as close as possible to the slope's normal direction, and the shooting deviation angle is controlled to be less than 5 degrees during flight path generation. This path planning mechanism effectively improves image coverage integrity on steep slopes or areas with abrupt structural changes, avoids modeling blind spots caused by overhead shooting, and enhances the model's geometric reconstruction capability and global structural continuity.

[0025] This invention calculates a wind disturbance stability factor using an extended Kalman filter in the environmental perception acquisition step, dynamically adjusting the shooting interval. When the wind disturbance intensity exceeds a preset threshold, the acquisition period is automatically extended to reduce inter-frame displacement. Simultaneously, during image quality optimization, a consistency vector is constructed by combining the image grayscale difference rate and the mean square error of the matching repetition rate. Using a fusion score as the criterion, multi-frame averaging fusion is performed only on high-consistency image sequences. This series of mechanisms works synergistically to effectively suppress image jitter, misalignment, and fusion artifacts in wind-affected environments, ensuring the stability of the image input and the robustness of the fusion output, thereby improving modeling accuracy and image utilization efficiency. Attached Figure Description

[0026] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings;

[0027] Figure 1 This is a schematic diagram of the rapid 3D reconstruction method for unmanned aerial vehicle (UAV) mine inspection scenarios in this invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0029] Reference Figure 1The following examples were obtained:

[0030] Example 1: In a mixed mining area combining open-pit and underground mining, there are numerous steep slopes formed by artificial excavation or natural weathering. This area has a discontinuous topography, drastic changes in slope angle, with some areas exhibiting steep inclines exceeding 45 degrees, and multiple stepped structures, making it a typical high-risk inspection area. During routine inspections, due to varying slope orientations, there are often dramatic changes in lighting conditions, such as simultaneous strong direct sunlight and deep shadows, leading to severe brightness contrasts during image acquisition. This makes conventional image feature point extraction and registration prone to failure, thus affecting the stability of 3D modeling. Furthermore, due to the rugged terrain, drones are limited by flight angle and safe altitude, making it difficult to cover all slope normal angles, resulting in blind spots in modeling data coverage. Simultaneously, this area is located at a windy valley entrance, where wind speed changes frequently and direction is unstable. Slight shaking or even attitude shifts often occur during flight, leading to unstable inter-frame image matching and texture misalignment or artifacts.

[0031] Therefore, this invention proposes a rapid 3D reconstruction method for unmanned aerial vehicle (UAV) mine inspection scenarios, comprising the following steps:

[0032] Environmental perception and data acquisition steps: The drone collects information on light intensity, wind speed, and slope in the current inspection path area using multiple sensors, and dynamically adjusts the shooting rhythm and shooting locations based on light stability and wind disturbance level.

[0033] Tilt-adaptive path planning steps: The shooting path is generated in real time based on the perceived slope tilt angle. When the tilt angle exceeds the set angle threshold, the path reconstruction mechanism is triggered to automatically adjust the flight attitude and generate a tilt reshoot path in the normal direction to improve the image coverage of steep slope areas.

[0034] Image quality optimization steps: The acquired image is subjected to multi-scale texture enhancement and brightness normalization, robust feature points are extracted, and low-stability frames caused by wind disturbance are removed by combining a multi-frame fusion strategy to form an image confidence scoring model.

[0035] The three-dimensional model construction steps are as follows: Based on the evaluation results of the image confidence scoring model, structural reconstruction and image registration are performed on images that are higher than the modeling confidence threshold. A reconstruction confidence judgment mechanism is introduced during the model construction process to obtain the modeling confidence value of the local area of ​​the model. When it is lower than the acceptable reconstruction threshold, re-photographing is performed to complete the generation of the three-dimensional model of the entire area.

[0036] In the environmental perception acquisition step, an image frame-level stability detection mechanism is further introduced. A stability index, Stab, is calculated using the feature point displacement vectors between consecutive images. This index is generated based on the average consistency of the optical flow field in the image sequence. When the Stab value falls below a preset dynamic stability threshold, the corresponding image frame is suspended from participating in subsequent modeling. The Stab index is obtained either by estimating optical flow consistency using the Lucas-Kanade method and then normalizing it, or by using a method based on bidirectional optical flow reprojection error calculation.

[0037] The image frame-level stability detection mechanism is used to automatically determine the visual stability between consecutive images during the image acquisition stage, so as to avoid using unstable image frames caused by wind disturbance, micro-drift of flight attitude or vibration for 3D modeling, and ensure the basic quality of the input images for modeling.

[0038] The detection process is as follows: Image sequence input: Continuously acquire image frame sequences A1, A2, A3, ..., A n The current frame is selected sequentially and compared with its adjacent previous frame to form a comparison pair (A). i-1 A i ), where i represents the number and n is the total quantity;

[0039] Feature point extraction and matching: For each frame of the image, several image feature points with high stability are extracted (corner detectors such as FAST and Shi-Tomasi can be used) to describe the motion consistency between frames. The extracted feature points retain the spatial distribution uniformity as the selection criterion to prevent misjudgment caused by local concentration.

[0040] Displacement vector calculation: For each pair of image frames, based on the feature points in the image sequence, the displacement relationship in the time dimension is estimated through the optical flow field, forming the displacement vectors Δx and Δy for each feature point. This displacement vector is used for subsequent calculation of stability indices.

[0041] The stability indicator Stab is calculated as follows:

[0042] Method 1: Estimating optical flow consistency based on the Lucas-Kanade method and then normalizing it:

[0043] Using the Lucas-Kanade optical flow method, a minimum squared error model with invariant brightness is constructed in the neighborhood of feature points. Local pixel displacements between image frames are calculated, the optical flow vector set of all feature points is collected, and its standard deviation σ_flow is calculated. The average optical flow consistency at all points is defined as Stab_pre. Finally, Stab_pre is normalized to the interval [0,1] to obtain the stability index Stab.

[0044] For example, if the optical flow vectors of most feature points in a set of image frames are similar in direction and their magnitudes do not change much, it indicates that the relative motion between frames is stable and the Stab value is close to 1; conversely, if the optical flow difference is large, the Stab value will decrease.

[0045] Method 2: Based on the bidirectional optical flow reprojection error calculation method, the following is obtained:

[0046] In image frame A i and A i-1 Calculate the forward optical flow (A) between them. i-1 A i ) and reverse optical flow (A i →A i-1 This method obtains a bidirectional set of matching points. For each feature point, a closed-loop path is formed in the original image using forward and reverse paths. The Euclidean distance between the starting and returning positions of the point is calculated as the reprojection error. The mean or median of the reprojection errors of all matching points is used as the error metric for the current frame. After normalization, this becomes the stability index, Stab. This method can significantly suppress the pseudo-steady-state problem caused by mismatches due to unidirectional errors, and is particularly suitable for frame pair analysis with significant wind disturbances or attitude fluctuations.

[0047] Let there be a dynamic stability threshold, denoted as T. s This threshold can be derived from historical data statistics based on the scene environment, or set by model training, for example, T. s A value of 0.75 indicates that the Stab metric for a given frame of an image is stable. <T s The image frame is marked as an unstable frame; this frame will be suspended from participating in the subsequent image registration and 3D modeling process and removed from the current modeling task, and will only be retained in the original image library for manual review or subsequent reshoot plan analysis; this operation ensures that the image frames entering the image quality optimization and 3D modeling steps have sufficient temporal consistency, displacement controllability and visual stability.

[0048] For example, in a slope protection drone operation, 120 frames of images were continuously acquired, and the Stab was calculated frame by frame using the method described above. The results showed that frames 37 and 88 had significantly higher average reprojection errors than other frames due to sharp attitude shifts caused by strong winds, with Stab values ​​of 0.52 and 0.47 respectively, far below the preset threshold of 0.75. These two frames were automatically set as "rejected frames" and not included in the 3D modeling process, thus effectively avoiding texture misalignment and abnormal structural diffusion.

[0049] In the slope-adaptive path planning step, when a change in slope inclination angle exceeds the adaptive threshold, the system automatically performs prediction calculations for the reshoot area. Using a digital elevation model combined with slope geometric normal fitting, the optimal shooting angle is derived from the local fitted plane. Based on the principle of minimum overlap blind zone, a slope-fitting path is generated. This path selects a flight trajectory perpendicular to the normal at locations of abrupt slope changes. In the generation of the flight path (i.e., the slope-fitting path), the shooting angle is optimized to have a deviation angle from the slope's central axis normal of less than 5 degrees.

[0050] In the slope adaptive path planning step, when the slope inclination angle change is detected to exceed the adaptive judgment angle threshold, the system automatically performs supplementary image prediction calculation. This supplementary image mechanism is mainly used to deal with uneven image coverage and blind spots in the modeling area caused by drastic slope changes, and is particularly suitable for mine slopes with stepped or cliff-like structures.

[0051] Angle threshold judgment: The UAV system uses sensors to obtain the slope inclination angle on the current flight path in real time and compares it with the angle between the slope and the flight direction. When a sudden change in slope is detected and the angle change exceeds the set adaptive judgment angle threshold (e.g., it can be set to 20 degrees, 25 degrees or other standards set experimentally in other scenarios), it is considered that there is a potential modeling blind spot or insufficient shooting coverage in the area, triggering subsequent path reconstruction actions.

[0052] Reshoot Area Prediction Calculation: The system first calls upon the region's Digital Elevation Model (DEM), which can be established through LiDAR scanning, photogrammetry, or historical data, representing changes in surface elevation. Upon identifying areas with significant elevation differences or abrupt changes in slope structure, the system constructs a local geometric plane using a normal fitting method based on the spatial distribution of points on the slope surface. Analysis is then performed within this fitted plane to deduce the optimal shooting angle. The normal fitting method can employ point set fitting to the plane, such as the least squares method for local plane approximation, thus obtaining the current slope normal vector, denoted as Np. The optimal shooting angle is then calculated based on the UAV's current flight attitude, ensuring the camera's imaging plane is as orthogonal as possible to the slope.

[0053] Flight Path Generation (Slope-Following Path): After the local shooting angle is determined, the path planning module generates a "slope-following path" based on the principle of minimum overlap blind zone. The minimum overlap blind zone refers to minimizing the area where image reconstruction cannot be effectively unfolded due to angular deviations, while still achieving the desired image acquisition overlap rate. To this end, the system prioritizes generating a flight trajectory aligned with the slope normal, ensuring the camera remains approximately perpendicular to the local slope during flight. Specifically, at locations with abrupt slope changes (e.g., a rock face changing instantaneously from 35 degrees to 60 degrees), to avoid image tilt distortion caused by angled shooting, the system switches its flight attitude at that location, minimizing the angle between the UAV's heading and the slope normal, thus generating a flight trajectory perpendicular to the slope normal at that section.

[0054] Shooting Angle Control Objective: During the generation of the flight path (i.e., the path that closely matches the slope), the system further optimizes the shooting angle to ensure that the deviation angle between the camera's optical axis and the slope's central axis normal does not exceed a set error range. Specifically, the system uses the slope's central axis normal as the reference direction and aims to control the shooting angle deviation to within 5 degrees. This control strategy effectively improves the orthorectifiability and consistency of image features, thereby enhancing the accuracy of subsequent image registration and 3D modeling.

[0055] Engineering Application Example: For instance, during a drone inspection of the southwest slope of an open-pit mine, the system detected a sudden transition from a gentle slope to a near-vertical rock surface in a certain area, with a local angle change exceeding 30 degrees, surpassing the system's set angle threshold of 25 degrees. The system immediately accessed the area's elevation map, performed geometric analysis on the slope section, extracted a set of points for plane fitting, and calculated the principal normal direction of the area. Subsequently, based on the normal direction, the system generated a set of supplementary flight paths perpendicular to the slope normal, ensuring the angle between the camera's shooting angle and the normal direction was controlled within 3.2 degrees. The final generated slope-hugging path completely covered the abrupt change area, effectively compensating for the modeling occlusion problem caused by the main path's overhead view.

[0056] The "flight path" here is mainly used for reshooting. It is a "second reshoot path" triggered by a sudden change in angle. It is not a globally preset main path, but it is part of the acquisition path system. Therefore, it can be considered as: "a secondary dynamic acquisition path triggered by a specific slope inclination change to make up for the image blind spots caused by insufficient coverage of the first acquisition path."

[0057] The flight path in this invention is a supplementary shooting path automatically generated based on abrupt changes in slope angle during the tilt-adaptive path planning step. Unlike the globally acquired path, it is specifically designed to compensate for image blind spots. During path generation, combining a digital elevation model (DEM) with the slope normal direction, existing algorithms such as RRT, A* combined with elevation grids, and polynomial attitude optimization are used to achieve stable flight trajectory planning along the slope. Simultaneously, it ensures that the shooting angles within the flight path meet the optimization target of deviating from the slope's central axis normal angle by less than 5 degrees, thereby improving image coverage and 3D reconstruction accuracy. The advantage of this invention lies in its real-time perception, path reconstruction, and supplementary shooting path execution when abrupt changes in slope slope occur. It features a path planning mechanism with "intelligent perception + local reconstruction + local optimized shooting," creatively combining path generation with depth geometric elements such as slope gradient changes and normal direction alignment, representing an improvement over traditional flight path planning.

[0058] The RRT algorithm is used for obstacle avoidance and free space path search. Its features include: being suitable for generating feasible paths in environments with complex slopes, obstructions, and significant terrain changes; its connection with this invention is: it can be used to explore preliminary trajectories for generating "slope-hugging paths" from the main path; it can construct supplementary flight paths leading to the normal direction in areas with abrupt slope changes; and it can incorporate flight safety constraints (such as minimum flight altitude and attitude change range).

[0059] The A+DEM rasterized path planning algorithm is used for: shortest flyable path planning combined with elevation data; features: suitable for two-dimensional + elevation fitting path generation based on digital elevation model (DEM) data; integration with this invention: converting slope DEM into raster map, setting "heavy weight areas" at abrupt slope changes; automatically generating supplementary shooting path segments to avoid blind spots in the main path; can be accompanied by flight attitude control, setting the target shooting angle to "perpendicular to the local slope normal".

[0060] The polynomial trajectory optimization algorithm is used for: smoothing optimization of UAV flight paths and attitude control; its features include: controlling the camera orientation to always point to a target direction (such as the normal direction) within a certain path segment; its connection with this invention is: after fitting the generated slope path, optimizing the camera attitude to keep its deviation angle less than 5°; generating "multi-segment stable shooting trajectories" in multi-angle reconstruction areas to improve the continuity of model reconstruction; and coordinating the control of flight speed / attitude changes with image acquisition intervals.

[0061] In the image quality optimization step, an illumination-robust feature extraction mechanism is introduced. This mechanism uses local feature operators with illumination invariance when performing feature extraction algorithms after multi-scale image enhancement. The algorithm used is any one of AKAZE, ORB, or SuperPoint.

[0062] Before structural reconstruction and image registration, images are screened by constructing an image confidence scoring model. The image confidence level is derived by considering factors such as the number of feature points, the spatial distribution balance of feature points, and image stability. If the score result is higher than the image confidence judgment threshold, the image is classified as a high-confidence image. Specifically, the scoring model uses a pre-set normalized feature point density index, a structural similarity index, and an image stability index for weighted fusion. This is ultimately used to determine the confidence level that the image should be assigned in 3D reconstruction, with the confidence level being either a high-confidence image or a low-confidence image.

[0063] In the image quality optimization step, an illumination-robust feature extraction mechanism is introduced. This mechanism aims to address the problems of unstable image feature extraction and increased matching error rate in scenes with drastic illumination changes, such as strong direct sunlight and interlacing shadows, within the inspection area. Before the image enters subsequent modeling processing, preprocessing enhancement and feature extraction are used together to improve the adaptability and consistency of image descriptors to illumination changes.

[0064] Multi-scale image enhancement: After the image enters the optimization step, the system first performs multi-scale image enhancement processing, constructing different scale versions of the image using Gaussian pyramid or Laplacian pyramid methods. In each scale image, the system performs local contrast stretching and brightness normalization to suppress the interference of local overexposed or underexposed areas caused by strong light and shadow on feature detection. After processing, the images at each scale are fused back to the main scale for feature extraction.

[0065] Illumination-robust feature extraction: When executing the feature extraction algorithm, the system selects local feature operators with illumination invariance or local texture preservation capabilities, including any one of the AKAZE (accelerated KAZE), ORB, or SuperPoint algorithms. The characteristics of each algorithm are as follows:

[0066] AKAZE: Based on nonlinear scale space for key point detection, it uses the Modified Local Difference Binary descriptor and has good adaptability to illumination and blur.

[0067] ORB: Combining FAST corner points and BRIEF descriptors, it incorporates orientation encoding to improve rotation invariance, resulting in high stability in fast applications.

[0068] SuperPoint: Employs neural networks for end-to-end feature point detection and description, exhibiting high robustness in scenarios involving lighting and texture loss.

[0069] The algorithm configuration mechanism is as follows: I. Flight Platform Performance: Flight platform performance mainly refers to the hardware parameters of the UAV, such as processor capabilities (e.g., whether it has GPU acceleration), RAM capacity, image sensor frame rate, and resolution. The system sets the preferred algorithm according to the following rules: Resource-constrained platforms (no GPU / ARM chip / low-power device): ORB algorithm is preferred. Its feature detection and description are lightweight calculations, suitable for scenarios with high real-time requirements and limited computing power. Mid-to-high-end platforms (with medium computing power): AKAZE algorithm can be selected. It supports non-linear scale spatial feature extraction, has higher illumination robustness, and is suitable for scenarios with medium texture complexity. High-performance platforms (with GPU / with deep model deployment capabilities): SuperPoint algorithm is preferred. It is an end-to-end neural network model with extremely strong anti-interference capabilities and relocation accuracy, suitable for scenarios with extreme lighting changes or weak texture areas. Upon system startup, the system will detect and process platform information and automatically load the matching default algorithm. Users can also manually select or disable an algorithm in the parameter configuration interface.

[0070] Based on the second point regarding the texture characteristics of the inspection area: For different regions, the system can dynamically switch or prioritize algorithms based on the following indicators: If a large area in the image exhibits drastic changes in illumination, shadow occlusion, or strong reflections (such as metal surfaces or rock edges), AKAZE or SuperPoint is preferred; if low-texture areas exist in the image (such as dust or smooth rock walls), SuperPoint is prioritized because it enhances the feature perception capability of weak-texture areas through end-to-end learning; if the region has rich texture, uniform illumination, and limited computational resources, ORB can be used directly. The system can automatically determine the scene category of the current image through a preset texture complexity discrimination model (such as image grayscale change rate and local variance) and switch the most suitable feature extraction algorithm according to the corresponding strategy.

[0071] Image Confidence Scoring Model: Before structural reconstruction and image registration, the system executes an image confidence scoring model to screen images, preventing low-quality images from entering the registration process and reducing error propagation. The image confidence score comprehensively considers three key factors: the number of feature points, the spatial distribution balance of feature points, and image stability. Specific details are as follows:

[0072] Number of feature points: The total number of feature points successfully extracted from the image by the system. This reflects the richness of the local texture of the image. Too few points can easily lead to registration failure.

[0073] Spatial uniformity of feature point distribution: The image is divided into m×n regions, and the standard deviation of the number of feature points in each region is calculated. The lower the standard deviation, the more uniform the distribution of feature points, which can provide a wider range of registration anchor points and is beneficial to the completeness of modeling;

[0074] Image stability: The Stab stability index, derived from the image frame-level stability detection mechanism, reflects the motion consistency between the image and adjacent frames. The higher the value, the smaller the inter-frame jitter and the higher the availability.

[0075] The three factors mentioned above are normalized into a scoring vector in the interval [0,1], defined as follows: F1: Normalized feature point density index; F2: Distribution balance index (higher balance results in a higher score); F3: Image stability index (Stab value); The final image confidence score F_total = w1×F1 + w2×F2 + w3×F3. The three weights can be preset or optimized empirically within the system, for example, w1:w2:w3 = 0.4:0.3:0.3. It should be noted that the original definition of F2 is "distribution balance index", that is, whether the spatial distribution of feature points in the image is uniform. In fact, "the uniformity of the spatial distribution of feature points" is a question of "whether the image structural information is expressed in a balanced way", which is an indicator of the image's ability to express spatial structure, so it is also a structural similarity index.

[0076] If the score is higher than the image confidence threshold (e.g., set to 0.75), the image is classified as a high-confidence image. Otherwise, it is classified as a low-confidence image. The confidence levels are as follows: High-confidence images: directly participate in structural reconstruction and image registration, and are the main image source for modeling; Low-confidence images: do not directly participate in modeling, but can be recorded for reshoot prompts or as auxiliary references.

[0077] Implementation example: In a certain inspection mission, the system acquired 50 frames of images from a drone and performed confidence scoring analysis frame by frame. The 12th frame had 850 feature points, a spatial distribution standard deviation of 7.1, a Stab index of 0.93, and a normalized score of F_total = 0.82, exceeding the set confidence threshold of 0.75, and was classified as a high-confidence image. The 27th frame had only 412 feature points, concentrated on the left side of the image, with a Stab value of 0.61. Its calculated score was F_total = 0.59, classifying it as a low-confidence image. It was removed from the 3D modeling process and recorded as a potential re-enhancing area.

[0078] In the 3D model construction step, based on the point cloud density gradient change, matching residual, and feature point redundancy in the reconstruction results, a region modeling confidence map is constructed to color-code the modeling confidence level of local regions of the model. If the modeling confidence value of a local region of the model is lower than the acceptable reconstruction threshold, the region is marked as a region that needs to be resampled.

[0079] The following three key indicators are used to extract local modeling quality information from the 3D reconstruction results:

[0080] Point cloud density gradient variation: The number of points in each local region is statistically analyzed, and the rate of change (gradient) of its spatial distribution is calculated. Dramatic gradient changes usually indicate discontinuous, sparse, or reconstructed edge regions of the point cloud, resulting in low reliability. The system normalizes the density gradient variation to obtain a density stability score D. m .

[0081] Matching residual: refers to the mean or median of the reprojection error of key points during image registration, reflecting the matching accuracy of the image in that region. A high region matching residual indicates a large registration error, suggesting potential structural misalignment or texture mismatch in that region. This term is defined as the matching accuracy score R. e The lower the score, the greater the error.

[0082] Feature point redundancy: This indicates whether a region is covered by multiple images from different angles; the more coverage, the higher the redundancy. Redundancy can be obtained by calculating the number of image frames projected onto the region. A higher score indicates more sufficient data support, denoted as a coverage richness score C. r .

[0083] Modeling confidence map generation method: The above three indicators are uniformly normalized to the [0,1] interval, and then fused according to the following weighting method to obtain the regional modeling confidence value:

[0084] Modeling confidence value S = α·D m +β·(1-R e )+γ·C r ; α, β, and γ are configurable weighting factors (e.g., α = 0.4, β = 0.4, γ = 0.2); D m Indicates point cloud density uniformity; R e The residual value is inverted to satisfy the direction that "the larger the value, the more reliable it is"; C r This indicates the redundancy of feature point coverage. After summing the S-values ​​of all regions, the system renders the 3D model surface in pseudo-color image form, constructing a region modeling confidence map. Color coding can use a standard gradient from red (low) to green (high).

[0085] Reconstruction Acceptable Threshold and Supplementary Sampling Mechanism: A built-in reconstruction acceptable threshold (denoted as T) r For example, 0.65), as a dividing line for judging whether the modeling confidence value of a region is reliable; if the modeling confidence value S ≥ T of a certain region r If S , then the modeling of this region is considered reliable; if S <T r If the area is not captured, the system will mark it as an "image acquisition area that needs to be acquired", highlight it in red on the 3D model, and add it to the image acquisition task queue.

[0086] Implementation Example: In a mine slope modeling task, after completing the initial point cloud modeling, the system performs a modeling confidence analysis on the model area. The results show that the point cloud in the southwest corner of the model (region ID: R27) is sparse, has high matching residuals, and is only covered by two image frames. The system calculates the modeling confidence value for this region to be 0.58, which is lower than the set threshold of 0.65. Therefore, this region is displayed as an orange-red block in the modeling view, and the system marks it as an area requiring re-sampling. A re-shooting command is sent to the flight mission system, requesting the generation of a new shooting route at this location. Other regions with high confidence values ​​are displayed in green, indicating complete structure and sufficient data coverage, requiring no repeated sampling.

[0087] In the environmental perception acquisition step, the wind disturbance stability factor is calculated by combining the slope openness, the current wind speed vector and the UAV flight attitude. The wind disturbance stability factor is calculated based on the extended Kalman filter method. If the factor is higher than the wind disturbance intensity threshold, the shooting interval during flight is adjusted to the longest interval time so that the minimum inter-frame displacement is maintained when acquiring images in high wind areas.

[0088] Slope openness: This indicates the degree of openness of the terrain space in the current inspection area. It is generally estimated by analyzing the local spatial openness through a terrain DEM model or by using the distribution density of image features. The more open the slope, the greater the airflow disturbance, which affects the stability of flight attitude.

[0089] Wind speed vector: obtained through the airborne wind speed sensor or air pressure difference calculation module of the UAV, including wind speed magnitude (unit m / s) and direction information, updated in real time, and used as input for dynamic interference sources.

[0090] Drone flight attitude: Obtain the current heading angle, pitch angle, roll angle and their rate of change, which are derived from the IMU (Inertial Measurement Unit) and attitude fusion algorithm. The greater the attitude fluctuation, the worse the flight stability and the higher the risk of image jitter.

[0091] The calculation method of wind disturbance stability factor: The wind disturbance stability factor is calculated based on the extended Kalman filter (EKF) method, which is a Bayesian filtering algorithm for optimal estimation of the state of nonlinear dynamic systems. It is widely used in UAV state estimation and flight control systems.

[0092] In this invention, the calculation process of the wind disturbance stability factor is as follows:

[0093] State modeling: Flight attitude (attitude angle and angular velocity) and wind speed disturbance are treated as system state variables, and a set of nonlinear state transition equations is constructed;

[0094] Observational modeling: IMU measurements and wind speed sensor readings are used as observation vectors;

[0095] Prediction Update: EKF uses prior estimates to predict the current state and performs posterior corrections based on observed residuals;

[0096] Wind disturbance stability factor generation: In the final output state covariance matrix, the joint variance of the attitude disturbance and wind disturbance direction is selected, and after normalization, the wind disturbance stability factor is generated. The value range is [0,1], and the larger the value, the stronger the disturbance.

[0097] Dynamic adjustment mechanism for shooting interval: If the wind disturbance stabilization factor is higher than the wind disturbance intensity threshold (e.g., set to 0.7), the system determines that it is currently in a high wind disturbance state, which is not conducive to frequent shooting. At this time, the system will automatically adjust the image shooting rhythm during the drone's flight, extending the shooting interval to the preset maximum interval (e.g., 5 seconds, 8 seconds, or the system-set maximum frame period). The purpose of this strategy is to maintain the minimum inter-frame displacement when acquiring images in high wind areas, thereby: reducing ghosting or texture misalignment caused by flight offset between images; preventing low-quality image frames from entering the subsequent registration process; and reserving an image stabilization time window for attitude fluctuations caused by wind disturbance.

[0098] Implementation Example: During a UAV inspection of a steep slope in a mining area, the system detected a wind speed of 8.6 m / s during the third flight segment, with the wind direction deviating northward and forming an angle of approximately 40 degrees with the flight heading. The IMU showed a significant increase in pitch angle fluctuation frequency. After fusing the attitude angle change rate and wind direction disturbance using the EKF (Electronic Keyframe Analysis), the wind disturbance stability factor was calculated to be 0.81, exceeding the system's preset wind disturbance intensity threshold of 0.75. The system immediately adjusted the shooting frequency from the original 2 seconds / frame to a maximum interval of 5 seconds / frame, reducing the inter-frame displacement from 0.75 meters to 0.3 meters, significantly reducing image misalignment and jitter artifacts caused by wind disturbance. After this flight segment ended, the wind disturbance weakened, the system reassessed the stability factor, and returned to the standard shooting interval.

[0099] In the image quality optimization step, a sequence of images under a preset time window is acquired. A consistency vector is constructed using the mean square error of the gray-level difference rate between adjacent frames and the matching repetition rate of feature matching. The Euclidean distance between the consistency vector and the preset standard vector is used as the fusion score. If the fusion score is higher than the fusion threshold, the sequence of images is averaged and fused over multiple frames. Otherwise, the highest quality frame in the sequence of images is selected for modeling according to the preset image selection strategy.

[0100] Definition of time window: A sliding image sequence window is defined based on time. For example, the time window is set to 2-3 seconds, or the frame window is a continuous N frames of images (e.g., N=5). This sequence of images is usually a set of images continuously captured by the same camera during the flight of the drone, covering the same scene area.

[0101] Construction of the consistency vector: To evaluate whether the image sequence has good consistency, the system extracts inter-frame consistency metrics from the following two aspects to form a consistency vector V. e :

[0102] Gray-level difference rate: For each pair of images, i.e., any two frames, calculate the average difference of pixel gray-level values ​​point by point, normalize it to the [0,1] interval, and then calculate the average of all combined results. The smaller the value, the smaller the change in gray-level of the sequence images, that is, the lighting is stable and the image consistency is good.

[0103] The mean squared error of the matching repetition rate in feature matching: The matching repetition rate refers to the proportion of successfully matched feature points in a series of image frames out of the total number of feature points extracted in the current image. It is obtained by calculating the mean squared error of the matching repetition rate of the sequence by statistically analyzing the matching repetition rate of all frames in the image sequence. The smaller the mean squared error, the more stable the matching result.

[0104] Consistency Vector V e =[ΔG_avg,σ 2 ], where ΔG_avg represents the average grayscale difference rate of all frames in the sequence, σ 2 This represents the mean squared error of the feature matching repetition rate. A preset reference standard vector Vs = [ΔG_ref, σ...] is defined. 2 [_ref] represents the consistency performance of the image sequence under the theoretically ideal fusion condition. This standard vector can be set through training data, manual calibration, or empirical values. For example, Vs = [0.05, 0.01], and V is calculated. e The Euclidean distance between Vs is defined as the fusion score. The smaller the fusion score, the closer the current image sequence is to the ideal consistency condition.

[0105] Fusion Decision and Image Selection Strategy: A fusion threshold T (e.g., 0.1) is set. When the fusion score is higher than the fusion threshold T, the image sequence is considered to have good consistency and meets the fusion conditions. The sequence is then subjected to multi-frame averaging: pixel-level averaging is used, averaging the pixels at the same location across all frames. If the fusion score is not higher than the fusion threshold T, the image sequence has poor consistency and is not suitable for fusion. In this case, the highest quality frame from the sequence is selected for modeling according to a preset image selection strategy.

[0106] The image selection strategy can evaluate the quality score of a single frame image based on the following factors: number of feature points; image stability; gray-level distribution uniformity; texture sharpness index (such as local contrast standard deviation). The image with the highest score will be selected as the representative frame to input into the subsequent modeling process, and the remaining frames are only used for archiving or auxiliary evaluation.

[0107] Implementation Example: In a mine inspection mission, image sequences I1 to I5 were acquired along a certain segment of the flight path. The average grayscale difference rate was calculated to be 0.042, and the mean square error of the matching repetition rate was 0.008. The preset reference vector was [0.05, 0.01]. The calculated fusion score was 0.0095, which is much higher than the set fusion threshold of 0.1. The system determined that the sequence was suitable for fusion and performed multi-frame average fusion to generate a fused image I_avg for modeling.

[0108] In another section, analysis of image sequences I6 to I100 revealed a grayscale difference rate as high as 0.17, and significant fluctuations in the matching repetition rate (σ). 2 =0.036), the fusion score is calculated to be 0.19, which is lower than the fusion threshold. The image selection strategy is activated, and I7, which has a balanced distribution of feature points and the best clarity, is selected as the representative frame for modeling. The other images are not included in this round of modeling.

[0109] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0110] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0111] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0112] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0113] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A rapid 3D reconstruction method for unmanned aerial vehicle (UAV) mine inspection scenarios, characterized in that, Includes the following steps: Environmental perception and data acquisition steps: The drone collects information on light intensity, wind speed, and slope in the current inspection path area using multiple sensors, and dynamically adjusts the shooting rhythm and shooting locations based on light stability and wind disturbance level. Tilt-adaptive path planning steps: The shooting path is generated in real time based on the perceived slope tilt angle. When the tilt angle exceeds the set angle threshold, the path reconstruction mechanism is triggered to automatically adjust the flight attitude and generate a tilt reshoot path in the normal direction to improve the image coverage of steep slope areas. Image quality optimization steps: The acquired image is subjected to multi-scale texture enhancement and brightness normalization, robust feature points are extracted, and low-stability frames caused by wind disturbance are removed by combining a multi-frame fusion strategy to form an image confidence scoring model. The three-dimensional model construction steps are as follows: Based on the evaluation results of the image confidence scoring model, structural reconstruction and image registration are performed on images that are higher than the modeling confidence threshold. A reconstruction confidence judgment mechanism is introduced during the model construction process to obtain the modeling confidence value of the local area of ​​the model. When it is lower than the acceptable reconstruction threshold, re-photographing is performed to complete the generation of the three-dimensional model of the entire area. In the tilt angle adaptive path planning step, when the slope tilt angle change is detected to exceed the adaptive judgment angle threshold, the supplementary shooting area prediction calculation is automatically performed. The digital elevation model is used in conjunction with the slope geometric normal fitting method to deduce the optimal shooting tilt angle in the local fitting plane, and a slope-fitting path is generated based on the principle of minimum overlap blind zone. This path selects a flight trajectory perpendicular to the normal at the location of the slope change. In the image quality optimization step, an illumination-robust feature extraction mechanism is introduced. This mechanism uses local feature operators with illumination invariance when executing feature extraction algorithms after multi-scale image enhancement. The local feature operators can be any one of the AKAZE algorithm, ORB algorithm, or SuperPoint algorithm. In the environmental perception acquisition step, the wind disturbance stability factor is calculated by combining the slope openness, the current wind speed vector and the UAV flight attitude. The wind disturbance stability factor is calculated based on the extended Kalman filter method. If the factor is higher than the wind disturbance intensity threshold, the shooting interval during flight is adjusted to the longest interval time so that the minimum inter-frame displacement is maintained when acquiring images in high wind areas.

2. The rapid 3D reconstruction method for unmanned aerial vehicle (UAV) mine inspection scenarios according to claim 1, characterized in that, In the environmental perception acquisition step, an image frame-level stability detection mechanism is further introduced. The stability index Stab is calculated by the feature point displacement vector between consecutive images. This index is generated based on the average consistency of the optical flow field in the image sequence. When the value of the stability index Stab is lower than the preset dynamic stability discrimination threshold, the corresponding image frame is suspended from participating in subsequent modeling.

3. The rapid 3D reconstruction method for unmanned aerial vehicle (UAV) mine inspection scenarios according to claim 2, characterized in that, The stability index Stab is obtained by estimating optical flow consistency using the Lucas-Kanade method and then normalizing it, or by using a method based on bidirectional optical flow reprojection error calculation.

4. The rapid 3D reconstruction method for unmanned aerial vehicle (UAV) mine inspection scenarios according to claim 3, characterized in that, In the generation of flight paths that are attached to slope surfaces, the shooting angle is optimized to ensure that the deviation angle of the slope centerline normal is less than 5 degrees.

5. The rapid 3D reconstruction method for unmanned aerial vehicle (UAV) mine inspection scenarios according to claim 4, characterized in that, Before structural reconstruction and image registration, images are screened by constructing an image confidence scoring model. The image confidence level is derived by considering factors such as the number of feature points, the spatial distribution balance of feature points, and image stability. If the score result is higher than the image confidence judgment threshold, the image is classified as a high-confidence image. Specifically, the scoring model uses a pre-set normalized feature point density index, a structural similarity index, and an image stability index for weighted fusion. This is ultimately used to determine the confidence level that the image should be assigned in 3D reconstruction, with the confidence level being either a high-confidence image or a low-confidence image.

6. The rapid 3D reconstruction method for unmanned aerial vehicle (UAV) mine inspection scenarios according to claim 5, characterized in that, In the 3D model construction step, based on the point cloud density gradient change, matching residual, and feature point redundancy in the reconstruction results, a region modeling confidence map is constructed to color-code the modeling confidence level of local regions of the model. If the modeling confidence value of a local region of the model is lower than the acceptable reconstruction threshold, the region is marked as a region that needs to be resampled.

7. The rapid 3D reconstruction method for unmanned aerial vehicle (UAV) mine inspection scenarios according to claim 6, characterized in that, In the image quality optimization step, a sequence of images under a preset time window is acquired. A consistency vector is constructed using the mean square error of the gray-level difference rate between adjacent frames and the matching repetition rate of feature matching. The Euclidean distance between the consistency vector and the preset standard vector is used as the fusion score. If the fusion score is higher than the fusion threshold, the sequence of images is averaged and fused over multiple frames. Otherwise, the highest quality frame in the sequence of images is selected for modeling according to the preset image selection strategy.

Images