Rapid three-dimensional reconstruction method for unmanned aerial vehicle mine inspection scene

Abstract

The invention discloses a rapid three-dimensional reconstruction method for an unmanned aerial vehicle mine inspection scene, and particularly relates to the technical field of image processing and three-dimensional modeling, and the method comprises four steps: environment perception collection, inclination angle adaptive path planning, image quality optimization and three-dimensional model construction. Sensing information such as illumination, wind speed and gradient through a sensor to dynamically adjust a shooting strategy, and generating a normal supplementary shooting path in a gradient sudden change area; image quality is improved through image enhancement and feature extraction, and a confidence scoring model is constructed to screen high-quality images to participate in modeling; carrying out confidence evaluation on a modeling result, triggering a supplementary shooting mechanism, and improving the precision and integrity of the model; according to the method, the feature extraction stability of the image under the condition of illumination dramatic change is improved, the image coverage integrity of a high and steep slope area is enhanced, and stable control of image acquisition and fusion in a wind disturbance environment is realized, so that the precision, continuity and robustness of three-dimensional modeling are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and 3D modeling. More specifically, the present invention relates to a fast 3D reconstruction method for the unmanned aerial vehicle (UAV) mine inspection scenario. Background Art

[0002] In the process of mine inspection and slope stability analysis, using UAVs to collect images of the target area and generating 3D models through image reconstruction algorithms has become an important data acquisition method. In the direction of image processing and generation, 3D modeling methods mainly include image registration, feature extraction, image fusion, depth estimation, sparse or dense point cloud generation, mesh reconstruction, and texture mapping and other sub-links of image data processing.

[0003] However, in the application of high-steep slope scenarios in open-pit mines, existing 3D reconstruction methods face multiple technical bottlenecks:

[0004] Light interference affects the consistency of image reconstruction: The terrain of the mine slope is complex, with a large number of areas with direct sunlight, backlighting, strong specular reflection, and alternating shadows, resulting in significant differences in the brightness distribution of images obtained in the same area at different times, which significantly reduces the accuracy of feature point-based registration and image fusion, and seriously affects the structural restoration and texture consistency of subsequent 3D models.

[0005] Sudden changes in slope structure lead to image coverage blind spots: When there are sharp changes in slope inclination or there are structural forms such as steps, faults, and vertical rock faces, due to the limitations of UAV flight attitude control and fixed flight path planning, UAVs often cannot obtain perspective images of the slope normal angle, resulting in the lack of images in key areas, and structural voids or deformations in the 3D model, which are difficult to meet the safety assessment requirements.

[0006] Wind interference causes dynamic instability between image frames: Since the inspection area is at the mountain pass in the open mountain, the wind speed fluctuates frequently, which easily causes slight jitter in the UAV flight attitude, and then causes misalignment, ghosting, or slight blurring of the collected images between frames. Traditional 3D modeling processes are difficult to automatically identify such unstable images, resulting in problems of misregistration and artifact diffusion. Therefore, the present invention proposes a fast 3D reconstruction method for the UAV mine inspection scenario to meet the 3D modeling requirements of image processing tasks in complex mine slope scenarios. Summary of the Invention

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A fast 3D reconstruction method for the UAV mine inspection scenario, comprising the following steps:

[0009] Environmental perception and acquisition steps: Collect information on light intensity, wind speed, and slope of the current inspection path area through various sensors installed on the drone, and dynamically adjust the shooting rhythm and shooting points based on light stability and wind disturbance level;

[0010] Inclination adaptive path planning steps: Generate a shooting path in real time according to the sensed slope inclination angle. When it is detected that the inclination angle exceeds the set angle threshold, trigger the path reconstruction mechanism, automatically adjust the flight attitude, and generate an inclined supplementary shooting path in the normal direction to improve the image coverage rate of the high-steep slope area;

[0011] Image quality optimization steps: Perform multi-scale texture enhancement and brightness normalization on the acquired images, extract robust feature points, and combine a multi-frame fusion strategy to eliminate low-stability frames caused by wind disturbance to form an image confidence score model;

[0012] In a preferred embodiment, three-dimensional model construction steps: Based on the evaluation results of the image confidence score model, images with a score higher than the modeling credibility threshold perform structure reconstruction and image registration, and introduce a reconstruction confidence judgment mechanism during the model construction process to obtain the modeling confidence value of the local area of the model. When it is lower than the reconstruction acceptable threshold, supplementary shooting is performed to complete the generation of the three-dimensional model of the overall area.

[0013] In a preferred embodiment, in the environmental perception and acquisition steps, further introduce an image frame-level stability detection mechanism. Calculate the stability index Stab through the feature point displacement vector between consecutive images. This index is generated based on the average consistency degree of the optical flow field in the image sequence, and when the value of the stability index Stab is lower than the preset dynamic stability discrimination threshold, the corresponding image frame is paused from participating in subsequent modeling.

[0014] In a preferred embodiment, the stability index Stab is obtained by normalizing the optical flow consistency estimated based on the Lucas-Kanade method, or by using a calculation method based on the bidirectional optical flow reprojection error.

[0015] In a preferred embodiment, in the inclination adaptive path planning steps, when it is detected that the change in the slope inclination angle exceeds the angle threshold for adaptive judgment, automatically perform supplementary shooting area prediction calculation. Use a digital elevation model combined with a slope surface geometric normal fitting method to inversely calculate the optimal shooting inclination angle in the local fitting plane, and generate a path along the slope surface based on the principle of the smallest overlapping blind area. This path selects a flight trajectory perpendicular to the normal direction at the position of the slope sudden change.

[0016] In a preferred embodiment, in the generation of the flight path, i.e., the path along the slope surface, the shooting angle satisfies the optimization goal that the deviation angle from the normal of the slope surface axis is less than 5 degrees.

[0017] In a preferred embodiment, in the image quality optimization step, a lighting robustness feature extraction mechanism is introduced. After multi-scale image enhancement, when performing the feature extraction algorithm, a local feature operator with lighting invariance is used, and any one of AKAZE, ORB, or SuperPoint is adopted as the algorithm.

[0018] In a preferred embodiment, before image structure reconstruction and image registration, the images are screened by constructing an image confidence scoring model. The image confidence is obtained considering factors such as the number of feature points, the balance of the spatial distribution of feature points, and image stability. If the scoring result is higher than the image credibility determination threshold, the image is classified as a high-confidence image. Specifically, the scoring model selects a preset normalized feature point density index, a structural similarity index, and an image stability index for weighted fusion, and finally is used to judge the trust level to be given to the image in 3D reconstruction. The trust levels are high-confidence images and low-confidence images.

[0019] In a preferred embodiment, in the 3D model construction step, based on the point cloud density gradient change, the matching residual, and the feature point redundancy in the reconstruction result, a regional modeling confidence map is constructed to represent the modeling credibility of the local area of the model by color coding. If the modeling confidence value of the local area of the model is lower than the reconstruction acceptable threshold, this area is marked as the area where images need to be supplemented.

[0020] In a preferred embodiment, in the environmental perception acquisition step, the wind disturbance stability factor is calculated by combining the openness of the slope, the current wind speed vector, and the UAV flight attitude. The calculation of the wind disturbance stability factor is obtained based on the extended Kalman filter method. If this factor is higher than the wind disturbance intensity threshold, the shooting interval time during flight is adjusted to the longest interval time to keep the minimum inter-frame displacement when collecting images in a high-wind area.

[0021] In a preferred embodiment, in the image quality optimization step, a sequence of images within a preset time window is obtained. A consistency vector is constructed based on the gray difference rate between adjacent frames and the mean square error value of the matching repetition rate of feature matching. The Euclidean distance value between it and a preset standard vector is used as the fusion score. If the fusion score is higher than the fusion threshold, the sequence of images is subjected to multi-frame average fusion. Otherwise, the frame with the highest quality in the sequence of images is selected according to the preset image selection strategy to participate in the modeling.

[0022] The technical effects and advantages of the present invention:

[0023] By introducing a light-robust feature extraction mechanism, the present invention combines multi-scale image enhancement with local feature operators with illumination invariance (such as AKAZE, ORB, SuperPoint) in the image quality optimization step, significantly improving the feature extraction and description capabilities of images in environments with alternating strong light and shadows. This mechanism avoids the problem of feature matching failure caused by uneven brightness distribution, effectively improving the registration accuracy between images and the stability of continuous modeling, thus overcoming the problem of unstable modeling accuracy of traditional methods in complex illumination scenarios in mines.

[0024] In the inclination angle adaptive path planning step of the present invention, a supplementary shooting area prediction mechanism is automatically triggered based on the change of the slope inclination angle, and a "slope-following path" aligned with the slope geometric structure is generated by combining the digital elevation model and the normal fitting algorithm, making the camera shooting angle as close as possible to the slope normal direction, and controlling the shooting deviation angle to be less than 5 degrees in the flight path generation. This path planning mechanism effectively improves the coverage integrity of images in high-steep slopes or areas with structural mutations, avoids the problem of modeling blind spots caused by downward shooting dead angles, and enhances the geometric reduction ability and global structure continuity of the model.

[0025] In the environmental perception and acquisition step of the present invention, the wind disturbance stability factor is calculated by the extended Kalman filter method to dynamically adjust the shooting interval time, and the acquisition period is automatically extended when the wind disturbance intensity exceeds the preset threshold to reduce the inter-frame displacement; at the same time, in the image quality optimization process, a consistency vector is constructed by combining the image gray difference rate and the mean square error of the matching repetition rate, and the multi-frame average fusion is only performed on high-consistency image sequences based on the fusion score. The synergistic effect of this series of mechanisms effectively suppresses the problems of image jitter, misalignment, and fusion artifacts in the wind disturbance environment, ensures the stability of image input and the robustness of fusion output, and improves the modeling accuracy and image utilization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] For the convenience of those skilled in the art to understand, the present invention will be further described below in conjunction with the accompanying drawings;

[0027] Figure 1 It is the schematic diagram of the fast three-dimensional reconstruction method for the UAV mine inspection scenario in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0029] Refer to Figure 1The following embodiments are obtained:

[0030] Embodiment 1: In a mixed mining area combining open-pit mining and underground goaf, there are a large number of high-steep slopes formed by manual excavation or natural weathering. The geomorphic structure of this area is discontinuous, the slope inclination angle changes violently, and some areas are steeply inclined at more than 45 degrees, accompanied by a multi-layer stepped structure, belonging to a typical high-risk inspection area. In daily inspection tasks, due to different orientations of the slopes, there are often drastic changes in lighting conditions with strong sunlight direct irradiation and deep shadow coverage at the same time, resulting in serious brightness contrast during the image acquisition stage. Conventional image feature point extraction and registration are prone to failure, thus affecting the stability of 3D modeling. In addition, due to the steep terrain, drones are limited by flight angles and safety heights and are difficult to cover all slope normal angles, causing blind spots in the modeling data coverage. At the same time, this area is located at the valley wind mouth, where the wind speed changes frequently and the direction is unstable. During the flight process, slight jitters or even attitude offsets often occur, resulting in unstable inter-frame matching of images and texture misalignment or artifacts.

[0031] Therefore, the present invention proposes a fast 3D reconstruction method for the UAV mine inspection scenario, including the following steps:

[0032] Environmental perception and acquisition step: Collect the light intensity, wind speed, and slope information of the current inspection path area through various sensors installed on the UAV, and dynamically adjust the shooting rhythm and shooting points based on the lighting stability and wind disturbance level;

[0033] Inclination adaptive path planning step: Generate the shooting path in real time according to the perceived slope inclination angle. When the detected inclination angle exceeds the set angle threshold, trigger the path reconstruction mechanism, automatically adjust the flight attitude, and generate the inclined supplementary shooting path in the normal direction to improve the image coverage rate of the high-steep slope area;

[0034] Image quality optimization step: Perform multi-scale texture enhancement and brightness normalization processing on the acquired images, extract robust feature points, and combine the multi-frame fusion strategy to eliminate the low-stability frames caused by wind disturbance to form an image confidence score model;

[0035] 3D model construction step: Based on the evaluation results of the image confidence score model, perform structure reconstruction and image registration on the images with a score higher than the modeling credibility threshold, and introduce a reconstruction confidence judgment mechanism during the model construction process to obtain the modeling confidence value of the local area of the model. When it is lower than the reconstruction acceptable threshold, perform supplementary shooting to complete the generation of the 3D model of the overall area.

[0036] In the environmental perception and acquisition step, an image frame-level stability detection mechanism is further introduced. The stability index Stab is calculated through the displacement vector of feature points between consecutive images. This index is generated based on the average consistency degree of the optical flow field in the image sequence. When the value of the stability index Stab is lower than the preset dynamic stability discrimination threshold, the corresponding image frame is paused from participating in subsequent modeling. The stability index Stab is obtained by normalizing the estimated optical flow consistency based on the Lucas-Kanade method or by using the method based on the bidirectional optical flow reprojection error calculation.

[0037] The image frame-level stability detection mechanism is used to automatically determine based on the visual stability between consecutive images during the image acquisition stage, so as to avoid using unstable image frames caused by wind disturbance, slight flight attitude drift or vibration for 3D modeling and ensure the basic quality of the modeling input images.

[0038] The detection process is as follows: Image sequence input: Continuously collect an image frame sequence A1, A2, A3, …, A n , and successively select the current frame and its adjacent previous frame for comparison between two frames to form comparison pairs (A i-1 , A i ), where i represents the number and n is the total quantity;

[0039] Feature point extraction and matching: For each frame of image, extract several image feature points with relatively high stability (corner detectors such as FAST and Shi-Tomasi can be selected) to describe the motion consistency between frames. The extracted feature points retain the spatial distribution uniformity as the selection criterion to prevent misjudgment caused by local concentration.

[0040] Displacement vector calculation: For each pair of image frames, based on the feature points in the image sequence, estimate their displacement relationship in the time dimension through the optical flow field to form the displacement vectors Δx and Δy of each feature point. This displacement vector is used for the subsequent calculation of the stability index.

[0041] The calculation method of the stability index Stab is as follows:

[0042] Method 1: Normalization processing after estimating optical flow consistency based on the Lucas-Kanade method:

[0043] Using the Lucas-Kanade optical flow method, construct a least-square error model with invariant brightness in the neighborhood of feature points, calculate the local pixel displacement between image frames, collect the optical flow vector set of all feature points, and calculate its standard deviation σ_flow. The average optical flow consistency at all points is defined as Stab_pre, and finally Stab_pre is normalized to the interval [0,1] to obtain the stability index Stab.

[0044] Example: In a set of image frames, if the directions of the optical flow vectors of most feature points are close and the magnitudes change little, it indicates that the relative motion between frames is stable, and the Stab value approaches 1; conversely, if the optical flow differences are large, Stab will decrease.

[0045] Method 2: Obtained based on the bidirectional optical flow reprojection error calculation method:

[0046] In image frame A i and A i-1 calculate the forward optical flow (A i-1 , A i ) and the backward optical flow (A i →A i-1 ) respectively to obtain a set of bidirectional matching points. For each feature point, a closed-loop path is formed in the original image using the forward and backward paths, and the Euclidean distance between the starting and returning positions of this point is calculated as the reprojection error. The mean or median of the reprojection errors of all matching points is used as the error metric for the current frame. After normalization, it is the stability index Stab. This method can significantly suppress the false matching pseudo-steady state problem caused by unidirectional errors and is particularly suitable for the analysis of frame pairs with obvious wind disturbances or attitude fluctuations.

[0047] There is a dynamic stability discrimination threshold, denoted as T s , which can be obtained by statistical analysis of historical data according to the scene environment or set by model training. For example, the value of T s is 0.75. When the stability index Stab corresponding to a certain frame of image < T s , this frame of image is marked as an unstable frame; this frame will pause from participating in the subsequent image registration and 3D modeling processes and be excluded from the current modeling task, only retained in the original image library for manual review or subsequent reshooting plan analysis; this operation ensures that the image frames entering the image quality optimization and 3D modeling steps have sufficient temporal consistency, displacement controllability, and visual stability.

[0048] Example: In a slope UAV operation, 120 frames of images are continuously collected, and Stab is calculated frame by frame using the above method. The results show that in the 37th and 88th frames, due to strong winds, the attitude shifted sharply, and the mean reprojection error was significantly higher than that of other frames. The Stab values were 0.52 and 0.47 respectively, far lower than the preset threshold of 0.75. These two frames were automatically set as "excluded frames" and did not enter the 3D modeling process, thus effectively avoiding the problems of texture misalignment and abnormal structure diffusion.

[0049] In the inclination angle adaptive path planning step, when it is detected that the change in the slope inclination angle exceeds the angle threshold for adaptive judgment, the supplementary shooting area prediction calculation is automatically performed. The digital elevation model is used in combination with the slope surface geometric normal fitting method to inversely deduce the optimal shooting inclination angle in the local fitting plane, and a path along the slope surface is generated based on the principle of the minimum overlapping blind area. The flight trajectory perpendicular to the normal direction is selected at the position where the slope changes abruptly in the flight path, i.e., the path along the slope surface. When generating the flight path, i.e., the path along the slope surface, the shooting angle is optimized with the deviation angle of the normal line of the slope surface axis less than 5 degrees as the goal.

[0050] In the inclination angle adaptive path planning step, when it is detected that the change in the slope inclination angle exceeds the angle threshold for adaptive judgment, the system automatically performs the supplementary shooting area prediction calculation. This supplementary shooting mechanism is mainly used to deal with the situation of uneven image coverage caused by drastic slope changes and blind areas in the modeling area, and is especially suitable for mine slopes with stepped or cliff-like structures.

[0051] Angle threshold judgment: The UAV system obtains the slope inclination angle on the current flight path in real time through sensors and compares it with the angle between its flight direction. When it is detected that the slope changes abruptly and the angle change exceeds the set adaptive judgment angle threshold (for example, it can be set to 20 degrees, 25 degrees or the standard set through experiments in other scenarios), it is considered that there are potential modeling blind areas or insufficient shooting coverage in this area, triggering subsequent path reconstruction actions.

[0052] Supplementary shooting area prediction calculation: The system first calls the digital elevation model (DEM) of the area, which can be established through lidar scanning, photogrammetry or historical data and represents the surface elevation change. After discovering areas with high elevation differences or slope surface structure mutations, the system constructs a local geometric plane using the normal fitting method according to the spatial distribution of the slope surface points and analyzes it within the fitting plane to inversely deduce the suitable optimal shooting inclination angle. The normal fitting method can adopt the method of fitting a plane with a point set, such as the least squares method for local plane approximation, so as to obtain the current normal vector of the slope surface, denoted as Np, and then calculate the optimal shooting angle according to the current flight attitude of the UAV to make the camera imaging plane as orthogonal to the slope surface as possible.

[0053] Flight path generation (ramp surface following path): After the local shooting angle is determined, the path planning module generates a "ramp surface following path" based on the principle of minimizing the overlapping blind area. The so-called minimum overlapping blind area refers to minimizing the area where image reconstruction cannot be effectively carried out due to angle deviation on the basis of the image acquisition overlapping rate. To this end, the system preferentially generates a flight trajectory aligned with the slope normal direction, so that the camera always remains approximately perpendicular to the local slope surface during flight. In particular, at the location of sudden slope change (such as a rock mass section that suddenly changes from 35 degrees to 60 degrees), to avoid image tilt distortion caused by oblique shooting, the system will switch the flight attitude at this position, so that the drone heading forms the smallest angle with the slope normal, thus generating a flight trajectory perpendicular to the slope normal at this section.

[0054] Shooting angle control objective: In the generation of the flight path, i.e., the ramp surface following path, the system further optimizes the control of the shooting angle so that the deviation angle between the camera optical axis direction and the slope central axis normal does not exceed the set error range. Specifically, the system takes the slope central axis normal as the reference direction and controls the shooting angle deviation to be less than 5 degrees as the optimization objective. This control strategy can effectively improve the orthogonality and consistency of image features and improve the accuracy of subsequent image registration and 3D modeling.

[0055] Engineering application example: For example, in a drone inspection mission of the southwest slope of an open-pit mine, the system detected through sensors that a certain area suddenly changed from a gentle slope to a nearly vertical rock surface, and its local angle change exceeded 30 degrees, exceeding the system-set angle threshold of 25 degrees. The system immediately called the regional elevation map, conducted geometric analysis on this section of the slope, extracted the point set for plane fitting, and calculated the main normal direction of this area. Subsequently, based on the normal direction, the system generated a set of supplementary shooting flight paths perpendicular to the slope normal, and controlled the included angle between the camera shooting angle and this normal direction to be within 3.2 degrees. The finally generated ramp surface following path completely covered this mutation area, effectively compensating for the modeling occlusion problem caused by the downward viewing angle of the main path.

[0056] The "flight path" here is mainly a path for supplementary shooting, which is a "second supplementary shooting path" triggered by the condition of angle mutation. It is not the globally preset main path, but it is part of the acquisition path system. Therefore, overall, it can be considered as: "A secondary dynamic acquisition path triggered by specific slope inclination changes, used to compensate for the image blind area caused by insufficient coverage of the first acquisition path."

[0057] The flight path in the present invention is a supplementary shooting path automatically generated based on the sudden change of the slope angle in the inclination angle adaptive path planning step. It is different from the global acquisition path and is designed specifically to make up for the image blind area. During the path generation process, by combining the digital elevation model and the slope normal direction, through existing algorithms such as RRT, A* combined with elevation grids, and polynomial attitude optimization, a stable flight trajectory planning along the slope surface is realized. At the same time, it is ensured that the shooting angle in the flight path meets the optimization goal that the deviation angle from the slope central axis normal angle is less than 5 degrees, thereby improving the image coverage rate and the 3D reconstruction accuracy. The advantage of the present invention lies in that instead of flying globally once and then modeling, when the slope gradient suddenly changes, it realizes real-time perception + path reconstruction + supplementary shooting path execution, which is a path planning mechanism with the characteristics of "intelligent perception + local reconstruction + local optimized shooting". It creatively combines path generation with in-depth geometric elements such as slope gradient changes and normal direction alignment, which is an improvement to the traditional flight path planning.

[0058] The RRT algorithm is used for: obstacle avoidance and path search in free space. Characteristics: suitable for generating feasible paths in environments with complex slopes, occlusions, and prominent terrain changes; The combination point with the present invention: can be used for the preliminary trajectory exploration of generating a "slope-following path" from the main path; constructing a supplementary shooting flight path leading to the normal direction in the slope mutation area; can integrate flight safety constraints (such as minimum flight height, attitude transformation range).

[0059] The A + DEM rasterized path planning algorithm is used for: shortest flyable path planning combined with elevation data; Characteristics: suitable for generating 2D + elevation fitting paths based on digital elevation model (DEM) data; The combination point with the present invention: converting the slope DEM into a raster map, setting a "reweighting area" at the slope mutation; automatically generating supplementary shooting path segments to avoid dead ends of the main path; can be accompanied by flight attitude control, setting the target shooting angle as "perpendicular to the local slope normal".

[0060] The polynomial trajectory optimization algorithm is used for: smoothing optimization and attitude control of the UAV flight path; Characteristics: used to control the camera orientation to always point to a certain target direction (such as the normal direction) within a certain path segment; The combination point with the present invention: after fitting and generating the slope-following path, optimizing the camera attitude to keep the deviation angle less than 5°; generating "multi-segment stable shooting trajectories" in the multi-angle reconstruction area to improve the continuity of model reconstruction; controlling the coordination of flight speed / attitude change and image acquisition interval.

[0061] In the image quality optimization step, a light robustness feature extraction mechanism is introduced. After multi-scale image enhancement, when performing the feature extraction algorithm, a local feature operator with light invariance is used, and the algorithm used is any one of AKAZE, ORB, or SuperPoint.

[0062] Before structural reconstruction and image registration of the images, screening is performed by constructing an image confidence scoring model. The image confidence is obtained considering factors such as the number of feature points, the balance of the spatial distribution of feature points, and the image stability. If the scoring result is higher than the image credibility determination threshold, the image is classified as a high-confidence image. Specifically, the scoring model selects a preset normalized feature point density index, a structural similarity index, and combines them with an image stability index for weighted fusion, and finally is used to judge the trust level that should be given to the image in 3D reconstruction. The trust levels are high-confidence images and low-confidence images.

[0063] In the image quality optimization step, a light robustness feature extraction mechanism is introduced. This mechanism aims to solve the problems of unstable image feature extraction and increased matching error rate in scenarios with drastic light changes such as direct strong light and alternating shadows in the inspection area. Before the image enters the subsequent modeling process, the adaptability and consistency of the image descriptor to light changes are jointly improved through two stages of preprocessing enhancement and feature extraction.

[0064] Multi-scale image enhancement: After the image enters the optimization step, the system first performs multi-scale image enhancement processing on the image, and constructs different scale versions of the image using the Gaussian pyramid or Laplacian pyramid method. In each scale image, the system performs local contrast stretching and brightness normalization to suppress the interference of local overexposed or underexposed areas caused by strong light and shadows on feature detection. After processing, the images of each scale are fused back to the main scale for feature extraction.

[0065] Light robustness feature extraction: When performing the feature extraction algorithm, the system selects a local feature operator with light invariance or local texture preservation ability, including any one of the AKAZE (Accelerated KAZE), ORB, or SuperPoint algorithms. The characteristics of each algorithm are as follows:

[0066] AKAZE: Detects key points based on a non-linear scale space, and uses the Modified Local Difference Binary descriptor, which has good adaptability to light and blur.

[0067] ORB: Combines the FAST corner point and the BRIEF descriptor, adds direction encoding to improve rotational invariance, and has high stability in fast applications.

[0068] SuperPoint: Performs end-to-end feature point detection and description using a neural network, and has high robustness in scenarios such as light and texture loss.

[0069] The algorithm configuration mechanism is as follows: Based on 1. Flight platform performance: Flight platform performance mainly refers to hardware parameters such as the processor capabilities of the drone (such as whether it has GPU acceleration), the operating memory capacity, the frame rate and resolution of the image sensor, etc. The system sets the preferred algorithm according to the following rules: Resource-constrained platforms (without GPU / ARM chips / low-power devices): Prefer to use the ORB algorithm. Its feature detection and description are both lightweight calculations, suitable for scenarios with high real-time requirements and limited computing power. Mid-to-high-end platforms (with medium computing capabilities): Can choose the AKAZE algorithm. It supports non-linear scale-space feature extraction and has higher light robustness, suitable for scenarios with medium texture complexity. High-performance platforms (with GPU / with the ability to deploy deep models): Prefer to use the SuperPoint algorithm. It is an end-to-end neural network model with extremely strong anti-interference ability and re-localization accuracy, suitable for scenarios with extreme light changes or weak texture areas. When the system starts, it will detect the processing platform information and automatically load the matching default algorithm. The user can also manually select or disable a certain algorithm in the parameter configuration interface.

[0070] Based on 2. Texture features of the inspection area: For the image texture characteristics of different areas, the system can dynamically switch or prioritize algorithms in combination with the following indicators: If there are large areas of sudden light changes, shadow occlusions, or strong reflections (such as metal surfaces, rock edges) in the image, give priority to using AKAZE or SuperPoint; If there are low-texture areas in the image (such as dust, smooth rock walls), SuperPoint is preferred because its end-to-end learning enhances the feature perception ability for weak-texture areas; If the area has rich texture, uniform lighting, and limited computing resources, ORB can be directly used. The system can automatically judge the scene category of the current image through a preset texture complexity discrimination model (such as the image gray change rate, local variance) and switch the most suitable feature extraction algorithm according to the corresponding strategy.

[0071] Image confidence scoring model: Before image structure reconstruction and image registration, the system executes an image confidence scoring model to screen images, avoiding low-quality images from entering the registration process and reducing error diffusion. The image confidence score comprehensively considers three key factors: the number of feature points, the balance of the spatial distribution of feature points, and image stability. The specific explanations are as follows:

[0072] Number of feature points: The system counts the total number of feature points successfully extracted in the image to reflect the richness of the local texture of the image. Too few points are likely to lead to registration failure;

[0073] Balance of the spatial distribution of feature points: The image is divided into m×n regions, and the standard deviation of the number of feature points in each region is counted. The lower the standard deviation, the more evenly the feature points are distributed, providing a wider range of registration anchor points and being beneficial to the integrity of modeling;

[0074] Image stability: The Stab stability index derived from the image frame-level stability detection mechanism, which reflects the motion consistency between this image and adjacent frames. The higher the value, the smaller the inter-frame jitter and the higher the usability.

[0075] The above three factors are respectively normalized into a scoring vector in the range of [0, 1], defined as: F1: Normalized feature point density index; F2: Distribution balance degree index (the higher the balance, the higher the score); F3: Image stability index (Stab value); The final image confidence score F_total = w1 × F1 + w2 × F2 + w3 × F3. The three weights can be preset or tuned empirically in the system. For example, w1:w2:w3 = 0.4:0.3:0.3. It should be noted that the original definition of F2 is the "distribution balance degree index", that is, whether the spatial distribution of feature points in the image is uniform. In fact, the "uniformity of the spatial distribution of feature points" is a question of "whether the image structure information is evenly expressed", which is an index of the image spatial structure expression ability, so it is also a structural similarity index.

[0076] If the scoring result is higher than the image credibility determination threshold (set to 0.75 for example), then this image is classified as a high-confidence image. Otherwise, it is classified as a low-confidence image. The trust levels are: High-confidence images: Directly participate in structure reconstruction and image registration, and are the source of the main images for modeling; Low-confidence images: Do not directly participate in modeling, and can be recorded for reshooting prompts or used as auxiliary references.

[0077] Implementation example: In a certain inspection mission, the system obtains 50 frames of images from the drone and performs confidence scoring analysis frame by frame. The 12th frame of the image has 850 feature points, the standard deviation of the spatial distribution is 7.1, the Stab index is 0.93, and the normalized score is F_total = 0.82, which is higher than the set credibility threshold of 0.75 and is classified as a high-confidence image. The 27th frame of the image has only 412 feature points, which are concentrated on the left side of the image, the Stab value is 0.61, and after calculation, the score F_total = 0.59, which is classified as a low-confidence image, is removed from the 3D modeling process, and is recorded as a potential reshooting area.

[0078] In the 3D model construction step, based on the point cloud density gradient change, matching residuals, and feature point redundancy in the reconstruction result, a regional modeling confidence map is constructed to represent the modeling credibility of the local area of the model by color coding. If the modeling confidence value of the local area of the model is lower than the reconstruction acceptable threshold, then this area is marked as an area that needs to reshoot images.

[0079] Extract local modeling quality information from the 3D reconstruction result through the following three key indicators:

[0080] Point cloud density gradient change: Count the number of point clouds in each local area and calculate the distribution change rate (gradient) in space. Drastic gradient changes usually mean that the point cloud distribution in the area is discontinuous, sparse, or reconstructed at the edge of the area, and the reliability is low. The system normalizes the density gradient change and obtains the density stability score D m .

[0081] Matching residual: refers to the mean or median of the reprojection error of key points during image registration, reflecting the image matching accuracy in the region. A high regional matching residual indicates a large registration error, and the model may have structural misalignment or texture mismatch in this region. This item is defined as the matching accuracy score R e , the lower the score, the greater the error.

[0082] Feature point redundancy: Indicates whether a certain area is covered by multiple images at different angles. The more coverage, the higher the redundancy. Redundancy can be obtained by calculating the number of image frames projected onto the area. The higher the score, the more sufficient the data support, which is expressed as the coverage richness score C. r .

[0083] Modeling confidence map generation method: normalize the above three indicators to the interval [0,1] and fuse them in the following weighted manner to obtain the regional modeling confidence value:

[0084] Modeling confidence value S = α·D m +β·(1-R e )+γ·C r ; α, β, γ are configurable weight factors (e.g. α = 0.4, β = 0.4, γ = 0.2); D m Indicates the point cloud density consistency; R e is the residual value, which is negated to satisfy the direction of “the larger the value, the more reliable it is”; C r Indicates the coverage redundancy of feature points. After summarizing the S values of all regions, the system renders the 3D model surface in the form of a pseudo-color map to construct a regional modeling confidence map. The color coding can use a standard gradient from red (low) to green (high).

[0085] Reconstruction acceptance threshold and supplementary mining mechanism: A reconstruction acceptance threshold (denoted as T r , for example, 0.65), as the dividing line for judging whether the regional modeling is credible; if the modeling confidence value S ≥ T r , then the modeling of this area is considered credible; if S <T r , the area is marked by the system as "area requiring additional image acquisition", highlighted in red on the 3D model, and enters the image acquisition task queue.

[0086] Implementation example description: In a modeling task of a mine slope, after the system completes the preliminary point cloud modeling, it conducts a modeling confidence analysis for the model area. The result shows that in the southwestern corner area of the model (area ID: R27), the point cloud is sparse, the matching residual is high, and it is only covered by 2 frames of images. The system calculates the modeling confidence value of this area as 0.58, which is lower than the set threshold of 0.65. Therefore, this area appears as an orange-red block in the modeling view. The system marks it as an area that needs to be re-mined and sends a reshooting instruction to the flight mission system, requesting to generate a new shooting route at this location. Other areas with high confidence values are all displayed in green, indicating that the structure is complete and the data coverage is sufficient, and there is no need for repeated acquisition.

[0087] In the environmental perception acquisition step, the wind disturbance stability factor is calculated by combining the openness of the slope, the current wind speed vector, and the flight attitude of the unmanned aerial vehicle (UAV). The calculation of the wind disturbance stability factor is based on the extended Kalman filter method. If this factor is higher than the wind disturbance intensity threshold, the shooting interval time during flight is adjusted to the longest interval time, so as to maintain the lowest inter-frame displacement when collecting images in a high-wind area.

[0088] Openness of the slope: It represents the openness of the geomorphic space in the current inspection area. Generally, the local spatial openness is analyzed through the terrain DEM model, or estimated using the distribution density of image features. The more open the slope is, the greater the air flow disturbance, which affects the flight attitude stability.

[0089] Wind speed vector: It is obtained through the on-board wind speed sensor of the UAV or the air pressure difference calculation module, and contains the wind speed magnitude (unit: m / s) and direction information, which is updated in real time and input as a dynamic disturbance source.

[0090] Flight attitude of the UAV: The current heading angle, pitch angle, roll angle (yaw, pitch, roll) and their change rates are obtained, which come from the IMU (inertial measurement unit) and the attitude fusion algorithm. The greater the attitude fluctuation, the worse the flight stability and the higher the risk of image jitter.

[0091] Calculation method of the wind disturbance stability factor: The calculation of the wind disturbance stability factor is based on the extended Kalman filter method (EKF). This method is a Bayesian filtering algorithm for optimal estimation of the state of a nonlinear dynamic system, and is widely used in UAV state estimation and flight control systems.

[0092] In the present invention, the calculation process of the wind disturbance stability factor is as follows:

[0093] State modeling: The flight attitude (attitude angle and angular velocity) and wind speed disturbance are used as system state variables to construct a nonlinear state transition equation set;

[0094] Observation modeling: The IMU measurement value and the wind speed sensor reading are used as the observation vector;

[0095] Prediction Update: The EKF uses the prior estimate to predict the current state and performs posterior correction based on the observation residual;

[0096] Wind Disturbance Stability Factor Generation: In the finally output state covariance matrix, the joint variance of the attitude disturbance and the wind disturbance direction is selected, and after normalization, the wind disturbance stability factor is generated. The value range is between [0, 1], and the larger the value, the stronger the disturbance.

[0097] Dynamic Adjustment Mechanism of Shooting Interval Time: If the wind disturbance stability factor is higher than the wind disturbance intensity threshold (such as set to 0.7), the system determines that the current is in a high wind disturbance state, which is not conducive to frequent shooting. At this time, the system will automatically adjust the image shooting rhythm during the flight of the UAV and extend the shooting interval time to the preset maximum interval time (such as 5 seconds, 8 seconds or the maximum frame period set by the system). The purpose of this strategy is to maintain the lowest inter-frame displacement when collecting images in high wind areas, thereby: reducing ghosting or texture misalignment caused by flight offset between images; preventing low-quality image frames from entering the subsequent registration process; reserving an image stability time window for the attitude fluctuations caused by wind disturbances.

[0098] Implementation Example Illustration: During a UAV inspection of a high-steep slope in a goaf, the system detected that the wind speed reached 8.6 m / s in the 3rd flight segment, with the direction being northward and forming an angle of about 40 degrees with the flight heading. The IMU showed that the pitching angle fluctuation frequency increased significantly. After the EKF fused the attitude angle change rate and the wind direction disturbance, the calculated wind disturbance stability factor was 0.81, exceeding the system's preset wind disturbance intensity threshold of 0.75. The system immediately adjusted the shooting frequency from the original 2 s / frame to the maximum interval of 5 s / frame, reducing the inter-frame displacement between images from the original 0.75 meters to 0.3 meters, significantly reducing the image misalignment and jitter artifacts caused by wind disturbances. After the end of this flight segment, the wind disturbance weakened, and the system re-evaluated the stability factor and restored to the standard shooting interval.

[0099] In the image quality optimization step, a sequence of images within a preset time window is obtained. The Euclidean distance value between the consistency vector constructed by the gray difference rate between adjacent frames and the mean square value of the matching repetition rate of feature matching and a preset standard vector is used as the fusion score. If the fusion score is higher than the fusion threshold, the sequence of images is subjected to multi-frame average fusion; otherwise, the frame with the highest quality in the sequence of images is selected according to the preset image selection strategy to participate in the modeling.

[0100] Definition of Time Window: A sliding image sequence window is defined based on time. For example, the time window is set to 2 - 3 seconds, or the frame number window is a continuous set of N frames of images (such as N = 5). This sequence of images is usually a set of images continuously collected by the same camera during the flight of the UAV, covering the same scene area.

[0101] Construction of the Consistency Vector: To evaluate whether the image sequence has good consistency, the system extracts the inter-frame consistency metrics from the following two aspects to form the consistency vector V e :

[0102] Gray-scale Difference Rate: For each pair of images, that is, any two frames of images, calculate the average of the point-by-point differences of the pixel gray-scale values and normalize it to the interval [0, 1]. Then calculate the average of all combination results. The smaller the value, the smaller the change in the gray-scale level of the sequence images, that is, the more stable the illumination and the better the image consistency.

[0103] Mean Square Error of the Matching Repetition Rate of Feature Matching: The matching repetition rate refers to the proportion of successfully matched feature points in the total number of feature points extracted from the current image in consecutive image frames. By counting the matching repetition rates of all frames in the image sequence, calculate the mean square error of the matching repetition rate of this sequence. The smaller the mean square error, the more stable the matching result.

[0104] Consistency Vector V e = [ΔG_avg, σ 2 , where ΔG_avg represents the average gray-scale difference rate of all frames in this sequence, and σ 2 represents the mean square error of the matching repetition rate of feature matching. Set a preset reference standard vector Vs = [ΔG_ref, σ 2 _ref], which represents the consistency performance of the image sequence under the theoretical ideal fusion condition. This standard vector can be set through training data, manual calibration or empirical values. For example, Vs = [0.05, 0.01]. Calculate the Euclidean distance between V e and Vs, which is defined as the fusion score. The smaller the fusion score value, the closer the current image sequence is to the ideal consistency condition.

[0105] Fusion Decision and Image Selection Strategy: Set the fusion threshold T (such as 0.1). When the fusion score is higher than the fusion threshold T, it is determined that the image sequence has good consistency and meets the fusion condition. Perform multi-frame average fusion on the sequence images: Adopt the pixel-level average method to average the pixels at the same position in all frames. If the fusion score is not higher than the fusion threshold T, it means that the image sequence has poor consistency and is not suitable for fusion. According to the preset image selection strategy, select the frame with the highest quality from this sequence of images to participate in the modeling.

[0106] The image selection strategy can evaluate the quality score of a single-frame image based on the following factors: the number of feature points; image stability; gray-scale distribution balance; texture clarity index (such as the standard deviation of local contrast). The image with the highest score will be selected as the representative frame and input into the subsequent modeling process, and the remaining frames are only used for archiving or auxiliary evaluation.

[0107] Implementation example illustration: In a mine inspection task, for a certain section of the flight path, image sequences I1 to I5 are collected. The average value of the grayscale difference rate is calculated to be 0.042, and the mean square deviation of the matching repetition rate is 0.008. The preset reference vector is [0.05, 0.01], and the calculated fusion score is 0.0095, which is much higher than the set fusion threshold of 0.1. The system determines that this sequence is suitable for fusion and performs multi-frame average fusion to generate the fused image I_avg for modeling.

[0108] In another section, analyzing image sequences I6 to I100, it is found that the grayscale difference rate is as high as 0.17, and the matching repetition rate fluctuates significantly (σ 2 = 0.036). The calculated result of the fusion score is 0.19, which is lower than the fusion threshold. The image selection strategy is activated, and I7 with a balanced distribution of feature points and the best clarity is selected as the representative frame for modeling from them, and the remaining images do not participate in this round of modeling tasks.

[0109] The above formulas are all dimensionless and take their numerical values for calculation. The formula is obtained by collecting a large amount of data for software simulation to get a formula closest to the real situation. The preset parameters in the formula are set by those skilled in the art according to the actual situation.

[0110] It should be understood that in various embodiments of the present application, the magnitudes of the serial numbers of the above processes do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0111] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0112] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0113] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present application, and all should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A rapid three-dimensional reconstruction method for the unmanned aerial vehicle (UAV) mine inspection scenario, characterized in that, It includes the following steps: Environmental perception and acquisition step: Collect the light intensity, wind speed, and slope information of the current inspection path area through various sensors installed on the drone, and dynamically adjust the shooting rhythm and shooting points based on the light stability and wind disturbance level; Inclination adaptive path planning step: Generate a shooting path in real time according to the perceived slope inclination angle. When it is detected that the inclination angle exceeds the set angle threshold, trigger the path reconstruction mechanism, automatically adjust the flight attitude, and generate an inclined supplementary shooting path in the normal direction to improve the image coverage rate of the high-steep slope area; Image quality optimization step: Perform multi-scale texture enhancement and brightness normalization processing on the acquired images, extract robust feature points, and combine the multi-frame fusion strategy to eliminate the low-stability frames caused by wind disturbance to form an image confidence score model; Three-dimensional model construction step: Based on the evaluation results of the image confidence score model, the images with a score higher than the modeling credibility threshold perform structure reconstruction and image registration, and introduce a reconstruction confidence judgment mechanism during the model construction process to obtain the modeling confidence value of the local area of the model. When it is lower than the reconstruction acceptable threshold, supplementary shooting is performed to complete the generation of the three-dimensional model of the overall area.

2. The rapid three-dimensional reconstruction method for the drone-based mine inspection scenario according to claim 1, wherein In the environmental perception and acquisition step, further introduce an image frame-level stability detection mechanism. Calculate the stability index Stab through the feature point displacement vector between consecutive images. This index is generated based on the average consistency degree of the optical flow field in the image sequence. When the value of the stability index Stab is lower than the preset dynamic stability discrimination threshold, pause the corresponding image frame to participate in the subsequent modeling.

3. The rapid three-dimensional reconstruction method for the drone mine inspection scenario according to claim 2, wherein The stability index Stab is obtained by normalizing the optical flow consistency estimated by the Lucas-Kanade method or by using the method based on the bidirectional optical flow reprojection error calculation.

4. The rapid three-dimensional reconstruction method for the drone-based mine inspection scenario according to claim 3, wherein In the inclination adaptive path planning step, when it is detected that the change in the slope inclination angle exceeds the angle threshold of the adaptive judgment, automatically perform the prediction calculation of the supplementary shooting area. Use the digital elevation model combined with the method of fitting the geometric normal of the slope surface to inversely calculate the optimal shooting inclination angle in the local fitting plane, and generate a path along the slope surface based on the principle of the minimum overlapping blind area. This path selects a flight trajectory perpendicular to the normal direction at the position of the slope mutation.

5. The rapid three-dimensional reconstruction method for the drone-based mine inspection scenario according to claim 4, wherein In the generation of the flight path, that is, the path along the slope surface, the shooting angle satisfies the optimization goal that the deviation angle of the normal line of the slope surface axis is less than 5 degrees.

6. The rapid three-dimensional reconstruction method for the drone-based mine inspection scenario according to claim 5, wherein, In the image quality optimization step, introduce a light robustness feature extraction mechanism. After multi-scale image enhancement, when performing the feature extraction algorithm, use a local feature operator with light invariance. The algorithms used are any one of AKAZE, ORB, or SuperPoint.

7. The rapid three-dimensional reconstruction method for the drone mine inspection scenario according to claim 6, wherein Before structural reconstruction and image registration of the images, screening is carried out by constructing an image confidence scoring model. The image confidence is obtained considering factors such as the number of feature points, the balance of the spatial distribution of feature points, and the image stability factor. If the scoring result is higher than the image credibility determination threshold, the image is classified as a high-confidence image. Specifically, the scoring model selects a preset normalized feature point density index, a structural similarity index, and combines them with an image stability index for weighted fusion, and finally uses it to judge the trust level that should be given to the image in 3D reconstruction. The trust levels are high-confidence images and low-confidence images.

8. The rapid three-dimensional reconstruction method for the unmanned aerial vehicle mine inspection scenario according to claim 7, characterized in that In the 3D model construction step, based on the point cloud density gradient change, matching residuals, and feature point redundancy in the reconstruction result, a regional modeling confidence map is constructed to represent the modeling credibility of the local area of the model by color coding. If the modeling confidence value of the local area of the model is lower than the reconstruction acceptable threshold, the area is marked as an area that needs to be re-captured with images.

9. The rapid three-dimensional reconstruction method for the unmanned aerial vehicle mine inspection scenario according to claim 8, wherein In the environmental perception acquisition step, the wind disturbance stability factor is calculated by combining the openness of the slope, the current wind speed vector, and the UAV flight attitude. The calculation of the wind disturbance stability factor is obtained based on the extended Kalman filter method. If this factor is higher than the wind disturbance intensity threshold, the shooting interval time during flight is adjusted to the longest interval time, so as to maintain the lowest inter-frame displacement when collecting images in high-wind areas.

10. The rapid three-dimensional reconstruction method for the UAV mine inspection scenario according to claim 9, characterized in that, In the image quality optimization step, sequence images within a preset time window are obtained, and a consistency vector is constructed using the mean square error value of the gray difference rate between adjacent frames and the matching repetition rate of feature matching. The Euclidean distance value between this vector and a preset standard vector is used as the fusion score. If the fusion score is higher than the fusion threshold, the sequence images are fused by multi-frame averaging. Otherwise, the frame with the highest quality in the sequence images is selected according to the preset image selection strategy to participate in the modeling.

Images