Underground coal mine unmanned aerial vehicle inspection system based on visual SLAM navigation

By combining visual SLAM with laser inertial fusion technology, along with visual degradation perception and dynamic dust point processing, the problem of unstable positioning and inaccurate modeling in coal mine underground drone inspection systems under insufficient lighting and dusty environments has been solved, achieving stable inspection results and environmental adaptability.

CN121702377APending Publication Date: 2026-03-20SHANGHAI RUOFOW INTELLIGENT TECHNOLOGY CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202511975407.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-12-25
Publication Date
2026-03-20

AI Technical Summary

Technical Problem

Existing UAV underground inspection systems suffer from image quality degradation in coal mines with insufficient lighting and dust, leading to unstable positioning and modeling, and a lack of environmental adaptability and reliability.

Method used

Visual SLAM and laser inertial fusion technology are adopted. Visual and laser modeling results are constructed separately through visual degradation perception and dynamic dust scattering processing. The results are selectively fused during the fusion stage to adapt to different environmental interferences.

Benefits of technology

This improves the positioning stability and modeling reliability of UAVs in underground coal mine inspections, ensures accurate correlation between inspection path control and results, and enhances environmental adaptability and overall operational reliability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121702377A_ABST
    Figure CN121702377A_ABST
Patent Text Reader

Abstract

The invention discloses a coal mine underground unmanned aerial vehicle inspection system based on visual SLAM navigation, and the system comprises a data collection module which is used for collecting data and carrying out the synchronous preprocessing of the data, and generating a synchronous sequence; the visual degradation perception module is used for calculating a visual quality score based on ORB-SLAM3, generating a visual degradation mark and outputting a confirmation key frame set, a candidate key frame set and a corresponding map point set; the dust dynamic scatter point processing module is used for identifying and screening dust dynamic scatter points based on LIO-SAM to generate a screening point cloud sequence and outputting a laser inertial pose sequence; the inspection modeling and fusion module is used for constructing a visual modeling result and a laser modeling result and outputting a fusion pose sequence and a fusion map; and the inspection execution and result association module is used for executing inspection based on the fusion pose sequence and generating an inspection record. According to the method, stable navigation and environment modeling of underground unmanned aerial vehicle inspection are realized, and the positioning reliability and the inspection result association quality in a degraded environment are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation for unmanned aerial vehicles (UAVs), and more particularly to an underground UAV inspection system for coal mines based on visual SLAM navigation. Background Technology

[0002] Underground mine inspection is a crucial aspect of ensuring safe production in coal mines. Inspection tasks involve the continuous monitoring of the tunnel environment, equipment operating status, and potential safety hazards. With the development of drone technology, utilizing drones for autonomous inspections in the underground environment can reduce the risks of manual inspections and improve efficiency, placing higher demands on the positioning and navigation capabilities of drones.

[0003] Current UAV-based underground inspection methods mostly employ visual SLAM-based localization and mapping, using image features to achieve pose estimation and environmental modeling. However, underground coal mines commonly experience insufficient lighting, strong glare, and suspended dust, which easily leads to image quality degradation, unstable keyframes and map points, and affects the reliability of localization and modeling. To improve robustness, some technologies introduce laser-inertial fusion with lidar and inertial measurement units. However, in dusty environments, laser echoes are prone to generating dynamic scattered points, interfering with the point cloud modeling process.

[0004] Existing vision and laser fusion methods mostly adopt a uniform processing strategy, which lacks differentiation of different environmental factors such as dimness and dust interference. It is difficult to flexibly select the modeling method according to the actual inspection situation, which can easily introduce unreliable modeling results and affect the spatial correlation and application effect of inspection data.

[0005] Therefore, how to provide a visual SLAM navigation-based unmanned aerial vehicle (UAV) inspection system for coal mines is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a coal mine underground UAV inspection system based on visual SLAM navigation. This invention employs visual SLAM and laser inertial fusion technology to achieve autonomous positioning, environmental modeling, and inspection navigation for underground UAVs. By sensing dim lighting and dust interference, it rationally selects visual and laser modeling results to provide stable positioning and spatial correlation capabilities for inspection tasks. It has the advantages of strong environmental adaptability, high inspection reliability, and high safety.

[0007] A visual SLAM-based unmanned aerial vehicle (UAV) inspection system for underground coal mines according to an embodiment of the present invention includes: The data acquisition module is used to collect image data, point cloud data, and inertial measurement data during the drone inspection flight in the coal mine, and to perform time synchronization and preprocessing on the collected data to generate synchronized image sequences, synchronized point cloud sequences, and synchronized inertial data. The visual degradation perception module is used to process synchronized image sequences based on ORB-SLAM3, calculate visual quality scores and generate visual degradation markers, perform keyframe insertion control and map point management based on visual degradation markers, and output the confirmed keyframe set, candidate keyframe set and corresponding map point set. The dust dynamic scatter point processing module is used to perform dust dynamic scatter point identification and filtering on the synchronous point cloud sequence based on LIO-SAM, generate a filtered point cloud sequence, and calculate the laser inertial pose sequence based on the filtered point cloud sequence and synchronous inertial data. The inspection modeling and fusion module is used to construct visual modeling results based on the confirmed keyframe set and the confirmed map point set during the inspection process, construct laser modeling results based on laser inertial pose sequence and point cloud data, and fuse the visual modeling results and laser modeling results to generate fused pose sequence and fused map. The inspection execution and result association module is used to control the UAV to perform underground inspection flight missions in coal mines based on the fused pose sequence. During the inspection, it collects inspection images and spatially associates the inspection images with the fused pose sequence and fused map to form an inspection record with location information.

[0008] Optionally, modules can be integrated using the following methods: S1. The UAV collects data and performs preprocessing to obtain synchronized image sequences, synchronized point cloud sequences, and synchronized inertial data; S2. In ORB-SLAM3, calculate the visual quality score of the synchronized image sequence and generate a visual degradation state sequence. The visual degradation state sequence is written into the internal control parameters of ORB-SLAM3. S3 and ORB-SLAM3 perform keyframe insertion control based on the visual degradation state sequence, and output the confirmed keyframe set and the candidate keyframe set. S4 and ORB-SLAM3 perform map point management based on the visual degradation state sequence, writing the map points corresponding to the candidate keyframe set into the candidate map point set, and writing the map points corresponding to the confirmed keyframe set into the confirmed map point set. S5. Perform dynamic dust point identification and filtering on the synchronous point cloud sequence in LIO-SAM to obtain the filtered point cloud sequence. Based on the filtered point cloud sequence and synchronous inertial data, LIO-SAM outputs the laser inertial pose sequence and point cloud sub-map. S6. Based on the synchronized image sequence and synchronized inertial data, obtain the visual inertial pose sequence, construct a unified factor graph and add the laser inertial pose sequence, point cloud sub-map, visual inertial pose sequence, confirmed keyframe set, and confirmed map point set corresponding constraints, perform graph optimization fusion, and output the fused pose sequence and fused map. S7. Based on the fused pose sequence and fused map, perform confirmation and elimination cleaning on the candidate keyframe set and candidate map point set, and output the final pose sequence and final map.

[0009] Optionally, S2 specifically includes: S21. Calculate the global brightness average for each image frame in the synchronized image sequence, and compare the global brightness average with the preset imaging brightness threshold. When the global brightness average is lower than the imaging brightness threshold, mark the corresponding image frame as a dark light candidate frame. S22. Perform brightness change analysis on dark light candidate frames within a continuous time window. When the brightness change of several consecutive dark light candidate frames is less than the preset change threshold, determine that the corresponding time period is in a dim and degraded state. S23. Calculate the difference between the pixel brightness and the average brightness of the neighborhood for each pixel in the image frame. When the difference exceeds the brightness anomaly threshold, mark the corresponding pixel as a brightness anomaly point and perform spatial clustering on the brightness anomaly points to generate a set of brightness anomaly regions. S24. Calculate the area of ​​the set of abnormal brightness areas. When the area is less than the preset area threshold, mark the corresponding area as a dust area. S25. Perform position and shape change analysis on the dust area in consecutive image frames. When the blurred abnormal area shows position drift or shape change characteristics in consecutive frames, the corresponding area is determined to be a dust interference area. S26. Based on the area ratio of the dimly degraded state and the dust interference area in the image frame, a visual quality score is generated through weighted averaging and exponential mapping, and the visual quality scores corresponding to consecutive image frames are combined into a visual degradation state sequence.

[0010] Optionally, S3 specifically includes: S31. Receive visual quality scores in ORB-SLAM3. The visual quality scores are defined as continuous values ​​in the range of 0 to 1, and are divided into high quality, medium quality and low quality ranges according to the size of the visual quality scores. S32. When the visual quality score corresponding to the current image frame is in the high quality range, the current image frame is determined as a formal key frame and written into the confirmation key frame set. S33. When the visual quality score corresponding to the current image frame is in the medium quality range, the current image frame is determined as a candidate keyframe and written into the candidate keyframe set. S34. When the visual quality score corresponding to the current image frame is in the low quality range, the current image frame is determined to be a tracking frame. The tracking frame only performs pose tracking and is prohibited from participating in mapping.

[0011] Optionally, S4 specifically includes: S41. Receive the confirmed keyframe set and candidate keyframe set output in step S3 in ORB-SLAM3, and perform validity checks on the map points associated with the confirmed keyframe set. S42. Perform temporary storage management on the map points associated with the candidate keyframe set, mark the corresponding map points as candidate map points and restrict the candidate map points from participating in global mapping and global optimization, and only allow the candidate map points to participate in local matching and tracking calculations.

[0012] Optionally, the dynamic dust scatter point identification in S5 specifically includes: S51. Read the ranging value, echo intensity value, scan line number and sampling time corresponding to each point in the synchronous point cloud sequence, and perform distortion removal processing on the point cloud frame in combination with the synchronous inertial data to obtain the distortion removal point cloud frame. S52. Construct a co-bundle sequence within the distortion-free point cloud frame using the scan line number as an index, perform first-order and second-order difference calculations on the co-bundle sequence, and extract the set of isolated short-range abrupt change points. S53. Construct a cross-beam neighborhood within the distortion-free point cloud frame based on the adjacent relationship of scan line numbers. Perform a cross-beam consistency check on the set of isolated short-distance mutation points. Calculate the spatial continuity of the distance measurement corresponding to the mutation point in the cross-beam neighborhood. When the cross-beam neighborhood does not form a continuous surface intersection line, mark the mutation point as a dust candidate point. S54. Perform temporal reproducibility test on dust candidate points in continuous distortion-free point cloud frames, establish a neighborhood grid index with sampling time as the sequence, calculate the number of times dust candidate points reappear in the same grid neighborhood in subsequent point cloud frames and the ranging stability. When the number of reappearances is lower than the preset number threshold and the ranging stability is lower than the preset stability threshold, the dust candidate points are determined to be dust dynamic scattered points. S55. Remove the dynamic dust scatter points to obtain the filtered point cloud sequence.

[0013] Optionally, the output process in S5 specifically includes: S56. Input the selected point cloud sequence after removing dust dynamic scattered points and synchronous inertial data into LIO-SAM, and construct laser-inertial joint constraints based on the inertial pre-integration results and the selected point cloud sequence. S57. In LIO-SAM, the laser-inertial joint constraints are used to perform pose recursion and optimization calculations to obtain a continuous laser-inertial pose sequence. S58. Perform coordinate transformation and cumulative update on the selected point cloud sequence based on the laser inertial pose sequence.

[0014] Optionally, S6 specifically includes: S61. Based on the visual inertial pose sequence output by ORB-SLAM3, the confirmed keyframe set and the confirmed map point set, a visual modeling result is constructed. Based on the laser inertial pose sequence output by LIO-SAM and the point cloud sub-map, a laser modeling result is constructed. The visual modeling result uses keyframes and map points to form a sparse spatial structure. The laser modeling result uses laser inertial pose and point cloud accumulation to form a sub-map structure. S62. Perform time and space alignment on the visual modeling results and the laser modeling results to establish the correspondence between visual pose and laser inertial pose within the same time period. S63. Perform pose difference calculation on the corresponding visual pose and laser inertial pose, and determine whether the visual modeling result should participate in the fusion modeling or be temporarily deferred based on the pose difference result. S64. The visual modeling results and laser modeling results participating in the fusion modeling are written together into the fusion modeling structure to generate a fusion pose sequence; S65. Based on the fused pose sequence, perform a unified update on the visual sparse map structure and the point cloud sub-map to generate a fused map.

[0015] Optionally, S7 specifically includes: S71. Read the candidate keyframe set and candidate map point set retained during the fusion modeling process, and obtain the visual degradation markers corresponding to the candidate keyframes; S72. When the visual degradation marker corresponding to the candidate keyframe indicates dim degradation, the candidate keyframe and candidate map points are prohibited from participating in visual modeling, and the laser modeling results output by LIO-SAM are directly used to update the fused map. S73. When the visual degradation marker corresponding to the candidate keyframe does not indicate dim degradation, dust interference judgment is performed on the image frame corresponding to the candidate keyframe, and dust dynamic scatter point judgment is performed on the point cloud data in LIO-SAM within the same time period. S74. When both the image frame dust interference determination result and the point cloud dust dynamic scatter point determination result indicate the presence of dust interference, the current state is determined to be a dust interference state, the UAV's movement speed is reduced, and the fused map is updated using the visual modeling results of ORB-SLAM3. S75. When the dust interference determination result of the image frame is different from the dust dynamic scatter point determination result of the point cloud, the candidate key frame is converted into the input key frame and written into ORB-SLAM3. The visual modeling result of ORB-SLAM3 and the laser modeling result of LIO-SAM are used together to update the fused map and output the corresponding image data.

[0016] S76. After completing the operations on all candidate keyframes and candidate map points, output the final pose sequence and the final map.

[0017] The beneficial effects of this invention are: (1) By introducing a visual degradation perception and dust dynamic scattering processing mechanism into the inspection process of UAVs, this invention can distinguish different degradation causes such as the dim environment and dust interference in coal mines, and adopt appropriate visual or laser modeling methods respectively, effectively avoiding low-quality images or dust scattering points from participating in modeling, reducing the impact of incorrect modeling on the inspection positioning results from the source, and improving the stability of positioning and modeling in the underground inspection process.

[0018] (2) By constructing visual modeling results and laser modeling results separately, and selectively fusing the two types of modeling results in the fusion stage, the present invention enables the UAV to maintain continuous and reliable pose output and spatial modeling capabilities in complex downhole environments, ensuring the accurate correlation between inspection path control and inspection data and spatial position, and improving the availability and consistency of inspection results.

[0019] (3) The present invention uses the modeling results as supporting information for inspection navigation and result association, rather than as an independent target, so that the UAV can dynamically adjust its flight and modeling strategies according to environmental changes during the inspection process, ensure inspection safety in dim or dusty environments, and make full use of visual information when the lighting conditions are good, thereby improving the environmental adaptability and overall operational reliability of UAV inspection in coal mines. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0021] Fig. 1 This is a structural diagram of the coal mine underground unmanned aerial vehicle (UAV) inspection system based on visual SLAM navigation proposed in this invention. Fig. 2 The flowchart shows the coal mine underground drone inspection system based on visual SLAM navigation proposed in this invention. Fig. 3 This is a flowchart illustrating the final modeling decision-making process for the visual SLAM-based unmanned aerial vehicle (UAV) inspection system for underground coal mines proposed in this invention. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0023] refer to Figs. 1-3 A visual SLAM-based unmanned aerial vehicle (UAV) inspection system for underground coal mines includes: The data acquisition module is used to collect image data, point cloud data, and inertial measurement data during the drone inspection flight in the coal mine, and to perform time synchronization and preprocessing on the collected data to generate synchronized image sequences, synchronized point cloud sequences, and synchronized inertial data. The visual degradation perception module is used to process synchronized image sequences based on ORB-SLAM3, calculate visual quality scores and generate visual degradation markers, perform keyframe insertion control and map point management based on visual degradation markers, and output the confirmed keyframe set, candidate keyframe set and corresponding map point set. The dust dynamic scatter point processing module is used to perform dust dynamic scatter point identification and filtering on the synchronous point cloud sequence based on LIO-SAM, generate a filtered point cloud sequence, and calculate the laser inertial pose sequence based on the filtered point cloud sequence and synchronous inertial data. The inspection modeling and fusion module is used to construct visual modeling results based on the confirmed keyframe set and the confirmed map point set during the inspection process, construct laser modeling results based on laser inertial pose sequence and point cloud data, and fuse the visual modeling results and laser modeling results to generate fused pose sequence and fused map. The inspection execution and result association module is used to control the UAV to perform underground inspection flight missions in coal mines based on the fused pose sequence. During the inspection, it collects inspection images and spatially associates the inspection images with the fused pose sequence and fused map to form an inspection record with location information.

[0024] In this embodiment, the modules are interconnected using the following method: S1. The UAV collects data and performs preprocessing to obtain synchronized image sequences, synchronized point cloud sequences, and synchronized inertial data; S2. In ORB-SLAM3, calculate the visual quality score of the synchronized image sequence and generate a visual degradation state sequence. The visual degradation state sequence is written into the internal control parameters of ORB-SLAM3. S3 and ORB-SLAM3 perform keyframe insertion control based on the visual degradation state sequence, and output the confirmed keyframe set and the candidate keyframe set. S4 and ORB-SLAM3 perform map point management based on the visual degradation state sequence, writing the map points corresponding to the candidate keyframe set into the candidate map point set, and writing the map points corresponding to the confirmed keyframe set into the confirmed map point set. S5. Perform dynamic dust point identification and filtering on the synchronous point cloud sequence in LIO-SAM to obtain the filtered point cloud sequence. Based on the filtered point cloud sequence and synchronous inertial data, LIO-SAM outputs the laser inertial pose sequence and point cloud sub-map. S6. Based on the synchronized image sequence and synchronized inertial data, obtain the visual inertial pose sequence, construct a unified factor graph and add the laser inertial pose sequence, point cloud sub-map, visual inertial pose sequence, confirmed keyframe set, and confirmed map point set corresponding constraints, perform graph optimization fusion, and output the fused pose sequence and fused map. S7. Based on the fused pose sequence and fused map, perform confirmation and elimination cleaning on the candidate keyframe set and candidate map point set, and output the final pose sequence and final map.

[0025] In this embodiment, S1 specifically includes: During the inspection flight, the UAV acquires multi-source sensor data, which consists of image data, point cloud data, and inertial measurement data. The image data is continuously acquired by the airborne visual sensing device to form an image sequence, the point cloud data is continuously scanned by the airborne laser rangefinder to form a point cloud sequence, and the inertial measurement data is output by the airborne inertial measurement device to form an inertial data sequence. The acquired image sequence, point cloud sequence, and inertial data sequence are time-aligned to establish a unified time axis; the image sequence is format-unified and distortion-corrected to obtain a synchronized image sequence; the point cloud sequence is coordinate-unified and invalid data is removed to obtain a synchronized point cloud sequence with consistent structure; and the inertial data sequence is offset-compensated and rearranged to obtain continuous and stable synchronized inertial data.

[0026] In this embodiment, S2 specifically includes: S21. Calculate the global average brightness for each image frame in the synchronized image sequence, and compare the global average brightness with the preset imaging brightness threshold. When the global average brightness is lower than the imaging brightness threshold, mark the corresponding image frame as a dark light candidate frame. The imaging brightness threshold is set to 40. S22. Perform brightness change analysis on dark light candidate frames within a continuous time window. When the brightness change of several consecutive dark light candidate frames is less than the preset change threshold, it is determined that the corresponding time period is in a dim and degraded state; for 5 consecutive frames, the change threshold is 5. S23. Calculate the difference between the pixel brightness and the average brightness of the neighborhood for each pixel in the image frame. When the difference exceeds the brightness anomaly threshold, mark the corresponding pixel as a brightness anomaly point and perform spatial clustering on the brightness anomaly points to generate a set of brightness anomaly regions. The neighborhood window is set to 7×7 and the brightness anomaly threshold is 25. S24. Calculate the area of ​​the set of abnormal brightness areas. When the area is less than a preset area threshold, mark the corresponding area as a dust area. The area threshold is 0.3% of the total area. S25. Perform position and shape change analysis on the dust area in consecutive image frames. When the blurred abnormal area shows position drift or shape change characteristics in consecutive frames, the corresponding area is determined to be a dust interference area. For 3 consecutive frames, the position drift threshold is 5 pixels / frame, that is, the centroid displacement of the area is greater than 5 pixels and it is considered a drift. The intersection-union ratio of the bounding boxes of adjacent frames is less than 0.3 and it is considered a shape change. If there are at least 2 instances of "position drift" or "shape change" within the 3-frame window, it is determined to be a dust interference area. The area ratio threshold of the dust interference area is set to 2% of the total area as a limit. The area is calculated in pixels. S26. Based on the area ratio of the dimly degraded state and the dust interference area in the image frame, a visual quality score is generated through weighted averaging and exponential mapping. The visual quality scores corresponding to consecutive image frames are combined into a visual degradation state sequence. The weighting method is to first record the maximum and minimum values ​​of the area and brightness of all the dust interference areas that have been obtained, normalize the current area and brightness of the dust interference area to the maximum and minimum values, and calculate the weights according to the obtained proportions. The weights sum to 1. The maximum and minimum values ​​are the same when initialized. The exponential mapping adopts an exponential mapping method with a fixed exponential growth coefficient of 3.0.

[0027] In this embodiment, S3 specifically includes: S31. Receive visual quality scores in ORB-SLAM3. Visual quality scores are defined as continuous values ​​in the range of 0 to 1, and are divided into high quality, medium quality, and low quality ranges according to the size of the visual quality scores. In this embodiment, the total score range is divided into three ranges on average. S32. When the visual quality score corresponding to the current image frame is in the high quality range, the current image frame is determined as a formal key frame and written into the confirmation key frame set. S33. When the visual quality score corresponding to the current image frame is in the medium quality range, the current image frame is determined as a candidate keyframe and written into the candidate keyframe set. S34. When the visual quality score corresponding to the current image frame is in the low quality range, the current image frame is determined to be a tracking frame. The tracking frame only performs pose tracking and is prohibited from participating in mapping.

[0028] In this embodiment, S4 specifically includes: S41. Receive the confirmed keyframe set and candidate keyframe set output by S3 in ORB-SLAM3, and perform validity checks on the map points associated with the confirmed keyframe set. S42. Perform temporary storage management on the map points associated with the candidate keyframe set, mark the corresponding map points as candidate map points and restrict the candidate map points from participating in global mapping and global optimization, and only allow the candidate map points to participate in local matching and tracking calculations.

[0029] In this embodiment, the parameters of ORB-SLAM3 are configured as follows: The maximum number of ORB feature points per frame is 1000; the ORB pyramid has 8 layers; the ORB scale factor is 1.2; the initial FAST corner threshold is 20, with a minimum of 7; the minimum keyframe insertion interval is 15 frames; confirmed map points must be observed by at least 3 confirmed keyframes; the maximum number of candidate map points retained is 20 frames, and those that have not been upgraded to confirmed map points after more than 20 frames are removed; visual inertial odometry parameters include an IMU pre-integration time window with a 1-frame keyframe interval; IMU noise parameters include an acceleration noise standard deviation of 0.02 and an angular velocity noise standard deviation of 0.015; the bag-of-words model DBoW2 for loop closure detection has 10 local BA keyframes in the local optimization window size.

[0030] In this embodiment, the dynamic dust scatter point identification in S5 specifically includes: S51. Read the ranging value, echo intensity value, scan line number and sampling time corresponding to each point in the synchronous point cloud sequence, and perform distortion removal processing on the point cloud frame in combination with the synchronous inertial data to obtain the distortion removal point cloud frame. S52. Construct a co-bundle sequence within the distortion-free point cloud frame using the scan line number as an index, perform first-order and second-order difference calculations on the co-bundle sequence, and extract the set of isolated short-range abrupt change points. S53. Construct a cross-beam neighborhood within the distortion-reduced point cloud frame based on the adjacent relationship of scan line numbers. Perform a cross-beam consistency check on the set of isolated short-range mutation points. Calculate the spatial continuity of the distance corresponding to the mutation point in the cross-beam neighborhood. When the cross-beam neighborhood does not form a continuous surface intersection line, mark the mutation point as a dust candidate point. This operation is the routine structural analysis in point cloud processing. S54. Perform temporal reproducibility testing on dust candidate points in consecutive distortion-free point cloud frames, establish a neighborhood raster index with sampling time as the sequence, calculate the recurrence count and ranging stability of dust candidate points in the same raster neighborhood in subsequent point cloud frames, and determine dust candidate points as dust dynamic scattered points when the recurrence count is lower than a preset count threshold and the ranging stability is lower than a preset stability threshold; the count threshold is 2 and the ranging stability threshold is 0.30m. S55. Remove the dynamic dust scatter points to obtain the filtered point cloud sequence.

[0031] In this embodiment, the output process in S5 specifically includes: S56. Input the selected point cloud sequence after removing dust dynamic scattered points and synchronous inertial data into LIO-SAM, and construct laser-inertial joint constraints based on the inertial pre-integration results and the selected point cloud sequence. S57. In LIO-SAM, the laser-inertial joint constraints are used to perform pose recursion and optimization calculations to obtain a continuous laser-inertial pose sequence. S58. Perform coordinate transformation and cumulative update on the selected point cloud sequence based on the laser inertial pose sequence.

[0032] This part is the standard LIO-SAM process. In this implementation, the parameters are configured as follows: The laser scanning frequency is 10Hz, the number of laser scanning lines is 16, the horizontal angular resolution is 0.2°, the maximum effective range is 100m, and the minimum effective range is 1m. The IMU related parameter configuration is consistent with ORB-SLAM3. The IMU pre-integration time span is a single frame point cloud scanning cycle, the weight ratio of laser factor to IMU factor is 1:1, and the pose optimization window size is 10 frames. The planar feature extraction threshold is 0.10m, the edge feature extraction threshold is 0.15m, the maximum number of points participating in mapping per frame is 2000, and the point cloud sub-map update interval is 1 frame.

[0033] In this embodiment, S6 specifically includes: S61. Based on the visual inertial pose sequence output by ORB-SLAM3, the confirmed keyframe set and the confirmed map point set, a visual modeling result is constructed. Based on the laser inertial pose sequence output by LIO-SAM and the point cloud sub-map, a laser modeling result is constructed. The visual modeling result uses keyframes and map points to form a sparse spatial structure. The laser modeling result uses laser inertial pose and point cloud accumulation to form a sub-map structure. S62. Perform time and space alignment on the visual modeling results and the laser modeling results to establish the correspondence between visual pose and laser inertial pose within the same time period. S63. Perform pose difference calculation on the corresponding visual pose and laser inertial pose, and determine whether the visual modeling result should participate in the fusion modeling or be temporarily suspended from participating in the fusion modeling based on the pose difference result; the acceptable translation difference threshold is 0.50m, and the acceptable rotation difference threshold is 10°. If both are met, the visual modeling result should participate in the fusion modeling, and the average of the two coordinates should be used. If the conditions are not met, the visual modeling result should be temporarily suspended from participating in the modeling. S64. The visual modeling results and laser modeling results participating in the fusion modeling are written together into the fusion modeling structure to generate a fusion pose sequence; S65. Based on the fused pose sequence, perform a unified update on the visual sparse map structure and the point cloud sub-map to generate a fused map.

[0034] In this embodiment, S7 specifically includes: S71. Read the candidate keyframe set and candidate map point set retained during the fusion modeling process, and obtain the visual degradation markers corresponding to the candidate keyframes; S72. When the visual degradation marker corresponding to the candidate keyframe indicates dim degradation, the candidate keyframe and candidate map points are prohibited from participating in visual modeling, and the laser modeling results output by LIO-SAM are directly used to update the fused map. S73. When the visual degradation marker corresponding to the candidate keyframe does not indicate dim degradation, dust interference judgment is performed on the image frame corresponding to the candidate keyframe, and dust dynamic scatter point judgment is performed on the point cloud data in LIO-SAM within the same time period. S74. When both the image frame dust interference determination result and the point cloud dust dynamic scatter point determination result indicate the presence of dust interference, the current state is determined to be a dust interference state, the UAV's movement speed is reduced, and the fused map is updated using the visual modeling results of ORB-SLAM3. S75. When the dust interference determination result of the image frame is different from the dust dynamic scatter point determination result of the point cloud, the candidate key frame is converted into the input key frame and written into ORB-SLAM3. The visual modeling result of ORB-SLAM3 and the laser modeling result of LIO-SAM are used to update the fused map together and output the corresponding image data. In this embodiment, the joint update here is the process of S6, and the corresponding image data is output for comparison. S76. After completing the operations on all candidate keyframes and candidate map points, output the final pose sequence and the final map.

[0035] Example 1: To verify the feasibility of this invention in practice, it was applied to a coal mine underground drone inspection scenario. In this scenario, the inspection area has a complex spatial structure, the roadway has multiple turns, the environment is under low light for extended periods, and there is periodic dust suspension. Traditional manual inspection methods suffer from high operational risks and discontinuous inspection coverage. Existing drone inspection technologies in such environments are prone to unstable positioning, distorted modeling, and difficulty in accurately correlating inspection images with spatial locations, severely impacting the reliability and application value of inspection results.

[0036] In this scenario, the drone performs autonomous flight along a pre-set inspection path, simultaneously acquiring environmental perception data through onboard visual sensors, laser rangefinders, and inertial measurement units. The acquired multi-source data first undergoes time synchronization and preprocessing to generate synchronized image sequences, synchronized point cloud sequences, and synchronized inertial data.

[0037] During the inspection flight, the system analyzes continuous images based on a visual degradation perception mechanism. By calculating image brightness features and local anomaly region features, the system can distinguish between overall dim lighting and dust interference, and convert the results into visual degradation markers. In areas with insufficient lighting but weak dust influence, the system automatically reduces the involvement of visual modeling to avoid introducing unstable map points from low-quality images; in areas with dust interference, the system separately judges visual information and laser information.

[0038] Before entering the laser inertial modeling process, the synchronized point cloud sequence undergoes dynamic dust point identification and filtering. Through in-beam ranging abrupt change analysis, cross-beam spatial continuity verification, and temporal reproducibility analysis, the system can effectively identify dynamic anomalies caused by dust scattering and remove them from the point cloud. The filtered point cloud sequence is then input into the laser inertial modeling process to generate a stable laser inertial pose sequence and a point cloud sub-map.

[0039] During the inspection process, the system constructs visual modeling results and laser modeling results separately, and aligns the two types of modeling results in time and space. By analyzing the difference between the visual pose and the laser inertial pose, the system can determine whether the visual modeling result is suitable for fusion modeling. In sections with severe visual degradation, the system prioritizes the use of laser modeling results; in sections with good visual conditions and pose differences meeting a set threshold, the system uses both visual modeling results and laser modeling results for fusion modeling.

[0040] Through the above method, the system maintains continuous and stable pose output throughout the inspection process and generates a fused map consistent with the inspection path. Inspection images can be accurately correlated to their corresponding spatial locations after acquisition. Compared to traditional methods that do not incorporate visual degradation perception and dynamic dust point processing, this implementation demonstrates advantages in positioning stability, modeling integrity, and the usability of inspection results.

[0041] Statistical results from multiple inspections show that, after adopting the method of this invention, the number of pose estimation interruptions during UAV inspections is significantly reduced, the proportion of invalid points in the fused map is significantly decreased, and the success rate of matching inspection images with spatial locations is improved. In areas with significant dust interference, the system can maintain a stable inspection speed and output continuous modeling results, effectively avoiding inspection failures caused by environmental interference.

[0042] Table 1: Comparison of Visual-LiDAR Detection Performance

[0043] As shown in Table 1, traditional vision methods exhibit significant positioning errors and a high number of modeling interruptions in low-light and dusty environments, resulting in a low success rate in associating inspection images with spatial locations, making it difficult to meet the stable operation requirements of complex inspection environments. Traditional laser methods offer some improvement in positioning accuracy, but still suffer from insufficient modeling stability in dusty areas. Simple fusion methods improve overall performance to some extent, but due to a lack of differentiation between environmental degradation types, they still suffer from a high proportion of invalid map points and unstable modeling results.

[0044] In comparison, the method of this invention demonstrates advantages in all indicators. Positioning errors are reduced, the number of modeling interruptions is minimized, the proportion of invalid map points is effectively controlled, and the success rate of associating inspection images with spatial locations approaches perfect matching. Even in areas with significant dust interference, the method of this invention can maintain a high proportion of stable modeling output, indicating that the introduced visual degradation perception and dynamic dust point processing mechanism can effectively improve the overall reliability and applicability of the UAV inspection system in complex environments.

[0045] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A visual SLAM navigation-based unmanned aerial vehicle (UAV) inspection system for underground coal mines, characterized in that, include: The data acquisition module is used to collect image data, point cloud data, and inertial measurement data during the drone inspection flight in the coal mine, and to perform time synchronization and preprocessing on the collected data to generate synchronized image sequences, synchronized point cloud sequences, and synchronized inertial data. The visual degradation perception module is used to process synchronized image sequences based on ORB-SLAM3, calculate visual quality scores and generate visual degradation markers, perform keyframe insertion control and map point management based on visual degradation markers, and output the confirmed keyframe set, candidate keyframe set and corresponding map point set. The dust dynamic scatter point processing module is used to perform dust dynamic scatter point identification and filtering on the synchronous point cloud sequence based on LIO-SAM, generate a filtered point cloud sequence, and calculate the laser inertial pose sequence based on the filtered point cloud sequence and synchronous inertial data. The inspection modeling and fusion module is used to construct visual modeling results based on the confirmed keyframe set and the confirmed map point set during the inspection process, construct laser modeling results based on laser inertial pose sequence and point cloud data, and fuse the visual modeling results and laser modeling results to generate fused pose sequence and fused map. The inspection execution and result association module is used to control the UAV to perform underground inspection flight missions in coal mines based on the fused pose sequence. During the inspection, it collects inspection images and spatially associates the inspection images with the fused pose sequence and fused map to form an inspection record with location information.

2. The coal mine underground UAV inspection system based on visual SLAM navigation according to claim 1, characterized in that, The modules are connected in the following way: S1. The UAV collects data and performs preprocessing to obtain synchronized image sequences, synchronized point cloud sequences, and synchronized inertial data; S2. In ORB-SLAM3, calculate the visual quality score of the synchronized image sequence and generate a visual degradation state sequence. The visual degradation state sequence is written into the internal control parameters of ORB-SLAM3. S3 and ORB-SLAM3 perform keyframe insertion control based on the visual degradation state sequence, and output the confirmed keyframe set and the candidate keyframe set. S4 and ORB-SLAM3 perform map point management based on the visual degradation state sequence, writing the map points corresponding to the candidate keyframe set into the candidate map point set, and writing the map points corresponding to the confirmed keyframe set into the confirmed map point set. S5. Perform dynamic dust point identification and filtering on the synchronous point cloud sequence in LIO-SAM to obtain the filtered point cloud sequence. Based on the filtered point cloud sequence and synchronous inertial data, LIO-SAM outputs the laser inertial pose sequence and point cloud sub-map. S6. Based on the synchronized image sequence and synchronized inertial data, obtain the visual inertial pose sequence, construct a unified factor graph and add the laser inertial pose sequence, point cloud sub-map, visual inertial pose sequence, confirmed keyframe set, and confirmed map point set corresponding constraints, perform graph optimization fusion, and output the fused pose sequence and fused map. S7. Based on the fused pose sequence and fused map, perform confirmation and elimination cleaning on the candidate keyframe set and candidate map point set, and output the final pose sequence and final map.

3. The coal mine underground UAV inspection system based on visual SLAM navigation according to claim 2, characterized in that, S2 specifically includes: S21. Calculate the global brightness average for each image frame in the synchronized image sequence, and compare the global brightness average with the preset imaging brightness threshold. When the global brightness average is lower than the imaging brightness threshold, mark the corresponding image frame as a dark light candidate frame. S22. Perform brightness change analysis on dark light candidate frames within a continuous time window. When the brightness change of several consecutive dark light candidate frames is less than the preset change threshold, determine that the corresponding time period is in a dim and degraded state. S23. Calculate the difference between the pixel brightness and the average brightness of the neighborhood for each pixel in the image frame. When the difference exceeds the brightness anomaly threshold, mark the corresponding pixel as a brightness anomaly point and perform spatial clustering on the brightness anomaly points to generate a set of brightness anomaly regions. S24. Calculate the area of ​​the set of abnormal brightness areas. When the area is less than the preset area threshold, mark the corresponding area as a dust area. S25. Perform position and shape change analysis on the dust area in consecutive image frames. When the blurred abnormal area shows position drift or shape change characteristics in consecutive frames, the corresponding area is determined to be a dust interference area. S26. Based on the area ratio of the dimly degraded state and the dust interference area in the image frame, a visual quality score is generated through weighted averaging and exponential mapping, and the visual quality scores corresponding to consecutive image frames are combined into a visual degradation state sequence.

4. The coal mine underground UAV inspection system based on visual SLAM navigation according to claim 3, characterized in that, S3 specifically includes: S31. Receive visual quality scores in ORB-SLAM3. The visual quality scores are defined as continuous values ​​in the range of 0 to 1, and are divided into high quality, medium quality and low quality ranges according to the size of the visual quality scores. S32. When the visual quality score corresponding to the current image frame is in the high quality range, the current image frame is determined as a formal key frame and written into the confirmation key frame set. S33. When the visual quality score corresponding to the current image frame is in the medium quality range, the current image frame is determined as a candidate keyframe and written into the candidate keyframe set. S34. When the visual quality score corresponding to the current image frame is in the low quality range, the current image frame is determined to be a tracking frame. The tracking frame only performs pose tracking and is prohibited from participating in mapping.

5. The coal mine underground UAV inspection system based on visual SLAM navigation according to claim 4, characterized in that, S4 specifically includes: S41. Receive the set of confirmed keyframes and the set of candidate keyframes in ORB-SLAM3, and perform validity checks on the map points associated with the set of confirmed keyframes. S42. Perform temporary storage management on the map points associated with the candidate keyframe set, mark the corresponding map points as candidate map points and restrict the candidate map points from participating in global mapping and global optimization, and only allow the candidate map points to participate in local matching and tracking calculations.

6. The coal mine underground UAV inspection system based on visual SLAM navigation according to claim 5, characterized in that, The dynamic dust scatter point identification in S5 specifically includes: S51. Read the ranging value, echo intensity value, scan line number and sampling time corresponding to each point in the synchronous point cloud sequence, and perform distortion removal processing on the point cloud frame in combination with the synchronous inertial data to obtain the distortion removal point cloud frame. S52. Construct a co-bundle sequence within the distortion-free point cloud frame using the scan line number as an index, perform first-order and second-order difference calculations on the co-bundle sequence, and extract the set of isolated short-range abrupt change points. S53. Construct a cross-beam neighborhood within the distortion-free point cloud frame based on the adjacent relationship of scan line numbers. Perform a cross-beam consistency check on the set of isolated short-distance mutation points. Calculate the spatial continuity of the distance measurement corresponding to the mutation point in the cross-beam neighborhood. When the cross-beam neighborhood does not form a continuous surface intersection line, mark the mutation point as a dust candidate point. S54. Perform temporal reproducibility test on dust candidate points in continuous distortion-free point cloud frames, establish a neighborhood grid index with sampling time as the sequence, calculate the number of times dust candidate points reappear in the same grid neighborhood in subsequent point cloud frames and the ranging stability. When the number of reappearances is lower than the preset number threshold and the ranging stability is lower than the preset stability threshold, the dust candidate points are determined to be dust dynamic scattered points. S55. Remove the dynamic dust scatter points to obtain the filtered point cloud sequence.

7. The coal mine underground unmanned aerial vehicle inspection system based on visual SLAM navigation according to claim 6, characterized in that, The output process in S5 specifically includes: S56. Input the selected point cloud sequence after removing dust dynamic scattered points and synchronous inertial data into LIO-SAM, and construct laser-inertial joint constraints based on the inertial pre-integration results and the selected point cloud sequence. S57. In LIO-SAM, the laser-inertial joint constraints are used to perform pose recursion and optimization calculations to obtain a continuous laser-inertial pose sequence. S58. Perform coordinate transformation and cumulative update on the selected point cloud sequence based on the laser inertial pose sequence.

8. The coal mine underground UAV inspection system based on visual SLAM navigation according to claim 7, characterized in that, S6 specifically includes: S61. Based on the visual inertial pose sequence output by ORB-SLAM3, the confirmed keyframe set and the confirmed map point set, a visual modeling result is constructed. Based on the laser inertial pose sequence output by LIO-SAM and the point cloud sub-map, a laser modeling result is constructed. The visual modeling result uses keyframes and map points to form a sparse spatial structure. The laser modeling result uses laser inertial pose and point cloud accumulation to form a sub-map structure. S62. Perform time and space alignment on the visual modeling results and the laser modeling results to establish the correspondence between visual pose and laser inertial pose within the same time period. S63. Perform pose difference calculation on the corresponding visual pose and laser inertial pose, and determine whether the visual modeling result should participate in the fusion modeling or be temporarily deferred based on the pose difference result. S64. The visual modeling results and laser modeling results participating in the fusion modeling are written together into the fusion modeling structure to generate a fusion pose sequence; S65. Based on the fused pose sequence, perform a unified update on the visual sparse map structure and the point cloud sub-map to generate a fused map.

9. The coal mine underground UAV inspection system based on visual SLAM navigation according to claim 8, characterized in that, Specifically, S7 includes: S71. Read the candidate keyframe set and candidate map point set retained during the fusion modeling process, and obtain the visual degradation markers corresponding to the candidate keyframes; S72. When the visual degradation marker corresponding to the candidate keyframe indicates dim degradation, the candidate keyframe and candidate map points are prohibited from participating in visual modeling, and the laser modeling results output by LIO-SAM are directly used to update the fused map. S73. When the visual degradation marker corresponding to the candidate keyframe does not indicate dim degradation, dust interference judgment is performed on the image frame corresponding to the candidate keyframe, and dust dynamic scatter point judgment is performed on the point cloud data in LIO-SAM within the same time period. S74. When both the image frame dust interference determination result and the point cloud dust dynamic scatter point determination result indicate the presence of dust interference, the current state is determined to be a dust interference state, the UAV's movement speed is reduced, and the fused map is updated using the visual modeling results of ORB-SLAM3. S75. When the image frame dust interference determination result is different from the point cloud dust dynamic scatter point determination result, the candidate key frame is converted into the input key frame and written into ORB-SLAM3. The visual modeling result of ORB-SLAM3 and the laser modeling result of LIO-SAM are used together to update the fused map and output the corresponding image data. S76. After completing the operations on all candidate keyframes and candidate map points, output the final pose sequence and the final map.