Multi-sensor close-coupled slam and safety monitoring method, device and equipment for underground mine and storage medium

Abstract

The application discloses a kind of underground mine-oriented multi-sensor tight coupling SLAM and safety monitoring method, device, equipment and storage medium, the method includes: based on IMU pre-integration result to the multi-sensor data collected is preprocessed, based on corrected point cloud data generation global point cloud map, respectively from global point cloud map, thermal infrared data and visible light image data extract geometric semantic features, infrared semantic features and texture semantic features, and fused semantic features generate multi-modal information semantic map, based on multi-modal information semantic map identifies risk area, risk area includes water seepage area and deformation area, risk area is embedded into navigation cost map, and based on global risk index, the navigation path of mine movable carrier is monitored safety, to effectively couple analysis multi-sensor data, significantly improve the positioning accuracy and reliability under the complex environment of mine, provide technical support for unmanned inspection and mine safety production.

Description

Technical Field

[0001] This invention relates to the field of intelligent safety technology in mines, and in particular to a multi-sensor tightly coupled SLAM and safety monitoring method, device, equipment and storage medium for underground mines. Background Technology

[0002] Simultaneous localization and mapping (SLAM) technology is core to realizing applications such as unmanned driving and intelligent inspection in underground mines. However, the underground mine environment is characterized by poor lighting conditions, severe dust interference, complex spatial structure, high similarity, unstructured environment, and complete absence of GNSS signals, posing serious challenges to the accuracy, robustness, and real-time performance of SLAM systems.

[0003] Existing SLAM schemes each have their limitations: pure LiDAR-based schemes (such as LOAM and LeGO-LOAM) are prone to cumulative drift in feature-sparse tunnels and are sensitive to point cloud distortion; visual SLAM schemes are basically ineffective in low-light and high-dust environments; while existing multi-sensor fusion schemes, such as LIO-SAM, although they fuse IMU and LiDAR data through factor maps, suffer from poor initialization and decreased positioning accuracy during vigorous robot movements; the FAST-LIO2 series of algorithms, although highly efficient, lacks robustness in front-end feature extraction and exhibits poor stability in dynamic environments. Furthermore, existing technologies generally lack effective utilization of the unique environment of mines and fail to deeply integrate high-precision SLAM results with dynamic perception of mine safety risks, such as deformation, cracks, and water seepage, making it difficult to form a closed loop from perception to safety decision-making. Summary of the Invention

[0004] The main objective of this invention is to provide a multi-sensor tightly coupled SLAM and safety monitoring method, device, equipment, and storage medium for underground mines. This invention aims to solve the technical problems in the prior art where SLAM schemes are not well adapted to the extreme environment of underground mines, and the data from multiple sensors cannot be effectively coupled and analyzed, resulting in the inability to accurately identify mine risks.

[0005] To achieve the above objectives, this invention provides a multi-sensor tightly coupled SLAM and safety monitoring method for underground mines. The method is applied to a mobile mining vehicle equipped with a multi-sensor system. The mobile mining vehicle is configured to perform safety monitoring of the underground mine during movement. The method includes the following steps:

[0006] Based on the IMU pre-integration results, the acquired multi-sensor data is preprocessed to obtain an initial dataset, which includes corrected point cloud data, denoised IMU data, denoised thermal infrared data, and denoised visible light image data.

[0007] A global point cloud map is generated based on the corrected point cloud data;

[0008] Geometric semantic features, infrared semantic features, and texture semantic features are extracted from the global point cloud map, the denoised thermal infrared data, and the denoised visible light image data, respectively. The geometric semantic features, the infrared semantic features, and the texture semantic features are then fused to generate a multimodal information semantic map.

[0009] Infrared threshold segmentation is performed on the multimodal information semantic map to identify low-temperature areas. The low-temperature areas are then spatially fused with the geometric information of the tunnel walls and roof in the point cloud map of the multimodal information semantic map to determine the seepage areas.

[0010] The point cloud map of the multimodal information semantic map is registered with the mine design model, the point cloud distance field is calculated, and the area with a distance exceeding a preset distance threshold is identified as the deformation area based on the point cloud distance field. The seepage area and the deformation area are then embedded as risk areas into the navigation cost map.

[0011] A global risk index is calculated based on the risk area, and a navigation path for the mine's mobile carrier is dynamically planned based on the global risk index and the navigation cost map. The mine's mobile carrier is configured to move based on the navigation path, and safety monitoring of the underground mine is carried out during the movement.

[0012] Optionally, generating a global point cloud map based on the corrected point cloud data includes:

[0013] The corrected point cloud data is subjected to ground segmentation and non-ground point clustering to output structured feature point cloud clusters;

[0014] The corrected point cloud data is stitched together based on the denoised IMU data to generate a local point cloud map;

[0015] Based on the geometric distribution information of the structured feature point cloud clusters on the cross-section of the alley, a geometric feature descriptor is constructed, and the local point cloud map is optimized based on the geometric feature descriptor to obtain a global point cloud map.

[0016] Optionally, the step of performing ground segmentation and non-ground point clustering on the corrected point cloud data to output structured feature point cloud clusters includes:

[0017] The corrected point cloud data is sorted based on elevation coordinates, and the average elevation is calculated based on the sorting results. The elevation filtering interval is then determined based on the average elevation.

[0018] Points within the height filtering interval are used as the initial ground seed point set, and the three-dimensional centroid and covariance matrix of the initial ground seed point set are calculated.

[0019] Perform singular value decomposition on the covariance matrix, and use the right singular vector corresponding to the smallest singular value obtained by decomposition as the initial ground normal vector;

[0020] Calculate the planar parameters of the initial ground normal vector, and generate a ground planar model based on the planar parameters;

[0021] Calculate the distance between each point in the corrected point cloud data and the ground plane model, and take the points whose distance is less than a preset distance threshold as ground points, and iterate to optimize the ground plane model;

[0022] The non-ground point set is obtained based on the optimized ground plane model, and the non-ground normal vector and local curvature of each non-ground point in the non-ground point set are determined to determine the non-ground seed point. Non-ground point clustering is performed based on the non-ground seed point to output a structured feature point cloud cluster.

[0023] Optionally, the step of stitching together the corrected point cloud data based on the denoised IMU data to generate a local point cloud map includes:

[0024] Based on the IMU pre-integration results of the mine's mobile vehicle, the SLAM system is used to predict the state and obtain prior state estimation results. The prior state estimation results include prior estimates of the navigation state of the mine's mobile vehicle and the corresponding prior covariance matrix of the navigation state prior estimates.

[0025]

[0026]

[0027] in, This represents the prior estimate of the vehicle's navigation state at the next moment. This represents the posterior value of the navigation state of the vehicle at the current moment. This represents the original measurement vector of the carrier IMU at the current moment. This is the state transition function. This represents the process noise vector of the carrier IMU at the current moment. Let be the Jacobian matrix of the state transition function of the IMU. Let be the transpose of the Jacobian matrix of the IMU's state transition function. Let be the state covariance matrix. This represents the prior covariance matrix corresponding to the prior estimate of the carrier's navigation state at the next moment. The process noise covariance matrix;

[0028] The process noise covariance matrix is ​​dynamically adjusted based on the process noise adaptive factor, and the measurement noise covariance is dynamically adjusted based on the matching residual of the corrected point cloud data.

[0029] The Kalman gain coefficient is calculated based on the prior state estimation results, the dynamically adjusted process noise covariance matrix, and the measurement noise covariance. The dynamically adjusted process noise covariance matrix participates in the calculation of the Kalman gain coefficient through the prior covariance matrix. The calculation process of the prior covariance matrix directly depends on the dynamically adjusted process noise covariance matrix. The calculation of the Kalman gain coefficient refers to the following formula:

[0030]

[0031] in, Indicates the Kalman gain coefficient. This represents the measurement noise covariance. The Jacobian matrix represents the carrier laser measurement model. The transpose of the Jacobian matrix representing the carrier laser measurement model;

[0032] A tightly coupled odometry is constructed based on the Kalman gain coefficients, and the state prior estimation results of the SLAM system are iteratively updated using the tightly coupled odometry. The corrected point cloud data is then stitched together based on the state posterior estimation results obtained after the iterative update to generate a local point cloud map.

[0033] Optionally, the step of constructing a geometric feature descriptor based on the geometric distribution information of the structured feature point cloud clusters on the cross-section of the alleyway, and optimizing the local point cloud map based on the geometric feature descriptor to obtain a global point cloud map, includes:

[0034] The distribution of structured feature point cloud clusters on the cross-section of the tunnel is analyzed, geometric distribution information is extracted, and a geometric feature descriptor for the current frame is constructed based on the geometric distribution information. The geometric distribution information includes the maximum height, maximum width, arch height of the cross-section, and the angle between the fitted straight lines of the two side walls.

[0035] Obtain the descriptors of historical keyframes stored in the hash table, and calculate the cosine similarity between the geometric feature descriptor of the current frame and the descriptors of the historical keyframes;

[0036] Feature matching based on cosine similarity is used to determine the historical matching frames that match the current frame in the historical keyframes.

[0037] The transformation matrix between the current frame point cloud and the historical matching frame point cloud is calculated based on the singular value decomposition strategy, and the current frame and the historical matching frame point cloud are aligned based on the transformation matrix.

[0038] Calculate the point cloud overlap between the current frame and the historical matching frames after point cloud alignment, and optimize the local point cloud map based on the point cloud overlap to obtain a global point cloud map.

[0039] Optionally, the step of calculating a global risk index based on the risk area and dynamically planning the navigation path of the mine's movable carrier based on the global risk index and the navigation cost map includes:

[0040] Risk weights for each risk area are determined based on historical mine safety incidents.

[0041] The local risk index of each risk area is calculated based on the area, distribution density, and current distance between each risk area and the mine's mobile carrier.

[0042] The global risk index is calculated based on the risk weights and the local risk indices of each risk region.

[0043] The navigation path of the mine's mobile carrier is dynamically planned based on the local risk index and the navigation cost map.

[0044] The local risk index is updated in real time based on multi-sensor data collected during the movement of the mobile carrier in the mine, and the change of the global risk index is monitored based on the local risk index.

[0045] If the change in the global risk index exceeds a preset threshold, the navigation path of the mine's mobile carrier is dynamically replanned based on the current local risk index of each risk area. The process then returns to the steps of updating the local risk index in real time based on multi-sensor data collected during the movement of the mine's mobile carrier, and monitoring the change in the global risk index based on the local risk index.

[0046] Furthermore, to achieve the above objectives, the present invention also proposes a multi-sensor tightly coupled SLAM and safety monitoring device for underground mines, the multi-sensor tightly coupled SLAM and safety monitoring device for underground mines comprising:

[0047] The data processing module is used to preprocess the acquired multi-sensor data based on the IMU pre-integration results to obtain an initial dataset, which includes corrected point cloud data, denoised IMU data, denoised thermal infrared data, and denoised visible light image data.

[0048] A point cloud map generation module is used to generate a global point cloud map based on the corrected point cloud data.

[0049] The semantic map generation module is used to extract geometric semantic features, infrared semantic features, and texture semantic features from the global point cloud map, the denoised thermal infrared data, and the denoised visible light image data, respectively, and to fuse the geometric semantic features, the infrared semantic features, and the texture semantic features to generate a multimodal information semantic map.

[0050] The water seepage identification module is used to perform infrared threshold segmentation on the multimodal information semantic map, identify low-temperature areas, and spatially fuse the low-temperature areas with the geometric information of the alleyway wall and roof in the point cloud map of the multimodal information semantic map to determine the water seepage area.

[0051] The deformation recognition module is used to register the point cloud map of the multimodal information semantic map with the mine design model, calculate the point cloud distance field, and based on the point cloud distance field, identify areas whose distance exceeds a preset distance threshold as deformation areas, and embed the seepage area and the deformation area as risk areas into the navigation cost map.

[0052] The dynamic safety monitoring module is used to calculate a global risk index based on the risk area, and to dynamically plan the navigation path of the mine's mobile carrier based on the global risk index and the navigation cost map. The mine's mobile carrier is configured to move based on the navigation path, and to monitor the underground mine during the movement.

[0053] Optionally, the dynamic safety monitoring module is further configured to: determine the risk weight of each risk area based on historical mine safety events; calculate the local risk index of each risk area based on its area, distribution density, and current distance to the mine's mobile carrier; calculate the global risk index based on the risk weight and the local risk index of each risk area; dynamically plan the navigation path of the mine's mobile carrier based on the local risk index and the navigation cost map; update the local risk index in real time based on multi-sensor data collected during the movement of the mine's mobile carrier, and monitor the change in the global risk index based on the local risk index; and, in response to the change in the global risk index exceeding a preset threshold, re-plan the navigation path of the mine's mobile carrier based on the current local risk index of each risk area, and return to the steps of updating the local risk index in real time based on multi-sensor data collected during the movement of the mine's mobile carrier, and monitoring the change in the global risk index based on the local risk index.

[0054] Furthermore, to achieve the above objectives, this application also proposes a multi-sensor tightly coupled SLAM and safety monitoring device for underground mines. The device includes: a mobile unit, a multi-sensor system, a memory, a processor, and a multi-sensor tightly coupled SLAM and safety monitoring program for underground mines stored in the memory. The processor is used to run the multi-sensor tightly coupled SLAM and safety monitoring program for underground mines. The computer program is configured to implement the steps of the multi-sensor tightly coupled SLAM and safety monitoring method for underground mines as described above.

[0055] In addition, to achieve the above objectives, this application also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the multi-sensor tightly coupled SLAM and safety monitoring method for underground mines as described above.

[0056] This invention preprocesses multi-sensor data based on IMU pre-integration results to obtain an initial dataset. This initial dataset includes corrected point cloud data, denoised IMU data, denoised thermal infrared data, and denoised visible light image data. A global point cloud map is generated based on the corrected point cloud data. Geometric semantic features, infrared semantic features, and texture semantic features are extracted from the global point cloud map, the denoised thermal infrared data, and the denoised visible light image data, respectively. These features are then fused to generate a multimodal information semantic map. Based on this multimodal information semantic map, seepage areas and deformation areas in underground mines are identified. Through the fusion of risk areas and navigation cost maps and dynamic path planning, safe and autonomous movement of mine mobile carriers is achieved, reducing the cost and safety risks of manual inspections, improving the automation and intelligence level of underground mine safety monitoring, enhancing the early warning capability for seepage and deformation risks, and significantly improving unmanned inspection capabilities in complex underground mine environments. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a schematic diagram of the structure of a multi-sensor tightly coupled SLAM and safety monitoring device for underground mines, which is part of the hardware operating environment of the embodiment of the present invention.

[0059] Figure 2This is a flowchart illustrating the first embodiment of the multi-sensor tightly coupled SLAM and safety monitoring method for underground mines according to the present invention;

[0060] Figure 3 This is a flowchart illustrating the second embodiment of the multi-sensor tightly coupled SLAM and safety monitoring method for underground mines according to the present invention;

[0061] Figure 4 This is a structural block diagram of the first embodiment of the multi-sensor tightly coupled SLAM and safety monitoring device for underground mines according to the present invention.

[0062] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0063] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0064] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a multi-sensor tightly coupled SLAM and safety monitoring device for underground mines, which is part of the hardware operating environment of the embodiment of the present invention.

[0065] like Figure 1 As shown, the multi-sensor tightly coupled SLAM and safety monitoring device for underground mines may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk storage device. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0066] Those skilled in the art will understand that Figure 1The structure shown does not constitute a limitation on multi-sensor tightly coupled SLAM and safety monitoring equipment for underground mines, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0067] like Figure 1 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a multi-sensor tightly coupled SLAM and safety monitoring program for underground mines.

[0068] exist Figure 1 In the multi-sensor tightly coupled SLAM and safety monitoring device for underground mines shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the multi-sensor tightly coupled SLAM and safety monitoring device for underground mines can be set in the device. The multi-sensor tightly coupled SLAM and safety monitoring device for underground mines calls the multi-sensor tightly coupled SLAM and safety monitoring program for underground mines stored in the memory 1005 through the processor 1001, and executes the multi-sensor tightly coupled SLAM and safety monitoring method for underground mines provided in the embodiments of the present invention.

[0069] This invention provides a multi-sensor tightly coupled SLAM and safety monitoring method for underground mines, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the multi-sensor tightly coupled SLAM and safety monitoring method for underground mines according to the present invention.

[0070] In this embodiment, the method is applied to a mobile mining vehicle equipped with a multi-sensor system. The mobile mining vehicle is configured to perform safety monitoring of the underground mine during movement. The multi-sensor tightly coupled SLAM and safety monitoring method for underground mines includes the following steps:

[0071] Step S10: Preprocess the collected multi-sensor data based on the IMU pre-integration results to obtain the initial dataset.

[0072] It should be noted that this embodiment addresses the issue of poor SLAM accuracy and robustness caused by the harsh environment and complex structure of underground mines. By simultaneously acquiring lidar, IMU, and multispectral image data, it first performs adaptive segmentation of the point cloud based on region growing and plane fitting to extract high-quality features. Then, an adaptive iterative Kalman filter is introduced into the tightly coupled state estimation to dynamically adjust the noise covariance and improve stability under intense movement. Simultaneously, it innovatively utilizes the geometric features of the roadway cross-section to construct descriptors for efficient loop closure detection, effectively correcting accumulated errors. Finally, the generated high-precision point cloud map is fused with thermal infrared and visible light images to achieve automatic identification and graded early warning of safety risks such as water seepage and deformation, forming a closed loop from precise positioning to safety decision-making. This significantly improves positioning accuracy and system reliability in complex mine environments, providing key technical support for unmanned inspection and safe production.

[0073] It should be understood that the executing entity of this embodiment can be a mobile mining vehicle with data processing, network communication, and program execution functions, such as a mining robot, unmanned vehicle, etc., or a terminal electronic device capable of realizing the above functions. The following uses a multi-sensor tightly coupled SLAM and safety monitoring device (hereinafter referred to as the monitoring device) for underground mines as an example to describe this embodiment and the following embodiments.

[0074] It should be noted that the initial dataset includes corrected point cloud data, denoised IMU data, denoised thermal infrared data, and denoised visible light image data.

[0075] It should be noted that IMU pre-integration refers to the process of calculating the relative motion state (attitude, position, velocity) of the carrier by integrating the angular velocity and acceleration data output by the inertial measurement unit (IMU) within a preset time interval during the operation of the inertial measurement unit (IMU). The core purpose is to avoid re-integrating the original IMU data for each optimization and improve data processing efficiency.

[0076] Multi-sensor data refers to the raw data collected by four types of sensors—LiDAR, IMU, thermal infrared camera, and visible light camera—on a mobile carrier in a mining setting. It covers spatial geometric information, inertial motion information, temperature radiation information, and visual texture information.

[0077] It should be noted that the corrected point cloud data refers to the point cloud data obtained after motion distortion correction of the original point cloud data. This eliminates the point cloud position offset problem caused by the time difference of lidar scanning during carrier movement and improves the spatial position accuracy of the point cloud.

[0078] Denoising-reduced IMU data refers to the data obtained after the original IMU data has been processed by a filtering algorithm. This process removes invalid interference signals caused by environmental vibrations and sensor errors, while retaining the carrier's true inertial motion information (angular velocity, acceleration).

[0079] Denoising-reduced thermal infrared data refers to the data obtained after the original thermal imaging data collected by the thermal infrared camera has been denoised. This process eliminates noise points caused by interference from high-temperature dust, equipment radiation, etc., and clearly preserves the temperature distribution characteristics of different objects in the scene.

[0080] Denoising-reduced visible light image data refers to the data obtained after denoising processing of the original image captured by the visible light camera. This process eliminates image noise caused by sudden changes in illumination, dust scattering, etc., and improves the image's texture clarity and detail recognition.

[0081] In practice, the monitoring equipment collects raw data through multiple sensors deployed on mobile carriers in the mine (such as inspection robots and driverless mining trucks). The sensors involved include lidar, IMU (inertial measurement unit), thermal infrared camera and visible light camera.

[0082] Understandably, multi-source data synchronous acquisition and preprocessing involves simultaneously acquiring lidar point clouds, IMU data, visible light images, and thermal infrared images through a sensor system mounted on a mobile platform. Hardware triggering and software timestamp alignment ensure data synchronization, and IMU pre-integration results are used to correct motion distortion in the lidar point cloud. This embodiment effectively eliminates noise interference and distortion issues in the raw multi-sensor data by preprocessing the multi-sensor data, improving data accuracy and reliability, avoiding interference from invalid data in subsequent processing flows, and achieving time synchronization and attitude uniformity of multi-sensor data through IMU pre-integration, thus solving the problem of inconsistent acquisition frequencies and references among different sensors.

[0083] In some embodiments, the monitoring device uses the raw angular velocity and acceleration data output by the IMU and a quaternion-based attitude integration algorithm to calculate the relative attitude (attitude angle, position offset, velocity change) of the carrier between two adjacent lidar scanning moments without relying on external positioning information, and generates IMU pre-integration results. Next, using the IMU pre-integration results as the time and attitude references, unified preprocessing was performed on the multi-sensor data: For the raw point cloud data acquired by the lidar, the relative attitude information obtained from the IMU pre-integration was used to eliminate point cloud distortion caused by the carrier's motion through a motion distortion correction algorithm, resulting in corrected point cloud data; for the raw IMU data, a Kalman filter algorithm was used to filter out interference from environmental vibration and sensor noise, resulting in denoised IMU data; for the raw thermal infrared data acquired by the thermal infrared camera, a median filter algorithm was used to remove noise points caused by high-temperature dust and equipment radiation interference, while retaining effective temperature distribution information, resulting in denoised thermal infrared data; for the raw visible light image data acquired by the visible light camera, a Gaussian filter algorithm was used to smooth image noise caused by sudden changes in illumination and dust scattering, improving image clarity, resulting in denoised visible light image data. Finally, the processed data were integrated to form an initial dataset containing four types of data.

[0084] Step S20: Generate a global point cloud map based on the corrected point cloud data.

[0085] It should be noted that a global point cloud map refers to a complete 3D point cloud map covering the target work area, containing the 3D spatial geometric information of all objects in the area, and possessing global consistency and high spatial accuracy.

[0086] In some embodiments, the monitoring equipment employs a "local mapping + global optimization" strategy to generate a global point cloud map: First, based on the corrected point cloud data obtained in step one, point cloud segments are divided according to time series. The NDT (Normal Distribution Transform) registration algorithm is used to calculate the pose transformation relationship between adjacent point cloud segments, and adjacent point cloud segments are merged into a local point cloud map to achieve the stitching of local spatial information. Second, during the local mapping process, denoised IMU data is introduced, and the real-time pose information of the IMU data is used to assist point cloud registration, reducing the number of iterations in the registration process and improving the efficiency and accuracy of local registration. Then, the pose information of all local point cloud maps is collected to construct a global pose map, and the global pose map is optimized to eliminate pose errors accumulated during the local mapping process (such as drift problems). Finally, all optimized local point cloud maps are globally stitched together to generate a global point cloud map covering the mining operation area. This map contains three-dimensional geometric information of scenes such as mine roadways, equipment, and walls.

[0087] Step S30: Extract geometric semantic features, infrared semantic features, and texture semantic features from the global point cloud map, the denoised thermal infrared data, and the denoised visible light image data, respectively, and fuse the geometric semantic features, the infrared semantic features, and the texture semantic features to generate a multimodal information semantic map.

[0088] It should be noted that geometric semantic features refer to the feature information extracted from point cloud maps, which includes the geometric attributes (shape, location, size, topological relationship) and semantic categories (walls, equipment, ground, etc.) of the target space. They are the core features that describe the characteristics of the target space.

[0089] Infrared semantic features refer to the feature information extracted from thermal infrared data, which includes target temperature attributes (temperature value, temperature distribution, temperature gradient) and semantic categories (high temperature region, temperature anomaly region, etc.), and is used to characterize the thermal radiation characteristics of the target.

[0090] Texture semantic features refer to the feature information extracted from visible light image data, which includes the texture attributes (grayscale distribution, texture direction, texture density, texture roughness) and semantic categories (cracks, water seepage marks, etc.) of the target surface, and are used to describe the surface state of the target.

[0091] It should be noted that multimodal information semantic map refers to a semantic map that integrates geometric semantic features, infrared semantic features, and texture semantic features. It not only includes the three-dimensional spatial geometric information of the scene, but also the thermal radiation information and surface texture information of the target, and has multi-dimensional semantic description capabilities.

[0092] In practical implementation, the monitoring equipment extracts geometric semantic features and identifies different types of targets such as tunnel walls, ground, equipment, and pillars based on semantic segmentation of the global point cloud map. Then, by calculating descriptors for each target, it extracts geometric semantic features such as shape, spatial location, and size. Based on denoised thermal infrared data, a target detection algorithm is used to identify high-temperature areas and temperature anomaly areas in the thermal infrared image. Combining the temperature value information from the thermal infrared data, it extracts infrared semantic features such as temperature distribution and temperature gradient of the target, focusing on capturing abnormal temperature signals related to water seepage. Based on denoised visible light image data, the SIFT (Scale Invariant Feature Transform) algorithm is used to extract texture feature points in the image and calculate the grayscale values ​​of the feature points. Information such as texture direction and texture density is used to form texture semantic features, which are used to help identify the surface state of objects in the scene (such as wall cracks and water seepage marks). A feature-level fusion strategy is adopted to fuse geometric semantic features, infrared semantic features, and texture semantic features through a weighted average algorithm to generate a multimodal information semantic map containing spatial geometric information, temperature radiation information, and visual texture information. This enables the effective extraction of multimodal features, fully explores the core information (geometry, temperature, and texture) of different sensor data, and makes up for the one-sidedness of single-modal feature information. Through the feature-level fusion strategy, the complementary advantages of multimodal features are integrated, which improves the representation ability and robustness of semantic features and enhances the adaptability to the complex environment of the mine.

[0093] Step S40: Perform infrared threshold segmentation on the multimodal information semantic map to identify low-temperature areas, and spatially fuse the low-temperature areas with the geometric information of the alleyway wall and roof in the point cloud map of the multimodal information semantic map to determine the seepage area;

[0094] Step S50: Register the point cloud map of the multimodal information semantic map with the mine design model, calculate the point cloud distance field, and based on the point cloud distance field, designate areas with distances exceeding a preset distance threshold as deformation areas, and embed the seepage area and the deformation area as risk areas into the navigation cost map.

[0095] It should be noted that a risk area refers to an area within the mining operation area that has safety hazards and may threaten the operational safety of the carrier or the safety of personnel. In this embodiment, it may include water seepage areas and deformation areas.

[0096] Water seepage areas refer to damp or waterlogged areas in mine tunnels, such as walls and floors, caused by groundwater leakage or pipe damage. These areas can easily lead to slippery roads and softened walls, posing safety hazards.

[0097] Deformation zones refer to areas in mine roadways where the walls, pillars, roofs, and other structures have deformed (such as depressions, tilting, or cracks) due to geological pressure, mining effects, or other reasons. These areas are prone to safety accidents such as collapses.

[0098] In practical implementation, the monitoring equipment can combine the high-precision point cloud map M generated by SLAM with thermal infrared images. and visible light images Registration and fusion are performed to generate a semantic map with multimodal information. .

[0099] Water seepage identification is performed based on this map, utilizing thermal infrared imagery and threshold segmentation. Identify low-temperature areas and fuse them with geometric information from the tunnel walls and roof in the point cloud to accurately locate seepage areas. .

[0100] Deformation recognition is performed on the real-time point cloud map. With design model By comparison, the point cloud distance field is calculated using an iterative nearest-point algorithm, and points exceeding a threshold are extracted. Deformation region .

[0101] Step S60: Calculate the global risk index based on the risk area, and dynamically plan the navigation path of the mine mobile carrier based on the global risk index and the navigation cost map. The mine mobile carrier is configured to move based on the navigation path, and the underground mine is monitored for safety during the movement.

[0102] It should be noted that the navigation cost map refers to a two-dimensional or three-dimensional map used for path planning of mobile carriers in mines. By assigning different cost weights to different areas in the map (such as high cost for obstacle areas and low cost for flat roads), the path planning algorithm is assisted in selecting the optimal path.

[0103] The global risk index is a quantitative indicator calculated based on the area and risk level of all risk areas within a mining operation area. It is used to assess the overall safety status of the mining operation area, and the higher the value, the higher the overall safety risk.

[0104] In practice, the monitoring equipment embeds the identified risk areas into the navigation cost map to achieve avoidance, calculates the global risk index based on the risk areas, and triggers graded early warnings based on the global risk index.

[0105] Furthermore, in order to improve the efficiency of movement during the security monitoring process and ensure the safety of the carrier, step S60 above may include:

[0106] Step S601: Determine the risk weight of each risk area based on historical mine safety incidents.

[0107] It should be noted that historical mine safety incidents refer to records of past safety accidents related to water seepage and deformation in the mine, including information such as accident type, cause, scope of impact, degree of loss, and frequency of occurrence.

[0108] The aforementioned risk weights refer to weighting coefficients determined based on the probability and severity of mine safety accidents caused by different risk areas (seepage areas, deformation areas), and are used to quantify the impact of different types of risk areas on the overall safety status.

[0109] It is understandable that this embodiment uses historical safety event data to drive the setting of risk weights, thus eliminating the drawbacks of subjective experience-based assignment and making the weight values ​​more consistent with the actual risk evolution patterns and hazard levels of the mine.

[0110] Step S602: Calculate the local risk index of each risk area based on its area, distribution density, and current distance from the mine's movable carrier.

[0111] It should be noted that the local risk index can be a quantitative indicator calculated for a single risk area, taking into account its own geometric attributes (area), spatial distribution characteristics (distribution density), and relative position with the carrier (current distance), to characterize the independent hazard level of a single risk area.

[0112] In practice, the area and distribution density of a region can be extracted from the constructed multimodal information semantic map. The area is calculated by the outline of the risk region after semantic segmentation, and the distribution density is obtained by counting the number of similar risk regions within a unit cubic meter of space.

[0113] Step S603: Calculate the global risk index based on the risk weights and the local risk indices of each risk region.

[0114] In practice, the monitoring equipment calculates the global risk index by weighted summation based on the risk weights and the local risk indices of each risk area.

[0115] Step S604: Dynamically plan the navigation path of the mine's movable carrier based on the local risk index and the navigation cost map.

[0116] In the specific implementation, firstly, a mapping relationship between risk index and cost weight is established: the corresponding navigation cost weight is set according to the local risk index level (e.g., local risk index <2 corresponds to cost 5, 2-4 corresponds to cost 15, ≥4 corresponds to cost 30). The higher the risk index, the greater the corresponding grid cost weight. Secondly, the navigation cost map is updated: the local risk index of each risk area is mapped to the corresponding grid on the map, the grid cost weight is updated, and the safe area maintains the default low cost.

[0117] In practical implementation, dynamic programming may include: using an efficient dynamic path planning algorithm, setting "lowest total path cost" as the core objective (adjusting the weights of efficiency and safety based on risk level, such as a safety weight of 0.6 and an efficiency weight of 0.4 under medium risk level); finally, outputting the optimal path: the algorithm calculates the optimal path to avoid high-cost grids and sends the path coordinate information to the carrier motion control module.

[0118] Step S605: Update the local risk index in real time based on the multi-sensor data collected during the movement of the mine's mobile carrier, and monitor the change in the global risk index based on the local risk index.

[0119] It should be noted that the preset amplitude threshold can be the critical value of the allowable range of change of the global risk index based on the evolution law of mine risk. When the amplitude of the index change exceeds this value, it indicates that the environmental safety status has changed significantly, and path replanning needs to be triggered to replan the path and thus avoid high-risk areas.

[0120] In its implementation, the carrier collects multi-sensor data at a preset frequency to ensure data timeliness; it monitors changes in the seepage area using thermal infrared data and changes in the deformed area's contour using lidar point cloud data, dynamically correcting parameters such as area and distribution density, and recalculating the local risk index of the affected risk area; it recalculates the global risk index based on the updated local risk index, calculating the absolute difference (variance) between the current index and the previous index; it compares the variation amplitude with a preset amplitude threshold, and if it does not exceed the threshold, it continues monitoring; if it exceeds the threshold, it triggers a path replanning signal.

[0121] Step S606: In response to the change in the global risk index exceeding a preset threshold, the navigation path of the mine's mobile carrier is dynamically planned again based on the current local risk index of each risk area. The process then returns to the step of updating the local risk index in real time based on the multi-sensor data collected during the movement of the mine's mobile carrier, and monitoring the change in the global risk index based on the local risk index.

[0122] In some embodiments, the monitoring equipment uses the existing navigation cost map of the mine as a basis. Based on the risk area type and risk level (e.g., water seepage areas are classified as medium risk, and severely deformed areas as high risk), it assigns corresponding cost weights to different risk areas (the higher the risk level, the greater the cost weight). The spatial extent and cost weight of the risk areas are embedded into the navigation cost map to form an enhanced navigation cost map containing risk information. Risk areas are marked as high-cost areas, and normal areas as low-cost areas. Based on the enhanced navigation cost map, a weighted summation algorithm is used to calculate the global risk index, achieving a quantitative assessment of the overall safety status of the mine's operating area. Finally, based on the enhanced navigation cost map and the global risk index, an A... The path planning algorithm optimizes the preset navigation path of the mobile vehicle in the mine: if the preset path passes through a high-cost risk area, the path planning strategy is automatically adjusted to avoid the risk area; at the same time, the global risk index changes during the operation of the vehicle are monitored in real time. If the global risk index exceeds the preset threshold, a safety warning is immediately triggered, prompting the vehicle to stop operation or adjust the driving route, so as to realize full-process safety monitoring of the navigation path and realize risk-enhanced navigation and closed-loop warning.

[0123] This embodiment significantly improves the accuracy and robustness of SLAM mapping in mines by processing and coupling multi-sensor data. IMU pre-integration assists in multi-sensor data preprocessing and point cloud registration optimization, effectively solving the problems of mapping drift and insufficient accuracy in complex mine environments (low light, high dust, no GNSS signal). It achieves deep fusion and efficient mining of multi-modal information, fully leveraging the complementary advantages of lidar, IMU, thermal infrared cameras, and visible light cameras to improve the accuracy of scene semantic understanding and risk identification. It promotes the transformation of mine safety monitoring from manual inspection to automation and intelligence, replacing traditional manual risk investigation methods, significantly improving risk identification efficiency and reducing manual operating costs. It achieves synergistic linkage between navigation paths and safety monitoring, providing safe and efficient navigation solutions for mobile vehicles in mines through risk area embedding and global risk index assessment, effectively reducing the incidence of safety accidents. It enhances the adaptability to complex mine environments; the solution is designed specifically for the characteristics of mines such as low light, high dust, and complex spatial structures, possessing strong environmental adaptability.

[0124] refer to Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the multi-sensor tightly coupled SLAM and safety monitoring method for underground mines according to the present invention.

[0125] Based on the first embodiment described above, in this embodiment, step S20 further includes:

[0126] Step S201: Perform ground segmentation and non-ground point clustering on the corrected point cloud data to output structured feature point cloud clusters.

[0127] It should be noted that ground segmentation can be the process of separating ground point clouds from non-ground point clouds from corrected point cloud data, used to remove ground interference and focus on structured target point clouds within the roadway. Non-ground point clustering can refer to the process of performing density clustering on the segmented non-ground point clouds, removing isolated noise points and classifying them into the same type of structured targets, used to generate structured feature point cloud clusters.

[0128] Structured feature point cloud clusters refer to a collection of non-ground point clouds that contain the same type of structured targets (such as walls, columns, and equipment) after clustering. They have clear geometric structural features and are used to construct geometric feature descriptors for mine tunnel scenes.

[0129] Furthermore, in order to accurately extract structured feature point cloud clusters, step S201 above may include:

[0130] Step S2011: Sort the corrected point cloud data based on the height coordinates, calculate the average height based on the sorting results, and determine the height filtering interval based on the average height.

[0131] It should be noted that the height filtering range refers to the height range determined by combining the average height of the corrected point cloud data with a preset offset, which is used to initially filter out initial seed points that may belong to the ground.

[0132] Step S2012: Use the points within the height filtering interval as the initial ground seed point set, and calculate the three-dimensional centroid and covariance matrix of the initial ground seed point set.

[0133] It should be noted that the initial ground seed point set refers to the set of point clouds whose height coordinates are within the height selection range, and it is the basic data for constructing the initial ground plane model.

[0134] The three-dimensional centroid refers to the center position of the initial ground seed point set in three-dimensional space. It is calculated from the average of the X, Y, and Z coordinates of each point and is used to assist in constructing the ground planar model.

[0135] The covariance matrix is ​​a 3×3 matrix that reflects the degree of dispersion of the initial ground seed point set in three-dimensional space. It is used to obtain the ground normal vector through singular value decomposition.

[0136] Step S2013: Perform singular value decomposition on the covariance matrix, and use the right singular vector corresponding to the smallest singular value obtained by decomposition as the initial ground normal vector.

[0137] It should be noted that Singular Value Decomposition (SVD) refers to an algorithm that decomposes the covariance matrix to obtain singular values ​​and right singular vectors, which are used to determine the direction of the initial ground normal vector.

[0138] The initial ground normal vector is the vector perpendicular to the initial ground plane obtained through singular value decomposition. It is determined by the right singular vector corresponding to the minimum singular value of the covariance matrix and is used to generate the initial ground plane model.

[0139] Step S2014: Calculate the planar parameters of the initial ground normal vector, and generate a ground planar model based on the planar parameters.

[0140] It should be noted that the ground plane model can be a mathematical model (generally ax+by+cz+d=0) constructed based on the ground normal vector and the three-dimensional centroid to describe the spatial location of the ground, and is used to distinguish between ground points and non-ground points.

[0141] Step S2015: Calculate the distance between each point in the corrected point cloud data and the ground plane model, and take the points with a distance less than a preset distance threshold as ground points, and iterate to optimize the ground plane model.

[0142] It should be noted that the preset distance threshold can be a pre-set distance standard used to determine whether a point is a ground point. Points that are less than the ground plane model at a distance of less than the threshold are determined to be ground points, which is the core criterion for ground segmentation.

[0143] Step S2016: Obtain a set of non-ground points based on the optimized ground plane model, determine the non-ground normal vector and local curvature of each non-ground point in the set to determine non-ground seed points, and perform non-ground point clustering based on the non-ground seed points to output a structured feature point cloud cluster.

[0144] It should be noted that the non-ground point set can be a collection of non-ground points selected from the corrected point cloud data based on the optimized ground plane model, including structured targets such as walls, pillars, and equipment within the tunnel. The non-ground normal vector can be a vector perpendicular to the surface where the non-ground point is located, obtained by fitting a plane to the neighboring points of the non-ground point, and is used to select non-ground seed points.

[0145] Local curvature refers to a parameter reflecting the distance deviation between a non-ground point and its neighboring points. It is used to characterize the surface smoothness of the region where the point is located and to assist in the selection of non-ground seed points. Non-ground seed points are points selected from the set of non-ground points that have the same normal vector direction and small local curvature, and serve as the core points for DBSCAN clustering.

[0146] In the specific implementation, adaptive point cloud segmentation and feature extraction employ a hierarchical segmentation method combining region growing and plane fitting. This method is used for point clouds after motion distortion correction. Sort by their height coordinates in ascending order and select the one with the lowest height. Calculate the average height at each point. The height is in the range Points within the area are used as the initial ground seed point set. ,in The preset height threshold;

[0147] For the ground seed point set Calculate its three-dimensional centroid μ and covariance matrix C:

[0148]

[0149]

[0150] Singular value decomposition (SVD) is performed on the covariance matrix C to obtain... Take the right singular vector corresponding to the smallest singular value. As the initial ground normal vector And calculate the plane parameters. Thus, a ground planar model is obtained. ;

[0151] Based on the ground plane model, calculate the distance from all points in the point cloud P to the plane. Distance less than the threshold The points are classified as ground points. Iterate through the process to optimize the ground planar model;

[0152] For non-ground point sets Estimate the normal vector of each point and local curvature Subsequently, region growing begins with the point of minimum curvature as the seed point, ensuring that the angle between its normal vectors is less than a threshold. And the curvature difference is less than the threshold. The neighboring points are grouped into the same cluster, and the final output is a series of structured feature point cloud clusters. .

[0153] Step S202: Based on the denoised IMU data, the corrected point cloud data is stitched together to generate a local point cloud map.

[0154] It should be noted that a local point cloud map can be a small-scale, localized 3D point cloud map formed by stitching together corrected point cloud data collected during the short-distance movement of the carrier and pose information obtained by IMU pre-integration, which reflects the current environmental information of the carrier's surroundings.

[0155] In practical implementation, the monitoring equipment can perform ground segmentation and non-ground point clustering on the corrected point cloud through adaptive point cloud segmentation and feature extraction. It adopts a hierarchical segmentation method that combines region growing and plane fitting to output structured feature point cloud clusters. It introduces a tightly coupled odometer with adaptive iterative Kalman filtering, introduces a process noise adaptive factor to dynamically adjust the process noise covariance in state prediction, and adaptively adjusts the measurement noise covariance according to the point cloud matching residual to perform iterative state updates.

[0156] Furthermore, in order to effectively optimize the state estimation results of the SLAM system and effectively reduce the spatial offset error of point clouds in adjacent frames, step S202 above may include:

[0157] Step S2021: Based on the IMU pre-integration results of the mine mobile vehicle, perform state prediction on the SLAM system to obtain the state prior estimation results. The state prior estimation results include the prior estimation value of the navigation state of the mine mobile vehicle and the prior covariance matrix corresponding to the navigation state prior estimation value.

[0158] Step S2022: Dynamically adjust the process noise covariance matrix based on the process noise adaptive factor, and dynamically adjust the measurement noise covariance according to the matching residual of the corrected point cloud data;

[0159] Step S2023: Calculate the Kalman gain coefficient based on the aforementioned state prior estimation results and the dynamically adjusted process noise covariance matrix and measurement noise covariance.

[0160]

[0161] in, Indicates the Kalman gain coefficient. This represents the measurement noise covariance. The Jacobian matrix represents the carrier laser measurement model. The transpose of the Jacobian matrix representing the carrier laser measurement model;

[0162] Step S2024: Construct a tightly coupled odometry based on the Kalman gain coefficient, iteratively update the state prior estimation results of the SLAM system through the tightly coupled odometry, and stitch the corrected point cloud data based on the state posterior estimation results obtained after iterative update to generate a local point cloud map.

[0163] It is understandable that this embodiment introduces a tightly coupled odometry system using adaptive iterative Kalman filtering. Based on the tightly coupled framework of FAST-LIO2, an adaptive iterative Kalman filtering algorithm is proposed, specifically including:

[0164] Define the system state vector And perform state prediction based on IMU pre-integration results:

[0165]

[0166]

[0167] in, This represents the prior estimate of the vehicle's navigation state at the next moment. This represents the posterior value of the navigation state of the vehicle at the current moment. This represents the original measurement vector of the carrier IMU at the current moment. This is the state transition function. This represents the process noise vector of the carrier IMU at the current moment. Let be the Jacobian matrix of the state transition function of the IMU. Let be the transpose of the Jacobian matrix of the IMU's state transition function. Let be the state covariance matrix. This represents the prior covariance matrix corresponding to the prior estimate of the carrier's navigation state at the next moment. The process noise covariance matrix;

[0168] Introducing a process noise adaptive factor γ to dynamically adjust the process noise covariance:

[0169]

[0170]

[0171] in, The initial process noise covariance, The angular velocity and rate of change of acceleration of the IMU are given. This is the adjustment coefficient;

[0172] Based on the residual of the current laser point cloud matching Adaptive adjustment of measurement noise covariance :

[0173]

[0174] in, For the initial measurement noise covariance, This is the adjustment coefficient;

[0175] Perform iterative measurement updates: calculate Kalman gain And utilize feature point cloud clusters The residual between the system and the global map is used to iteratively update the system state. Covariance The Kalman gain coefficient is calculated as follows, until convergence:

[0176]

[0177] in, Indicates the Kalman gain coefficient. This represents the measurement noise covariance. The Jacobian matrix represents the carrier laser measurement model. The transpose of the Jacobian matrix of the carrier laser measurement model.

[0178] Step S203: Construct a geometric feature descriptor based on the geometric distribution information of the structured feature point cloud cluster on the cross-section of the alley, and optimize the local point cloud map based on the geometric feature descriptor to obtain a global point cloud map.

[0179] It should be noted that geometric feature descriptors can be vectors or parameters constructed based on the geometric distribution (such as contour shape, size, and location coordinates) of structured feature point cloud clusters on the cross-section of the alleyway, which can be used for matching and alignment between different local maps.

[0180] Furthermore, in order to address the accumulated errors in long tunnels, step S203 above may include:

[0181] Step S2031: Analyze the distribution of structured feature point cloud clusters on the cross-section of the roadway, extract geometric distribution information, and construct the geometric feature descriptor of the current frame based on the geometric distribution information.

[0182] It should be noted that the geometric distribution information includes the maximum height, maximum width, arch height of the cross section, and the angle between the fitted straight lines of the two side walls. The geometric feature descriptor can be a vector or parameter constructed based on the geometric distribution (such as contour shape, size, and location coordinates) of the structured feature point cloud clusters on the cross section of the tunnel, which can be used for matching and alignment between different local maps.

[0183] Step S2032: Obtain the descriptors of historical keyframes stored in the hash table, and calculate the cosine similarity between the geometric feature descriptor of the current frame and the descriptor of the historical keyframes.

[0184] It should be noted that historical keyframes can be representative frames selected from point cloud frames that have been acquired and processed. Frames with significant pose changes and rich feature information are usually selected for matching and alignment with the current frame, reducing redundant calculations and ensuring matching reliability.

[0185] It should be noted that the descriptors of historical keyframes stored in the hash table can be the geometric feature descriptors corresponding to the historical keyframes stored in the hash table data structure. The key of the hash table is the keyframe ID, and the value is the geometric feature descriptor of the keyframe, which can realize fast retrieval and matching of descriptors.

[0186] Step S2033: Perform feature matching based on cosine similarity to determine the historical matching frames that match the current frame in the historical keyframes.

[0187] It should be noted that cosine similarity can be used as an indicator to measure the degree of similarity between two geometric feature descriptors (in vector form).

[0188] Step S2034: Calculate the transformation matrix between the current frame point cloud and the historical matching frame point cloud based on the singular value decomposition strategy, and align the current frame and the historical matching frame point clouds based on the transformation matrix.

[0189] It should be noted that the Singular Value Decomposition (SVD) strategy can be used to decompose the correspondence matrix between two point cloud sets into rotation matrix, scaling matrix and translation matrix. In point cloud registration, it is mainly used to solve the transformation matrix (including rotation and translation information) between the current frame and the historical matching frames.

[0190] It should be noted that the transformation matrix is ​​a matrix used to describe the position and orientation transformation relationship of the current frame point cloud relative to the historical matching frame point cloud in three-dimensional space. It includes two parts: rotation matrix and translation vector, which can realize the spatial alignment of two point cloud sets.

[0191] Step S2035: Calculate the point cloud overlap between the current frame and the historical matching frames after point cloud alignment, and optimize the local point cloud map based on the point cloud overlap to obtain a global point cloud map.

[0192] It should be noted that point cloud overlap can be used as an indicator to measure the size of the overlapping area between two point clouds in space. It is obtained by statistically analyzing the proportion of the number of matching point pairs whose distance is within a set threshold after the two point clouds are transformed and aligned, to the total number of point clouds. It is used to evaluate the point cloud alignment effect and map optimization quality.

[0193] Understandably, this embodiment designs a descriptor based on the geometric features of the tunnel cross-section to address the cumulative errors in long tunnels. This descriptor analyzes point cloud clusters. The distribution of features on the cross-section of the tunnel is analyzed, and geometric parameters such as height, width, and arch curvature are extracted to form a feature vector. A hash table is used to store descriptors of historical keyframes. The current frame is quickly matched by calculating the cosine similarity between descriptors. When the similarity exceeds the threshold and passes geometric verification, it is determined to be a valid loop closure. Constraints are added to the backend optimization to significantly correct trajectory drift.

[0194] This embodiment achieves structured processing of point cloud data through ground segmentation and non-ground point clustering, removing ground interference and highlighting key targets and structures in the mining environment, providing clear processing objects for subsequent feature descriptor construction and map optimization; local point cloud map stitching combined with IMU pre-integration results achieves accurate environmental modeling within a short distance range, ensuring the accuracy of the local map; global optimization effectively eliminates the accumulated errors in the point cloud stitching process, improving the consistency and accuracy of the global point cloud map.

[0195] Furthermore, this invention also proposes a computer-readable storage medium storing a multi-sensor tightly coupled SLAM and safety monitoring program for underground mines. When executed by a processor, the multi-sensor tightly coupled SLAM and safety monitoring program for underground mines implements the steps of the multi-sensor tightly coupled SLAM and safety monitoring method for underground mines as described above.

[0196] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0197] The aforementioned computer-readable storage medium may be included in a multi-sensor tightly coupled SLAM and safety monitoring device for underground mines; or it may exist independently and not be assembled into a multi-sensor tightly coupled SLAM and safety monitoring device for underground mines.

[0198] Furthermore, this invention also proposes a computer program product, including a multi-sensor tightly coupled SLAM and safety monitoring program for underground mines. When the multi-sensor tightly coupled SLAM and safety monitoring program for underground mines is executed by a processor, it implements the steps of the multi-sensor tightly coupled SLAM and safety monitoring method for underground mines as described above.

[0199] The specific implementation of the computer program product of the present invention is basically the same as the embodiments of the multi-sensor tightly coupled SLAM and safety monitoring method for underground mines described above, and will not be repeated here.

[0200] Reference Figure 4 , Figure 4 This is a structural block diagram of the first embodiment of the multi-sensor tightly coupled SLAM and safety monitoring device for underground mines according to the present invention.

[0201] like Figure 4 As shown in the embodiments of the present invention, the multi-sensor tightly coupled SLAM and safety monitoring device for underground mines includes:

[0202] Data processing module 10 is used to preprocess the acquired multi-sensor data based on the IMU pre-integration results to obtain an initial dataset, which includes corrected point cloud data, denoised IMU data, denoised thermal infrared data, and denoised visible light image data.

[0203] Point cloud map generation module 20 is used to generate a global point cloud map based on the corrected point cloud data;

[0204] The semantic map generation module 30 is used to extract geometric semantic features, infrared semantic features and texture semantic features from the global point cloud map, the denoised thermal infrared data and the denoised visible light image data respectively, and to fuse the geometric semantic features, the infrared semantic features and the texture semantic features to generate a multimodal information semantic map.

[0205] The water seepage identification module 40 is used to perform infrared threshold segmentation on the multimodal information semantic map, identify low-temperature areas, and spatially fuse the low-temperature areas with the geometric information of the alleyway wall and roof in the point cloud map of the multimodal information semantic map to determine the water seepage area.

[0206] The deformation recognition module 50 is used to register the point cloud map of the multimodal information semantic map with the mine design model, calculate the point cloud distance field, and based on the point cloud distance field, identify areas whose distance exceeds a preset distance threshold as deformation areas, and embed the seepage area and the deformation area as risk areas into the navigation cost map.

[0207] The dynamic safety monitoring module 60 is used to calculate a global risk index based on the risk area, and to dynamically plan the navigation path of the mine mobile carrier based on the global risk index and the navigation cost map. The mine mobile carrier is configured to move based on the navigation path and to monitor the underground mine during the movement.

[0208] Furthermore, the dynamic safety monitoring module 60 is also used to determine the risk weight of each risk area based on historical mine safety events; calculate the local risk index of each risk area based on its area, distribution density, and current distance to the mine's mobile carrier; calculate the global risk index based on the risk weight and the local risk index of each risk area; dynamically plan the navigation path of the mine's mobile carrier based on the local risk index and the navigation cost map; update the local risk index in real time based on multi-sensor data collected during the movement of the mine's mobile carrier, and monitor the change in the global risk index based on the local risk index; in response to the change in the global risk index exceeding a preset threshold, dynamically plan the navigation path of the mine's mobile carrier again based on the current local risk index of each risk area, and return to the steps of updating the local risk index in real time based on multi-sensor data collected during the movement of the mine's mobile carrier, and monitoring the change in the global risk index based on the local risk index.

[0209] This embodiment significantly improves the accuracy and robustness of SLAM mapping in mines by processing and coupling multi-sensor data. IMU pre-integration assists in multi-sensor data preprocessing and point cloud registration optimization, effectively solving the problems of mapping drift and insufficient accuracy in complex mine environments (low light, high dust, no GNSS signal). It achieves deep fusion and efficient mining of multi-modal information, fully leveraging the complementary advantages of lidar, IMU, thermal infrared cameras, and visible light cameras to improve the accuracy of scene semantic understanding and risk identification. It promotes the transformation of mine safety monitoring from manual inspection to automation and intelligence, replacing traditional manual risk investigation methods, significantly improving risk identification efficiency and reducing manual operating costs. It achieves synergistic linkage between navigation paths and safety monitoring, providing safe and efficient navigation solutions for mobile vehicles in mines through risk area embedding and global risk index assessment, effectively reducing the incidence of safety accidents. It enhances the adaptability to complex mine environments; the solution is designed specifically for the characteristics of mines such as low light, high dust, and complex spatial structures, possessing strong environmental adaptability.

[0210] The multi-sensor tightly coupled SLAM and safety monitoring device for underground mines provided in this application adopts the multi-sensor tightly coupled SLAM and safety monitoring method for underground mines in the above embodiments, and can solve the technical problems of multi-sensor tightly coupled SLAM and safety monitoring in underground mines. Compared with the prior art, the beneficial effects of the multi-sensor tightly coupled SLAM and safety monitoring device for underground mines provided in this application are the same as the beneficial effects of the multi-sensor tightly coupled SLAM and safety monitoring method for underground mines provided in the above embodiments, and other technical features in the multi-sensor tightly coupled SLAM and safety monitoring device for underground mines are the same as the features disclosed in the methods of the above embodiments, and will not be repeated here.

[0211] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0212] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0213] In addition, for technical details not described in detail in this embodiment, please refer to the multi-sensor tightly coupled SLAM and safety monitoring method for underground mines provided in any embodiment of the present invention, which will not be repeated here.

[0214] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0215] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0216] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0217] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A multi-sensor tightly coupled SLAM and safety monitoring method for underground mines, characterized in that, The method is applied to a mobile mining vehicle equipped with a multi-sensor system, and the mobile mining vehicle is configured to perform safety monitoring of the underground mine during movement. The method includes: Based on the IMU pre-integration results, the acquired multi-sensor data is preprocessed to obtain an initial dataset, which includes corrected point cloud data, denoised IMU data, denoised thermal infrared data, and denoised visible light image data. A global point cloud map is generated based on the corrected point cloud data; Geometric semantic features, infrared semantic features, and texture semantic features are extracted from the global point cloud map, the denoised thermal infrared data, and the denoised visible light image data, respectively. The geometric semantic features, the infrared semantic features, and the texture semantic features are then fused to generate a multimodal information semantic map. Infrared threshold segmentation is performed on the multimodal information semantic map to identify low-temperature areas. The low-temperature areas are then spatially fused with the geometric information of the tunnel walls and roof in the point cloud map of the multimodal information semantic map to determine the seepage areas. The point cloud map of the multimodal information semantic map is registered with the mine design model, the point cloud distance field is calculated, and the area with a distance exceeding a preset distance threshold is identified as the deformation area based on the point cloud distance field. The seepage area and the deformation area are then embedded as risk areas into the navigation cost map. A global risk index is calculated based on the risk area, and a navigation path for the mine's mobile carrier is dynamically planned based on the global risk index and the navigation cost map. The mine's mobile carrier is configured to move based on the navigation path, and safety monitoring of the underground mine is carried out during the movement.

2. The multi-sensor tightly coupled SLAM and safety monitoring method for underground mines as described in claim 1, characterized in that, The process of generating a global point cloud map based on the corrected point cloud data includes: The corrected point cloud data is subjected to ground segmentation and non-ground point clustering to output structured feature point cloud clusters; The corrected point cloud data is stitched together based on the denoised IMU data to generate a local point cloud map; Based on the geometric distribution information of the structured feature point cloud clusters on the cross-section of the alley, a geometric feature descriptor is constructed, and the local point cloud map is optimized based on the geometric feature descriptor to obtain a global point cloud map.

3. The multi-sensor tightly coupled SLAM and safety monitoring method for underground mines as described in claim 2, characterized in that, The step of performing ground segmentation and non-ground point clustering on the corrected point cloud data to output structured feature point cloud clusters includes: The corrected point cloud data is sorted based on elevation coordinates, and the average elevation is calculated based on the sorting results. The elevation filtering interval is then determined based on the average elevation. Points within the height filtering interval are used as the initial ground seed point set, and the three-dimensional centroid and covariance matrix of the initial ground seed point set are calculated. Perform singular value decomposition on the covariance matrix, and use the right singular vector corresponding to the smallest singular value obtained by decomposition as the initial ground normal vector; Calculate the planar parameters of the initial ground normal vector, and generate a ground planar model based on the planar parameters; Calculate the distance between each point in the corrected point cloud data and the ground plane model, and take the points whose distance is less than a preset distance threshold as ground points, and iterate to optimize the ground plane model; The non-ground point set is obtained based on the optimized ground plane model, and the non-ground normal vector and local curvature of each non-ground point in the non-ground point set are determined to determine the non-ground seed point. Non-ground point clustering is performed based on the non-ground seed point to output a structured feature point cloud cluster.

4. The multi-sensor tightly coupled SLAM and safety monitoring method for underground mines as described in claim 3, characterized in that, The step of stitching together the corrected point cloud data based on the denoised IMU data to generate a local point cloud map includes: Based on the IMU pre-integration results of the mine's mobile vehicle, the SLAM system is used to predict the state and obtain prior state estimation results. The prior state estimation results include prior estimates of the navigation state of the mine's mobile vehicle and the corresponding prior covariance matrix of the navigation state prior estimates. in, This represents the prior estimate of the vehicle's navigation state at the next moment. This represents the posterior value of the navigation state of the vehicle at the current moment. This represents the original measurement vector of the carrier IMU at the current moment. This is the state transition function. This represents the process noise vector of the carrier IMU at the current moment. Let be the Jacobian matrix of the state transition function of the IMU. Let be the transpose of the Jacobian matrix of the IMU's state transition function. Let be the state covariance matrix. This represents the prior covariance matrix corresponding to the prior estimate of the carrier's navigation state at the next moment. The process noise covariance matrix; The process noise covariance matrix is ​​dynamically adjusted based on the process noise adaptive factor, and the measurement noise covariance is dynamically adjusted based on the matching residual of the corrected point cloud data. The Kalman gain coefficient is calculated based on the prior state estimation results, the dynamically adjusted process noise covariance matrix, and the measurement noise covariance. The dynamically adjusted process noise covariance matrix participates in the calculation of the Kalman gain coefficient through the prior covariance matrix. The calculation process of the prior covariance matrix directly depends on the dynamically adjusted process noise covariance matrix. The calculation of the Kalman gain coefficient refers to the following formula: in, Indicates the Kalman gain coefficient. This represents the measurement noise covariance. The Jacobian matrix represents the carrier laser measurement model. The transpose of the Jacobian matrix representing the carrier laser measurement model; A tightly coupled odometry is constructed based on the Kalman gain coefficients, and the state prior estimation results of the SLAM system are iteratively updated using the tightly coupled odometry. The corrected point cloud data is then stitched together based on the state posterior estimation results obtained after the iterative update to generate a local point cloud map.

5. The multi-sensor tightly coupled SLAM and safety monitoring method for underground mines as described in claim 4, characterized in that, The process of constructing a geometric feature descriptor based on the geometric distribution information of the structured feature point cloud clusters on the cross-section of the tunnel, and optimizing the local point cloud map based on the geometric feature descriptor to obtain a global point cloud map includes: The distribution of structured feature point cloud clusters on the cross-section of the tunnel is analyzed, geometric distribution information is extracted, and a geometric feature descriptor for the current frame is constructed based on the geometric distribution information. The geometric distribution information includes the maximum height, maximum width, arch height of the cross-section, and the angle between the fitted straight lines of the two side walls. Obtain the descriptors of historical keyframes stored in the hash table, and calculate the cosine similarity between the geometric feature descriptor of the current frame and the descriptors of the historical keyframes; Feature matching based on cosine similarity is used to determine the historical matching frames that match the current frame in the historical keyframes. The transformation matrix between the current frame point cloud and the historical matching frame point cloud is calculated based on the singular value decomposition strategy, and the current frame and the historical matching frame point cloud are aligned based on the transformation matrix. Calculate the point cloud overlap between the current frame and the historical matching frames after point cloud alignment, and optimize the local point cloud map based on the point cloud overlap to obtain a global point cloud map.

6. The multi-sensor tightly coupled SLAM and safety monitoring method for underground mines as described in any one of claims 1 to 5, characterized in that, The process of calculating a global risk index based on the risk area and dynamically planning the navigation path of the mine's movable carrier based on the global risk index and the navigation cost map includes: Risk weights for each risk area are determined based on historical mine safety incidents. The local risk index of each risk area is calculated based on the area, distribution density, and current distance between each risk area and the mine's mobile carrier. The global risk index is calculated based on the risk weights and the local risk indices of each risk region. The navigation path of the mine's mobile carrier is dynamically planned based on the local risk index and the navigation cost map. The local risk index is updated in real time based on multi-sensor data collected during the movement of the mobile carrier in the mine, and the change of the global risk index is monitored based on the local risk index. If the change in the global risk index exceeds a preset threshold, the navigation path of the mine's mobile carrier is dynamically replanned based on the current local risk index of each risk area. The process then returns to the steps of updating the local risk index in real time based on multi-sensor data collected during the movement of the mine's mobile carrier, and monitoring the change in the global risk index based on the local risk index.

7. A multi-sensor tightly coupled SLAM and safety monitoring device for underground mines, characterized in that, The device includes: The data processing module is used to preprocess the acquired multi-sensor data based on the IMU pre-integration results to obtain an initial dataset, which includes corrected point cloud data, denoised IMU data, denoised thermal infrared data, and denoised visible light image data. A point cloud map generation module is used to generate a global point cloud map based on the corrected point cloud data. The semantic map generation module is used to extract geometric semantic features, infrared semantic features, and texture semantic features from the global point cloud map, the denoised thermal infrared data, and the denoised visible light image data, respectively, and to fuse the geometric semantic features, the infrared semantic features, and the texture semantic features to generate a multimodal information semantic map. The water seepage identification module is used to perform infrared threshold segmentation on the multimodal information semantic map, identify low-temperature areas, and spatially fuse the low-temperature areas with the geometric information of the alleyway wall and roof in the point cloud map of the multimodal information semantic map to determine the water seepage area. The deformation recognition module is used to register the point cloud map of the multimodal information semantic map with the mine design model, calculate the point cloud distance field, and based on the point cloud distance field, identify areas whose distance exceeds a preset distance threshold as deformation areas, and embed the seepage area and the deformation area as risk areas into the navigation cost map. The dynamic safety monitoring module is used to calculate a global risk index based on the risk area, and to dynamically plan the navigation path of the mine's mobile carrier based on the global risk index and the navigation cost map. The mine's mobile carrier is configured to move based on the navigation path, and to monitor the underground mine during the movement.

8. The multi-sensor tightly coupled SLAM and safety monitoring device for underground mines as described in claim 7, characterized in that, The dynamic safety monitoring module is also used to determine the risk weight of each risk area based on historical mine safety incidents; calculate the local risk index of each risk area based on the area, distribution density, and current distance between each risk area and the mine's movable carrier; and calculate the global risk index based on the risk weight and the local risk index of each risk area. The navigation path of the mine's mobile carrier is dynamically planned based on the local risk index and the navigation cost map. The system updates the local risk index in real time based on multi-sensor data collected during the movement of the mine's mobile vehicle, and monitors the change in the global risk index based on the local risk index. In response to the change in the global risk index exceeding a preset threshold, the system dynamically plans the navigation path of the mine's mobile vehicle based on the current local risk index of each risk area, and returns to the previous steps of updating the local risk index in real time based on multi-sensor data collected during the movement of the mine's mobile vehicle and monitoring the change in the global risk index based on the local risk index.

9. A multi-sensor tightly coupled SLAM and safety monitoring device for underground mines, characterized in that, The multi-sensor tightly coupled SLAM and safety monitoring device for underground mines includes: a mobile unit, a multi-sensor system, a memory, a processor, and a multi-sensor tightly coupled SLAM and safety monitoring program for underground mines stored in the memory. The processor is used to run the multi-sensor tightly coupled SLAM and safety monitoring program for underground mines, and the multi-sensor tightly coupled SLAM and safety monitoring program for underground mines is configured to implement the multi-sensor tightly coupled SLAM and safety monitoring method for underground mines as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multi-sensor tightly coupled SLAM and safety monitoring program for underground mines, which, when executed by a processor, implements the multi-sensor tightly coupled SLAM and safety monitoring method for underground mines as described in any one of claims 1 to 6.

Images