A three-dimensional map reconstruction and target detection method for a coal mine underground inspection robot

Abstract

The application discloses a kind of coal mine underground inspection robot three-dimensional map reconstruction and target detection method: step 1, obtain the point cloud data set containing selected scene information;Step 2, the point cloud data is registered using Map-to-Map method, and the three-dimensional map after establishment is obtained;Step 3, according to the three-dimensional map, the inspection route of inspection robot is positioned in real time;Step 4, during the travel of inspection robot, target detection is carried out using fusion algorithm;Step 5, the target detection result obtained in step 4 is optimally matched, the classification of detection result is realized, and the optimal output is used as obstacle target.The application can also effectively construct high-precision map in the case of complex environmental results;At the same time, part of the redundant data can be removed, the algorithm time complexity is reduced, and the mapping efficiency is improved;Fusion algorithm considers time domain information, and multiple detection results are fused, which can effectively reduce the false detection rate and the missing rate caused by shielding.

Description

Technical Field

[0001] This invention belongs to the field of inspection robot technology, and relates to a method for three-dimensional map reconstruction and target detection of underground inspection robots in coal mines. Background Technology

[0002] Since 2016, the annual procurement volume of inspection robots has been increasing exponentially. Most companies have recognized the necessity and trend of intelligent robot inspection replacing human labor. As industry segmentation and scenario-based approaches become increasingly clear, customers are offering more and more personalized customizations. During this period of upward development, the annual demand will inevitably increase further. Currently, underground coal mines use a wide variety of manually driven fuel-powered or electric vehicles for tasks such as material transportation, patrol inspections, and personnel transfers. These operations suffer from low efficiency, high human resource consumption, and high operating costs. Meanwhile, traditional manual inspections are labor-intensive, inefficient, produce low-quality inspections, and pose risks to personal safety. The large workload and harsh environmental conditions also make it difficult to guarantee the quality and quantity of inspections. Therefore, unmanned inspection robots are the best choice to replace traditional human inspections.

[0003] The core problem of unmanned inspection robots is the localization problem. Inaccurate localization will cause major errors in decision control. The goal of inspection robot path planning is to find the optimal collision-free path from the starting point to the destination using a suitable path planning method. In a controllable structured environment, the inspection robot has a certain degree of autonomy, can perceive the surrounding environment, detect obstacles, and plan a collision-free path to navigate to the destination. Reference [1] describes the SLAM (Simultaneous Localization and Mapping) path planning problem of inspection robots. Reference [2] compares two SLAM schemes to determine the choice under different scenarios. Reference [3] studies the use of probabilistic methods to optimize map localization accuracy and reduce computational complexity. Reference [4] uses sparse pose adjustment to solve the matrix direct solution problem in nonlinear optimization. After studying Reference [5], it is found that the Gauss-Newton method can be used to solve the scanning matching problem in map localization, but it has high requirements for sensors. Finally, through the research of Reference [6], the Cartographer algorithm is analyzed, which reduces the cumulative error by using closed loop detection on both local subgraphs and global graphs.

[0004] Meanwhile, traditional obstacle detection methods for unmanned inspection robots cannot meet the demands of environmental perception. Therefore, multi-sensor fusion methods have been proposed to improve the accuracy, real-time performance, and robustness of obstacle detection. Lu Feng et al. used a convolutional neural network (Faster R-CNN) algorithm model to train data collected by an actual robot. They selected the set of center points of the lower edge of the bounding box in image target detection and matched them with the target data point set from radar target detection. An improved Iterative Closest Point Matching (ICP) algorithm was used to remove false point pairs, achieving data fusion between the image and 3D point cloud, effectively improving the algorithm's accuracy and efficiency. Li Yanfang et al. used a lidar to emit laser beams and receive target echoes, extracting the nearest echo points for cluster analysis to determine whether the echo points were obstacles. Image detection used a YOLO network training dataset to generate target boxes. The detection boxes from radar target detection and camera target detection were fused, using the percentage of overlapping area of ​​the bounding boxes as the criterion for obstacle judgment. This achieved decision-level fusion of lidar and camera, improving detection accuracy by approximately 5% for vehicles and pedestrians.

[0005] In summary, how to construct high-precision maps and perform obstacle detection are currently hot research topics in unmanned inspection. The main problems with the existing technologies mentioned above are as follows: the construction of high-precision maps is limited by the sensors and algorithms, making it difficult to perform mapping and localization operations in complex environments; traditional obstacle detection methods acquire too much data, and much of it is redundant, resulting in inaccurate estimation of the location of obstacles in the real world.

[0006] The relevant literature is as follows:

[0007] [1] Chen Zhuo, Su Weihua, Wei Anning, et al. Implementation of SLAM and path planning for mobile robots under ROS framework [J]. Medical and Health Equipment, 2017, 38(02):109-113.

[0008] [2] Luan Jianing, Zhang Wei, Sun Wei, et al. High-precision positioning algorithm based on fusion of QR code vision and lidar [J]. Computer Applications, 2021, 41(05): 1484-1491.

[0009] [3] Liu Liwei, Zhu Xukang, Li Xiuhua, et al. Research on map evaluation of 2D SLAM algorithm for low-cost mobile robots [J]. Computer Simulation, 2021, 38(04):291-295+320.

[0010] [4] Zhao Ruoyu, Guan Zhiwei, Tong Minyong, et al. Functional optimization design of SLAM system based on single-line lidar [J]. China Automotive, 2021(02):4-9+43.

[0011] [5] Han Wenhua. Research on dynamic environment navigation of mobile robots based on single-line lidar [D]. Harbin Institute of Technology, 2019.

[0012] [6] Ji Xingliang. Research on Synchronous Localization and Mapping Algorithm Based on 3D Laser Point Cloud Segmentation and Matching [D]. University of Electronic Science and Technology of China, 2020. Summary of the Invention

[0013] The purpose of this invention is to provide a method for three-dimensional map reconstruction and target detection for underground inspection robots in coal mines, in order to solve the technical problems of how to construct high-precision maps in complex structural environments and how to accurately estimate obstacles in high-precision maps.

[0014] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0015] A method for 3D map reconstruction and target detection of an underground coal mine inspection robot includes the following steps:

[0016] Step 1: Select the scene to build the map, collect and process the point cloud data of the selected scene, and obtain a point cloud dataset containing the selected scene information;

[0017] Step 2 involves performing point cloud registration on the point cloud data of the scene obtained in Step 1 using a Map-to-Map method to obtain the constructed 3D map; this specifically includes the following sub-steps:

[0018] Step 21: Obtain n consecutive frames of point cloud data from the point cloud dataset obtained in Step 1, and create a local submap for the n frames of point cloud data, using it as the latest local submap.

[0019] Step 22: Obtain n consecutive frames of point cloud data from the remaining point cloud data in the point cloud dataset, and create a local submap for these n frames of point cloud data.

[0020] Step 23: Using the global method in the cartographer algorithm, perform loop closure detection in the backend process. If a loop closure is found, proceed to step 24; otherwise, discard the local subgraph established in step 22 and return to step 22.

[0021] Step 24: Match the local subgraph established in step 22 with the latest local subgraph to obtain the matched local subgraph, and update the matched local subgraph to the latest local subgraph; where matching means superimposing two local subgraphs.

[0022] Step 25: Return to step 22 and continue until all point cloud data in the point cloud dataset obtained in step 1 has been acquired. The latest local sub-map obtained at this time is the required 3D map.

[0023] Step 3: Based on the 3D map generated in Step 2, perform real-time positioning of the inspection robot's inspection route;

[0024] Step 4: During the inspection robot's movement, target detection is performed using a fusion algorithm;

[0025] Step 5: Perform optimal matching on the target detection results obtained in Step 4 to classify the detection results and select the best output as the obstacle target.

[0026] Furthermore, in step 1, a 16-line LiDAR is used to collect laser point clouds using reserved instructions, generating a bag file containing selected scene information; then, the bag file is subjected to noise reduction filtering to obtain the processed bag file, which is the point cloud dataset containing selected scene information.

[0027] Furthermore, in step 23, the backend refers to the mapping optimization operation part of the Cartographer algorithm; the occurrence of loop means that the similarity between the local subgraph established in step 22 and the latest local subgraph is greater than a certain weight.

[0028] Furthermore, in step 24, the superposition of the two local subgraphs refers to discarding redundant point cloud frame data in the two local subgraphs and then stitching the two local subgraphs after discarding the redundant data together to obtain a new local subgraph.

[0029] Furthermore, step 3 includes the following sub-steps:

[0030] Step 31: Acquire pose information using an IMU inertial sensor; output the information as a CSV file via instructions.

[0031] Step 32: Select the coordinate system of the IMU inertial sensor as the coordinate system for the positioning operation;

[0032] Step 33: Using the CSV file generated in Step 31 and the 3D map generated in Step 2, along with the source code of the Cartographer-SLAM algorithm for localization, the inspection robot's travel route is located.

[0033] Furthermore, step 4 includes the following sub-steps:

[0034] Step 41: Perform noise reduction processing on the point cloud data collected in Step 1 using statistical filtering, and output the noise-reduced point cloud information.

[0035] Step 42: Use the RealSense D435i camera to acquire image information of the scene selected in Step 1, and then input the image information into the CenterNet fusion algorithm network for target detection, outputting the target center point position and target category C, which is known from the KIITI dataset;

[0036] Step 43: Input the point cloud data output in Step 41 into the CenterPoint fusion algorithm network for target detection, output the target 3D detection box, distance and category C, and extract the center point of the 3D target detection box.

[0037] 7. The method for 3D map reconstruction and target detection of an underground coal mine inspection robot as described in claim 1, characterized in that step 5 includes the following sub-steps:

[0038] Step 51: Select the target center point positions obtained in Step 42 and the target 3D detection box center point positions obtained in Step 43, and pair them one-to-one as inputs to the improved KNN algorithm, resulting in a total of n×n point pairs; where the target center point positions obtained in Step 42 are represented as... The center point position of the target 3D detection box obtained in step 43 is represented as follows: n is the number of center points selected;

[0039] Step 52: Calculate the Euclidean distance between the two center points in each pair of points selected in Step 51;

[0040] Step 53: Among all the Euclidean distances of point pairs obtained in step 52, select... The pair of points with the smallest Euclidean distance among the corresponding n point pairs is taken as the optimal point pair, resulting in a total of n optimal point pairs. Each optimal point pair is then fused to obtain the detected obstacle target, resulting in a total of n obstacle targets.

[0041] The beneficial effects of this invention are:

[0042] (1) Mapping: Based on the Cartographer algorithm, a Map-to-Map method is proposed to match local sub-map information, thereby performing loop closure detection and improving the matching degree. Simultaneously, redundant information between data frames is removed, improving map accuracy and mapping efficiency. This method is more suitable for mapping mobile robots. After the algorithm is improved, it can effectively construct high-precision maps even in complex environmental situations; at the same time, it can remove some redundant data, reducing algorithm time complexity and improving mapping efficiency.

[0043] (2) Target Detection: Based on multi-sensor fusion, target detection and tracking are performed on the original image and point cloud data respectively. An improved KNN algorithm is used to perform optimal matching of the detection results, and the center points of unmatched images are further matched by combining the projection of the point cloud onto the image. Finally, the best fusion result is output. Sensor fusion improves the reliability of the data and enhances the accuracy of target detection. At the same time, the fusion algorithm considers temporal information and fuses the detection results of multiple frames, which can effectively reduce the false detection rate and false negative rate caused by occlusion. Attached Figure Description

[0044] Figure 1 This demonstrates that existing technology leads to backend loopback detection errors due to frontend matching errors;

[0045] Figure 2 This is a comparison of the mapping results (point cloud registration maps) before and after the Cartographer algorithm improvement;

[0046] Figure 3 It is a comparison of the mapping effects before and after the improvement;

[0047] Figure 4 This is a comparison of CPU utilization before and after the improvement on the same server.

[0048] Figure 5 This refers to the back-end closed-loop detection and localization effect under RVIZ;

[0049] Figure 6 This is a schematic diagram of an inspection robot;

[0050] Figure 7 It is a fusion algorithm framework;

[0051] Figure 8 This is a block diagram of the target center point matching structure;

[0052] Figure 9 It is a graph showing the connection relationship between the center point set and the image detection center points;

[0053] Figure 10 The results are the image detection test results; (a) is a scene with target occlusion, (b) is a scene with uneven lighting, and (c) is a scene at a distance.

[0054] Figure 11 It is point cloud target detection; where (a) is a target occlusion scene, (b) is a scene with uneven lighting, and (c) is a scene at a distance.

[0055] Figure 12 These are the detection results of LiDAR in different scenarios (detection regression 3D bounding boxes); where (a) is a scene with target occlusion, (b) is a scene with uneven lighting, and (c) is a scene at a distance.

[0056] Figure 13 The results are the fusion detection results of point cloud and image detection; where (a) is the image detection result and (b) is the detection result of the fusion algorithm.

[0057] The present invention will be further explained and described below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0058] 1. SLAM mapping and localization

[0059] 1.1 Graph Optimization Algorithm

[0060] The Cartographer algorithm belongs to the graph optimization SLAM algorithm. Graph optimization SLAM adjusts the front-end pose by detecting loop closures in the back-end, ultimately obtaining the robot's position and pose that is closest to the true value. The graph optimization SLAM problem is decomposed into two tasks:

[0061] (1) Construct a graph, with robot pose as vertices and relationships between poses as edges (front-end - accumulation of sensor information);

[0062] (2) Optimize the graph and adjust the robot pose vertices to satisfy the edge constraints (back-end);

[0063] 1.1.1 Cartographer Algorithm

[0064] The Cartographer algorithm is an open-source algorithm proposed by Google in 2016. It proposes a novel loop closure detection method based on LiDAR point cloud data. This method reduces computational cost, meets the needs of large-scale mapping, and performs real-time optimization for large-scale data. The entire algorithm consists of two main parts: Local SLAM (front-end matching) and Global SLAM (back-end loop closure). This invention improves the Local front-end, which, combined with the Cartographer algorithm's back-end loop closure component, can improve the accuracy of the reconstructed map.

[0065] The original algorithm uses the Scan-to-Map registration method in SLAM point cloud registration for front-end point cloud registration. After acquiring a laser scan (radar frame data) frame, it matches it with the most recently established Map and inserts the laser scan data of this frame into the Map. The drawback of this method is that when the environmental structure is similar, the acquired point cloud data frames may have similar and repeated situations. When used for loop closure detection in the back-end, this will lead to the inability to close loops, resulting in map construction failure.

[0066] like Figure 1 The problem stems from a front-end matching error, which prevents the back-end detection from closing the loop.

[0067] 1.1.2 Point Cloud Registration

[0068] Therefore, based on the Cartographer algorithm, a novel LiDAR point cloud registration method, Map-to-Map, is proposed. After acquiring several frames of point cloud data, a Submap is established. Each newly established Submap is matched with the most recently established Submap, thus avoiding matching point cloud data frames with similar environmental structures, which could lead to loop closure detection errors in the backend. The specific process is as follows: Figure 2 .

[0069] 2. Target Detection

[0070] Before target detection, joint calibration of the LiDAR and depth camera is required. Joint calibration is the foundation of data fusion. Through joint calibration, the transformation matrix between the camera coordinate system and the LiDAR coordinate system is obtained, fusing the radar and camera data to obtain more comprehensive and accurate information. Figure 6 This is a joint calibration diagram.

[0071] The method for 3D map reconstruction and target detection of an underground coal mine inspection robot of the present invention includes the following steps:

[0072] Step 1: Select the scene to build the map, collect and process the point cloud data of the selected scene, and obtain a point cloud dataset containing the selected scene information: Use a 16-line LiDAR (model Robsense) to collect LiDAR point clouds using the reserved command (rosbag record + topic name) to generate a bag file containing the selected scene information; then perform noise reduction filtering on the bag file (statistical filtering algorithm) to obtain the processed bag file.

[0073] Step 2: Apply the Map-to-Map method to the point cloud data of the scene obtained in Step 1 to perform point cloud registration, thereby obtaining the constructed 3D map; for example... Figure 2 As shown, the specific operation is as follows:

[0074] Step 21: Obtain n consecutive frames of point cloud data from the point cloud dataset obtained in Step 1, and create a local submap for the n frames of point cloud data, using it as the latest local submap.

[0075] Step 22: Obtain n consecutive frames of point cloud data from the remaining point cloud data in the point cloud dataset, and create a local submap for these n frames of point cloud data.

[0076] Step 23: Using the global method in the cartographer algorithm, perform loop closure detection in the backend process. If a loop closure is found, proceed to step 24; otherwise, discard the local subgraph established in step 22 and return to step 22.

[0077] The backend refers to the cartographer algorithm's mapping optimization operation; the occurrence of loops means that the similarity between the local subgraph established in step 22 and the latest local subgraph is greater than a certain weight, which can be set by the user.

[0078] Step 24: Match the local subgraph established in step 22 with the latest local subgraph to obtain the matched local subgraph, and update the matched local subgraph to the latest local subgraph; where matching means superimposing two local subgraphs (i.e. discarding redundant point cloud frame data in the two local subgraphs and stitching the two local subgraphs after discarding redundant data to obtain a new local subgraph).

[0079] In the process of matching two local subgraphs, the pose transformation formula when inserting the point cloud data of the local subgraph into the latest local subgraph in step 22 is as follows:

[0080]

[0081] in, This represents the pose of each point cloud in a point cloud dataset that contains scene information. Let x be the translation of the point cloud in the x and y directions. This represents the rotation of the plane; represents the pose of the point cloud in the current local subgraph; p is the probability value of the presence of obstacles, which can be set by the user.

[0082] Step 25: Return to step 22 and continue until all point cloud data in the point cloud dataset obtained in step 1 has been acquired. The latest local sub-map obtained at this point is the required 3D map.

[0083] By using the Map-to-Map method in step 2 to register point clouds and construct a 3D map, we can avoid the problems of existing methods that simply import each frame containing selected scene information into the established sub-map, which leads to matching of point cloud data frames with similar environmental structures and causes loop closure detection errors in the backend.

[0084] Step 3: Based on the 3D map generated in Step 2, perform real-time positioning of the inspection robot's inspection route; the specific operation is as follows:

[0085] Step 31: Use an IMU inertial sensor, connect it to a computer, and collect pose information; output the information as a CSV file via commands.

[0086] Step 32: Select the coordinate system of the IMU inertial sensor as the coordinate system for the positioning operation;

[0087] Step 33: Using the CSV file generated in Step 31 and the 3D map generated in Step 2, along with the Cartographer-SLAM algorithm localization part of the source code (by inputting the absolute path of the file in the source code and modifying the configuration file parameters), the inspection robot's travel route is located. Figure 5 Result (the solid line in the figure represents the trajectory of the positioning operation).

[0088] Step 4: During the inspection robot's movement, target detection is performed using a fusion algorithm. The specific operation is as follows:

[0089] Step 41: Perform noise reduction processing on the point cloud data collected in Step 1 using statistical filtering, and output the noise-reduced point cloud information.

[0090] Step 42: Use the RealSense D435i camera to acquire image information of the scene selected in Step 1, and then input the image information into the CenterNet fusion algorithm network for target detection, outputting the target center point position and target category C, which is known from the KIITI dataset;

[0091] Step 43: Input the point cloud data output in Step 41 into the CenterPoint fusion algorithm network for target detection, output the target 3D detection box, distance and category C, and extract the center point of the 3D target detection box.

[0092] Specifically, such as Figure 6 As shown, the inspection robot is 95cm tall, with a chassis height of 25cm. Both the lidar and camera are mounted on the robot, with the camera located directly below the lidar. The IMU inertial sensor is mounted on the robot's chassis.

[0093] Step 5 and Step 4 respectively output the target center point positions in the image and the point cloud. Among these center points, there are various detection scenarios, including correctly detected and tracked targets, and incorrectly detected targets. The improved KNN algorithm is used to perform optimal matching on the target detection results obtained in Step 4, classifying the detection results and selecting the best ones as obstacle targets. The specific operation is as follows:

[0094] Step 51: Select the target center point positions obtained in Step 42 and the target 3D detection box center point positions obtained in Step 43, and pair them one-to-one as inputs to the improved KNN algorithm, resulting in a total of n×n point pairs. The target center point positions obtained in Step 42 are represented as follows: The center point position of the target 3D detection box obtained in step 43 is represented as follows: n is the number of center points selected;

[0095] Step 52: Calculate the Euclidean distance between the two center points in each pair of points selected in Step 51;

[0096] Step 53: Among all the Euclidean distances of point pairs obtained in step 52, select... The pair of points with the smallest Euclidean distance among the corresponding n point pairs is taken as the optimal point pair, resulting in a total of n optimal point pairs. Each optimal point pair is then fused, which means merging the two points in the pair into one point (arbitrarily choosing one as the center point of the pair), and this point is used as the detected obstacle target, resulting in a total of n obstacle targets.

[0097] During the center point fusion process, the target point in the current frame image and the center point of the target detection bounding box in the point cloud are fused. Black points represent the set of target detection center points in the point cloud data, and white points represent the center points of the image detection results (see...). Figure 9 The two are then merged into the same target center point through nearest-neighbor matching in step 6, and the connection relationship is as follows: Figure 8 As shown.

[0098] To verify the feasibility and effectiveness of this invention, the following experiments were conducted:

[0099] 1. SLAM mapping and localization

[0100] To demonstrate that the improved Cartographer algorithm achieves better mapping results under similar environmental structures, this experiment used a simulated underground coal mine to collect laser point cloud data. Maps were constructed using both the improved and unimproved algorithms. The mapping time efficiency, mapping quality, and mapping accuracy were compared (mapping was performed using processed bag files; mapping efficiency is shown in Figure 1). Figure 4 As shown in the figure, this demonstrates that the improved algorithm is superior to the original algorithm.

[0101] (1) Comparison of mapping results:

[0102] Figure 3 To improve the mapping results before and after:

[0103] Before improvement: In cases with similar environmental structures, loop closure detection errors exist, resulting in low map accuracy;

[0104] Improved: Even in complex and similar environmental structures, loop closure detection can still be performed to achieve the desired loop closure effect.

[0105] (2) Comparison of mapping efficiency

[0106] By comparing the CPU utilization during the mapping process before and after the algorithm improvement (on the same server), the mapping efficiency before and after the algorithm improvement can be obtained.

[0107] This experiment uses a quad-core processor for mapping. Figure 4 It can be seen that the CPU utilization rate before the improvement was higher than that after the improvement, and the CPU utilization rate after the improvement was lower, tending to stabilize over time. Therefore, under the same experimental conditions, the improved Cartographer algorithm is more suitable for mapping mobile robots than the original one.

[0108] (3) Back-end closed-loop positioning

[0109] A novel LiDAR point cloud registration method, Map-to-Map, is proposed. After acquiring several frames of point cloud data, a submap is created for point cloud registration, while redundant data is removed. This data is then transmitted to the backend for loop closure detection and localization. Figure 5 The back-end closed-loop detection and localization effect under RVIZ:

[0110] Because the above steps have improved point cloud matching, the resulting 3D map effect is better than before. As the input for the positioning operation, the 3D map can guarantee the accuracy requirements, which can be seen from the RVIZ software (the trajectory line depicted in the figure), and the effect is good.

[0111] 2. Target Detection

[0112] ① After training and processing the KITTI dataset, three sets of data from different vehicle driving scenarios were selected for testing. The categories and target locations were labeled. The image detection test results are as follows: Figure 9 As shown;

[0113] ② CenterNet image object detection, which relies on a camera, has significant limitations. Figure 10 In scenarios with occlusion and uneven lighting, there is a significant problem of missed detection of occluded vehicles. The camera detection results are affected by lighting conditions, which greatly reduces the measurement accuracy and leads to a large number of missed targets. The test results show that camera-based target detection is difficult to meet the requirements of open-air autonomous mining trucks.

[0114] ③ Figure 11 (a), (b), and (c) show the detection results of LiDAR in different scenarios, from... Figure 11 The detection box in (a) shows that the image detection did not detect the target, while the LiDAR detected the target. From... Figure 11As shown in (b), there is a deviation between the target position detected by the lidar and the actual target position. Because point cloud data is discretely distributed, the positions and intervals of data points are irregularly distributed in three-dimensional space, making it difficult to select points of the same type, resulting in inaccurate target positioning. Figure 11 As shown in (b), due to the irregularity of the point cloud data itself, the positions of the red and green detection boxes deviate in different scenarios. To obtain more accurate target location information, the detection results of the image and the point cloud detection results are fused to solve this problem.

[0115] ④ In different test scenarios, image detection exhibits significant missed detections. In point cloud target detection, targets that are far away and blurry can be effectively located and their bounding boxes captured, such as... Figure 12 As shown in (b), the fusion algorithm increases the number of occluded vehicles detected by 7 compared to the single image detection method. Obviously, the fusion algorithm effectively reduces the false negative rate under conditions such as occlusion and uneven lighting.

[0116] in conclusion:

[0117] (1) Mapping and Localization: This paper studies the Graph-slam (graph optimization) mapping algorithm—Cartographer—and analyzes its point cloud registration method, Scan to Map (frame and local sub-map), which suffers from loop closure detection errors in cases of similar environmental structures, leading to low map accuracy. Furthermore, redundant data exists between frames, resulting in low mapping efficiency. To address these issues, a novel LiDAR point cloud registration method—Map to Map (local sub-map to local sub-map)—is proposed based on this algorithm. This method acquires point cloud data to construct local sub-maps, and registers these sub-maps with each other to perform loop closure detection. This improves the matching accuracy and removes redundant information between data frames, thereby increasing map accuracy and mapping efficiency. It is more suitable for mapping mobile robots.

[0118] (2) Obstacle Detection: This invention proposes a radar and camera decision layer fusion method. Point cloud data is acquired using a 16-line LiDAR and a Realsense D435 depth camera. Through joint calibration of the LiDAR and camera, the point cloud data and image data are projected onto the same coordinate system. Combined with a target detection algorithm, obstacles in the image are regressed to target center points (including depth, size, and category), and obstacles in the point cloud are regressed to a set of target center points. The KNN (K-Nearest Neighbor) method is used to fuse the target center points of the image and point cloud data to achieve optimal matching of the detection results, finally outputting a highly reliable fusion result.

Claims

1. A method for 3D map reconstruction and target detection for an underground coal mine inspection robot, characterized in that, Includes the following steps: Step 1: Select the scene to build the map, collect and process the point cloud data of the selected scene, and obtain a point cloud dataset containing the selected scene information; Step 2 involves performing point cloud registration on the point cloud data containing scene information obtained in Step 1 using a Map-to-Map method to obtain the constructed 3D map; this specifically includes the following sub-steps: Step 21: Obtain n consecutive frames of point cloud data from the point cloud dataset obtained in Step 1, and create a local submap for the n frames of point cloud data, using it as the latest local submap. Step 22: Obtain n consecutive frames of point cloud data from the remaining point cloud data in the point cloud dataset, and create a local submap for these n frames of point cloud data. Step 23: Using the global method in the cartographer algorithm, perform loop closure detection in the backend process. If a loop closure is found, proceed to step 24; otherwise, discard the local subgraph established in step 22 and return to step 22. Step 24: Match the local subgraph established in step 22 with the latest local subgraph to obtain the matched local subgraph, and update the matched local subgraph to the latest local subgraph; where matching means superimposing two local subgraphs. Step 25: Return to step 22 and continue until all point cloud data in the point cloud dataset obtained in step 1 has been acquired. The latest local sub-map obtained at this time is the required 3D map. Step 3: Based on the 3D map generated in Step 2, perform real-time positioning of the inspection robot's inspection route; Step 4: During the inspection robot's movement, target detection is performed using a fusion algorithm; Step 5: Perform optimal matching on the target detection results obtained in Step 4 to classify the detection results and select the best output as the obstacle target.

2. The method for 3D map reconstruction and target detection of an underground coal mine inspection robot as described in claim 1, characterized in that, In step 1, a 16-line LiDAR is used to collect laser point clouds using reserved instructions, generating a bag file containing selected scene information. Then, the bag file is subjected to noise reduction filtering to obtain the processed bag file, which is the point cloud dataset containing selected scene information.

3. The method for 3D map reconstruction and target detection of an underground coal mine inspection robot as described in claim 1, characterized in that, In step 23, the backend refers to the mapping optimization operation part of the Cartographer algorithm; the occurrence of loop means that the similarity between the local subgraph established in step 22 and the latest local subgraph is greater than a certain weight.

4. The method for 3D map reconstruction and target detection of an underground coal mine inspection robot as described in claim 1, characterized in that, In step 24, the superposition of the two local subgraphs means discarding redundant point cloud frame data in the two local subgraphs and then stitching the two local subgraphs after discarding the redundant data together to obtain a new local subgraph.

5. The method for 3D map reconstruction and target detection of an underground coal mine inspection robot as described in claim 1, characterized in that, Step 3 includes the following sub-steps: Step 31: Acquire pose information using an IMU inertial sensor; output the information as a CSV file via instructions. Step 32: Select the coordinate system of the IMU inertial sensor as the coordinate system for the positioning operation; Step 33: Using the CSV file generated in Step 31 and the 3D map generated in Step 2, along with the source code of the Cartographer-SLAM algorithm for localization, the inspection robot's travel route is located.

6. The method for 3D map reconstruction and target detection of an underground coal mine inspection robot as described in claim 1, characterized in that, Step 4 includes the following sub-steps: Step 41: Perform noise reduction processing on the point cloud data collected in Step 1 using statistical filtering, and output the noise-reduced point cloud information. Step 42: Use the RealSense D435i camera to acquire image information of the scene selected in Step 1, and then input the image information into the CenterNet fusion algorithm network for target detection, outputting the target center point position and target category C, which is known from the KIITI dataset; Step 43: Input the point cloud data output in Step 41 into the CenterPoint fusion algorithm network for target detection, output the target 3D detection box, distance and category C, and extract the center point of the 3D target detection box.

7. The method for 3D map reconstruction and target detection of an underground coal mine inspection robot as described in claim 1, characterized in that, Step 5 includes the following sub-steps: Step 51: Select the target center point positions obtained in Step 42 and the target 3D detection box center point positions obtained in Step 43, and pair them one-to-one as inputs to the improved KNN algorithm, resulting in a total of n×n point pairs; where the target center point positions obtained in Step 42 are represented as... The center point position of the target 3D detection box obtained in step 43 is represented as follows: n is the number of center points selected; Step 52: Calculate the Euclidean distance between the two center points in each pair of points selected in Step 51; Step 53: Among all the Euclidean distances of point pairs obtained in step 52, select P. yi The pair of points with the smallest Euclidean distance among the corresponding n point pairs is taken as the optimal point pair, resulting in a total of n optimal point pairs. Each optimal point pair is then fused to obtain the detected obstacle target, resulting in a total of n obstacle targets.

Images