An underground mine inspection and risk elimination robot target detection method and system

Abstract

This invention relates to a target detection method and system for a mine inspection and hazard removal robot. The method includes: acquiring 3D point cloud data and image data in front of the robot; performing semantic segmentation on the image data to obtain semantic features; fusing the 3D point cloud data and the semantic features of the image to obtain the mapping relationship between the semantic features of each pixel in the image and the point cloud, and removing points from the 3D point cloud data that do not have a mapping relationship; traversing each point in the 3D point cloud data, and for each point, obtaining a feature encoding by combining its nearest neighbors; then inputting the encoding into a continuous convolutional network with an attention mechanism and a point cloud pooling operation respectively; finally concatenating the output of the point cloud pooling operation with the output of the continuous convolutional network with an attention mechanism to obtain fused point cloud data; and performing target detection using the fused point cloud data. This invention significantly improves the accuracy and success rate of hazard detection.

Description

Technical Field

[0001] This invention relates to the field of intelligent robots, and in particular to a target detection method and system for a mine inspection and hazard removal robot. Background Technology

[0002] Currently, the main type of exploration robot used in mines is the intelligent inspection robot, which is suspended on a track and moves back and forth in the tunnel, and cannot freely inspect underground. In addition, the intelligent inspection robot uses a camera to collect images of the site in real time, which has certain limitations, such as (1) being limited by lighting conditions: the performance of the camera is affected by lighting conditions, and it may perform poorly at night or in low light conditions. (2) Depth information is not directly available: the camera can provide two-dimensional image information, but obtaining depth information requires additional processing and calculation, which may increase the complexity of the system. (3) being sensitive to occlusion: the camera may cause blind spots for occluded objects, which may result in the occluded objects not being detected or identified. Summary of the Invention

[0003] To address the aforementioned problems, this invention proposes a target detection method and system for a mine inspection and hazard mitigation robot.

[0004] The specific plan is as follows:

[0005] A target detection method for a mine inspection and hazard mitigation robot includes the following steps:

[0006] S1: The three-dimensional point cloud data and image data in front of the robot are collected by the lidar and binocular camera installed at the front of the mine inspection robot.

[0007] S2: Perform semantic segmentation on the image data to obtain the semantic features of the image;

[0008] S3: Fuse the 3D point cloud data with the semantic features of the image to obtain the mapping relationship between the semantic features of each pixel in the image and the point cloud, and remove the point cloud that has no mapping relationship from the 3D point cloud data.

[0009] S4: Traverse each point in the 3D point cloud data. For each point, first search for its K nearest neighbors in the 3D point cloud data and obtain the semantic features corresponding to the nearest neighbors based on the mapping relationship. Then, based on the semantic features of the K nearest neighbors and the geometric offset of the K nearest neighbors relative to the point, obtain the feature encoding of the point. Then, use the encoding result as two inputs: one is input into a continuous convolutional network and combined with an attention mechanism to fuse semantic and geometric features, and the other is used as input to the point cloud pooling operation. Finally, concatenate the output of the point cloud pooling operation with the output of the continuous convolutional network combined with the attention mechanism to obtain the fused point cloud data.

[0010] S5: Target detection is performed using the fused point cloud data.

[0011] Furthermore, step S1 also includes preprocessing the acquired 3D point cloud data, including point cloud filtering and denoising, and ground point cloud segmentation.

[0012] Furthermore, the DeepLabV3+ network is used for semantic segmentation.

[0013] Furthermore, when the field of view of the two cameras overlaps when the point cloud is projected onto the binocular camera, one of the following two strategies is adopted: one strategy is to randomly select a segmentation score vector from the two images; the other strategy is to select it by comparing the entropy of the two segmentation scores or the difference between the first two scores.

[0014] A target detection system for an underground mine inspection and hazard removal robot includes an underground mine inspection robot, a controller, a lidar, and a binocular camera. The system implements the steps of the method described in the embodiments of the present invention.

[0015] Furthermore, the mine inspection robot is equipped with two power wheels on its feet, which are driven by direct-drive joints and hub motors.

[0016] The present invention adopts the above technical solution, which integrates lidar and binocular camera, greatly improving the accuracy and success rate of hazard detection. Attached Figure Description

[0017] Figure 1 The diagram shown is a flowchart of a method according to an embodiment of the present invention.

[0018] Figure 2 The diagram shown is a schematic representation of the fusion process in this embodiment. Detailed Implementation

[0019] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments, and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention.

[0020] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0021] Example 1:

[0022] This invention provides a target detection method for mine inspection and hazard mitigation robots, such as... Figure 1 As shown, the method includes the following steps:

[0023] S1: The three-dimensional point cloud data and image data in front of the robot are collected by the lidar and binocular camera installed at the front end of the underground inspection robot.

[0024] Binocular cameras offer the following advantages: 1. High resolution: Binocular cameras can provide high-resolution image data, capturing richer scene information. 2. Lower cost: Binocular cameras are generally less expensive than LiDAR because they are more common and ubiquitous sensors. 3. Sensitive to color and texture: Binocular cameras can capture color and texture information in the environment, which is crucial for certain applications (such as object recognition).

[0025] LiDAR offers the following advantages: 1. High-precision ranging: LiDAR provides highly accurate distance measurements, typically within a few millimeters. 2. Unaffected by lighting conditions: LiDAR is independent of lighting conditions and can operate normally under various lighting conditions, including at night and in inclement weather. 3. Omnidirectional perception: LiDAR provides 360-degree omnidirectional perception, which is extremely useful for building environmental maps and for positioning and navigation.

[0026] During the scanning process of the surrounding environment, LiDAR generates a large number of point clouds. Currently, mainstream 16-line LiDARs scan approximately 300,000 point clouds per second, 32-line LiDARs scan approximately 600,000 point clouds per second, and 64-line LiDARs can scan up to approximately 2 million point clouds per second. However, not all point clouds are valid; they contain a large number of "interference points." For example, during environmental perception by a smart car, there are inevitably small, suspended obstacles such as fallen leaves and flying insects around it. If the reflected point clouds from these objects are also included in the data to be calculated, it will generate a huge computational load and also cause problems for the decision-making system. Furthermore, in applications that rely on point cloud matching for positioning, the large amount of noisy point clouds generated by dense and irregular objects such as bushes around the smart car can prevent the vehicle from accurately locating itself. Besides the above, the following factors also contribute to interference in the original point cloud:

[0027] (1) Vibrations in hardware such as pulses and detectors of the lidar transmitting system may generate noise;

[0028] (2) Some special materials, such as glass, have a greater refractive effect on lasers;

[0029] (3) Mutual occlusion of distant objects can also cause errors in point cloud acquisition.

[0030] The presence of this noise not only reduces data accuracy but also increases the system's computational load, wasting computing resources. Mapping and detection using point clouds generally require high precision; without eliminating the impact of this noise, the real-time performance and accuracy of detection will fall short. However, due to the complexity of the actual environment and conditions when using radar, it is difficult to completely eliminate these influencing factors at their source. Considering all factors, the most economical method to obtain high-precision mapping and detection results is to perform appropriate preprocessing operations on the point cloud data. Common point cloud preprocessing includes point cloud filtering, point cloud data registration, point cloud segmentation, and data compression. This embodiment mainly uses point cloud filtering for noise reduction and ground point cloud segmentation for point cloud preprocessing.

[0031] S2: Perform semantic segmentation on the image data to obtain the semantic features of the image.

[0032] Among the four major tasks of image processing—classification, segmentation, detection, and tracking—target segmentation performs pixel-level target detection in an image. This means not only separating the target from the background but also associating each pixel within the target with an object type, rather than simply marking the approximate location of the target in the image. Target segmentation is further divided into semantic segmentation and instance segmentation. Semantic segmentation only needs to identify whether a pixel belongs to a certain type of target; while instance segmentation needs to identify whether a pixel belongs to a specific instance within that type of target. The correspondence between radar point clouds and pixels has an inherent advantage over the correspondence between radar point clouds and a specific region. Considering the limitations of the application scenario and computing power, this embodiment chooses semantic segmentation as the image processing method in the fusion module.

[0033] Common semantic segmentation networks include Fully Convolutional Neural Networks (FCNs), SegNet, U-Net, and the DeepLab series. FCNs are more efficient than traditional CNNs, avoiding redundant storage and computation, but they have a small receptive field, low computational cost, and lack spatial consistency. SegNet is relatively accurate in segmenting small objects, but performs poorly on large objects. U-Net is fast, but because it can only perform local learning, it is generally used in the medical field. The DeepLab network introduces dilated convolutional networks and uses conditional random fields, improving the model's ability to capture details.

[0034] The DeepLab series currently has four versions: V1, V2, V3, and V3+, each optimized from the previous version. DeepLab V1 combines deep convolutional neural networks and probabilistic models, replacing the final fully connected layer of VGG-16 with a convolutional layer and expanding the receptive field through dilated convolutions. DeepLab V2, building on V1, proposed a multi-dilated spatial convolutional pooling pyramid structure. While this improved upon V1's approach to multi-scale target issues in some images, the results were still not ideal. DeepLab V3 uses dilated convolutions with different sampling rates and BN layers to form an ASSP module. DeepLab V3+ employs an encoder-decoder structure, using V3 as the encoder and designing a separate decoder structure, improving the accuracy and precision of image recognition. Considering the performance characteristics of these networks, this embodiment selects the DeepLab V3+ network to filter experimental targets from images.

[0035] S3: Fuse the 3D point cloud data with the semantic features of the image to obtain the mapping relationship between the semantic features of each pixel in the image and the point cloud, and remove the point cloud that has no mapping relationship from the 3D point cloud data.

[0036] The characteristic of point clouds being "sparser in distant areas and denser in nearby areas" often makes pure point cloud-based algorithms inefficient in some complex scenes. Therefore, this embodiment introduces visual information for fusion based on pure point clouds, providing a more solid foundation for subsequent detection models. The specific fusion approach is as follows: first, semantic segmentation is used to extract and classify the pixel information of the image; then, coordinate transformation is used to project (map) the point cloud onto the image coordinate system; finally, the extracted image semantic information is added to the feature dimension of the point cloud at the corresponding location, thereby achieving the fusion of the two information.

[0037] Once the semantic segmentation of the image is complete, the radar point cloud needs to be matched and fused with the semantic segmentation results. The basis for fusing point cloud and image semantic information is to map the point cloud coordinate system to the camera imaging plane coordinate system. Based on existing camera imaging models and the joint calibration principle of LiDAR and camera, it can be known that the point cloud can be projected onto the camera's imaging plane through a series of matrix transformations.

[0038] During the projection process described above, if a binocular camera is used, the fields of view of the two cameras may overlap, meaning that the point cloud from one radar is simultaneously projected onto two images. There are two strategies for handling this situation: one is to randomly select a segmentation vector from the two images; the other strategy is to select it by comparing the entropy of the two segmentation vectors or the difference between the first two vectors.

[0039] After the above transformation, the point cloud data of the LiDAR is projected onto the imaging plane of the camera, and a specific coordinate correspondence is established between the pixels of the point cloud and the image. Before performing point cloud and image registration, in order to improve overall operating efficiency, point cloud data outside the camera's field of view needs to be removed.

[0040] S4: Traverse each point in the 3D point cloud data. For each point, firstly, search for its K nearest neighbors in the 3D point cloud data and obtain the semantic features corresponding to the nearest neighbors based on the mapping relationship. Then, based on the semantic features of the K nearest neighbors and their geometric offsets relative to the point, obtain the feature encoding of the point. Then, use the encoding result as two inputs: one is input into a continuous convolutional network combined with an attention mechanism to fuse semantic and geometric features, and the other is input to the point cloud pooling operation. Finally, concatenate the output of the point cloud pooling operation with the output of the continuous convolutional network combined with the attention mechanism to obtain the fused point cloud data, such as... Figure 2 As shown,

[0041] In terms of the data dimensions of point clouds, point cloud data can be quantized as P = {p1, ..., p2}. N}, where a single point cloud can be represented as p i =(x i ,v i ). x i It is a three-dimensional matrix representing the spatial coordinates (x, y, z) of the original point cloud; v i ∈R k , representing the k-dimensional vector of the original point cloud attribute state. After the fusion process is complete, the pixel segmentation score at the corresponding index position is appended to the feature dimension of the radar point cloud, i.e., p. i =(x i ,v i v in ) i Compared to its original data format, it has added some data dimensions. In this way, we obtain point cloud data with enhanced semantic information, realizing the fusion of point cloud and image.

[0042] S5: Target detection is performed using the fused point cloud data.

[0043] This invention integrates lidar with a binocular camera, greatly improving the accuracy and success rate of hazard detection.

[0044] Example 2:

[0045] The present invention also provides a target detection system for an underground inspection and hazard removal robot, including an underground inspection robot, a controller, a lidar, and a binocular camera. The system executes the steps in the above-described method embodiment of Embodiment 1 of the present invention through the controller.

[0046] Furthermore, to adapt to the complex and ever-changing environment underground, this embodiment is equipped with two powered wheels on the robot's feet. Through direct-drive joints and hub motors, it perfectly combines the robot's speed and agility with the strong adaptability of legged robots. This allows the robot, compared to ordinary four-wheeled inspection and hazard-clearing robots used in underground mines, to move and work more efficiently and flexibly, conduct covert inspections, nimbly navigate, and smoothly transport materials in various terrains within the mine. Specifically, this embodiment installs six freely movable M1520D direct-drive robot joint motors in the robot.

[0047] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A target detection method for a mine inspection and hazard mitigation robot, characterized in that, Includes the following steps: S1: The three-dimensional point cloud data and image data in front of the robot are collected by the lidar and binocular camera installed at the front of the mine inspection robot. S2: Use the DeepLabV3+ network to perform semantic segmentation on the image data to obtain the semantic features of the image; S3: Fusion of 3D point cloud data with the semantic features of the image to obtain the mapping relationship between the semantic features of each pixel in the image and the point cloud, and removal of point clouds without mapping relationship from the 3D point cloud data; wherein, fusion of 3D point cloud data with the semantic features of the image includes: projecting the point cloud onto the image coordinate system through coordinate transformation, and adding the semantic features of the image to the feature dimension of the point cloud at the corresponding position. S4: Traverse each point in the 3D point cloud data. For each point, first search for its K nearest neighbors in the 3D point cloud data and obtain the semantic features corresponding to the nearest neighbors based on the mapping relationship. Then, based on the semantic features of the K nearest neighbors and the geometric offset of the K nearest neighbors relative to the point, obtain the feature encoding of the point. Then, use the encoding result as two inputs: one is input into a continuous convolutional network and combined with an attention mechanism to fuse semantic and geometric features, and the other is used as input to the point cloud pooling operation. Finally, concatenate the output of the point cloud pooling operation with the output of the continuous convolutional network combined with the attention mechanism to obtain the fused point cloud data. S5: Target detection is performed using the fused point cloud data.

2. The target detection method for the mine inspection and hazard removal robot according to claim 1, characterized in that: Step S1 also includes preprocessing the acquired 3D point cloud data, which includes point cloud filtering and noise reduction and ground point cloud segmentation.

3. The target detection method for the mine inspection and hazard removal robot according to claim 1, characterized in that: When the field of view of the two cameras overlaps when the point cloud is projected onto the binocular camera, one of the following two strategies is adopted: one strategy is to randomly select a segmentation score vector from the two images; the other strategy is to select it by comparing the entropy of the two segmentation scores or the difference between the first two scores.

4. A target detection system for a mine inspection and hazard mitigation robot, characterized in that: The system includes an underground inspection robot, a controller, a lidar, and a binocular camera, and implements the steps of the method as described in any one of claims 1 to 3.

5. The target detection system for the mine inspection and hazard mitigation robot according to claim 4, characterized in that: The mine inspection robot is equipped with two power wheels on its feet, which are driven by direct-drive joints and hub motors.

Images