A coal mine underground unmanned vehicle roadway environment sensing method

Abstract

The present application belongs to the application field of coal mine unmanned technology, and provides a coal mine unmanned vehicle roadway environment sensing method to solve the instability problem of multi-sensor multi-modal fusion sensing in the coal mine environment. By fusing the point cloud data of the laser detection system and the image data of the image acquisition device installed at both ends of the vehicle end and the roadway end, the limitations of single sensor in detecting obstacles during the process of unmanned driving in the coal mine are solved. At the same time, the detection results of the vehicle end and the detection results of the roadway end are fused at the decision layer to realize the information exchange between the vehicle end and the roadway end, expand the sensing range during the vehicle driving process, effectively improve the comprehensiveness of the target detection of the coal mine unmanned vehicle and the safety of the driving, and further align the data under different modes based on the bird's eye view space, optimize the fusion sensing algorithm, and improve the accuracy of target detection in the coal mine unmanned vehicle technology.

Description

TECHNICAL FIELD

[0001] The present application belongs to the application field of coal mine underground unmanned technology, and particularly relates to a coal mine underground unmanned vehicle roadway environment perception method. BACKGROUND

[0002] The coal mine underground is large in dust water mist and poor in illumination condition, and the vehicle needs to autonomously perceive the surrounding environment to realize unmanned driving. The accuracy of target detection around the vehicle is a prerequisite for realizing the environment perception of unmanned driving and is also a key to emergency obstacle avoidance and safe driving in the vehicle driving process.

[0003] At present, the sensors used by the coal mine underground vehicle unmanned perception system include a laser detection system, a millimeter wave radar and a camera. The special environment of the coal mine underground makes it difficult for a single sensor to guarantee the accuracy of the target detection result. The multi-modal fusion perception based on multiple sensors is a mainstream solution at present, but the relatively complex structure of the fusion algorithm is an unstable factor in actual application, and the optimization of the algorithm model is urgently needed to be solved. SUMMARY

[0004] The present application provides a coal mine underground unmanned vehicle roadway environment perception method to solve at least one of the above technical problems in the prior art.

[0005] The present application adopts the following technical solution: a coal mine underground unmanned vehicle roadway environment perception method, comprising the following steps:

[0006] S1: acquiring point cloud data of a vehicle end and a roadway end collected by a laser detection system and graphic data of the vehicle end and the roadway end collected by an image acquisition device;

[0007] S2: encoding the point cloud data of the same end to obtain regular point cloud data, then performing attribute extraction based on the regular point cloud data to obtain voxel features of the point cloud data, compressing the voxel features to obtain point cloud attributes of the point cloud data in a bird's eye view space;

[0008] S3: processing the graphic data of the same end to obtain primary graphic attributes and corresponding depth attributes, and finally combining the primary graphic attributes and the depth attributes to obtain a three-dimensional view cone of the graphic data, and processing the three-dimensional view cone to obtain graphic attributes of the graphic data in the bird's eye view space;

[0009] S4: concatenating the point cloud attributes and the graphic attributes of the same end based on the spatial dimension of the bird's eye view space, and obtaining a target detection result after processing; the detection result includes a target detection result of the vehicle end and a target detection result of the roadway end;

[0010] S5: Fuse the target detection result of the vehicle end and the target detection result of the tunnel end, perform obstacle detection according to the fused target detection result, and obtain obstacle information according to the detection result.

[0011] Preferably, before step S2 encodes the point cloud data of the same end, it further comprises:

[0012] The mean filtering method is adopted to perform denoising preprocessing on the point cloud data.

[0013] Preferably, in step S2, the attribute of the regular point cloud data is extracted to obtain the voxel feature of the point cloud data, comprising:

[0014] The regular point cloud data is processed into voxel form by using a sparse convolution layer to obtain three-dimensional voxel information;

[0015] The voxel feature in the three-dimensional voxel information is extracted by using a sparse 3D convolution network to obtain the voxel feature of the point cloud data.

[0016] Preferably, after the graphic data of the same end is processed in step S3, the primary graphic attribute and the corresponding depth attribute are obtained, comprising:

[0017] A ResNet101+FPN attribute extraction network is constructed to extract image information of different levels in the graphic data, and the image information of different levels is fused to obtain the primary graphic attribute;

[0018] A depth prediction network is constructed to extract multi-scale image information in the graphic data, and a static weight calculation is used to calculate the depth attribute corresponding to the multi-scale image information.

[0019] Preferably, in step S3, the three-dimensional frustum is processed to obtain the graphic attribute of the graphic data in the bird's eye view space, comprising:

[0020] The mapping relationship between the three-dimensional frustum space and the voxel space is established to obtain the graphic voxel network attribute of the three-dimensional frustum in the point cloud coordinate system;

[0021] In the three-dimensional frustum space, the three-dimensional frustum is compressed along the z-axis direction to obtain the graphic attribute of the graphic data in the bird's eye view space.

[0022] Preferably, after processing in step S4, the target detection result is obtained, comprising:

[0023] The concatenated bird's eye view is processed by a region proposal network to generate an obstacle target candidate box;

[0024] Based on the concatenated bird's eye view, key point information is extracted, the extracted key point information is transmitted into the obstacle target candidate box, and obstacle detection is performed to obtain the target detection result.

[0025] Preferably, further comprising:

[0026] The obstacle information is fed back to the unmanned vehicle in the coal mine, and the unmanned vehicle in the coal mine is controlled to adjust the driving strategy according to the feedback information.

[0027] Compared with the prior art, the present application has the following advantages:

[0028] The coal mine unmanned vehicle roadway environment perception method provided by the present application fuses the point cloud data of the laser detection system and the image data of the image acquisition device installed at the two ends of the vehicle and the roadway, solves the limitations of single sensor in obstacle detection during the unmanned driving in the coal mine, and fuses the detection results of the vehicle end and the detection results of the roadway end at the decision layer, realizes the information exchange between the vehicle end and the roadway end, expands the perception range during vehicle driving, effectively improves the comprehensiveness of target detection and the safety of driving of the unmanned vehicle in the coal mine, further aligns the data under different modalities based on the bird's eye view space, optimizes the fusion perception algorithm, improves the accuracy of target detection in the coal mine unmanned vehicle driving technology, so that the unmanned vehicle can more accurately perform corresponding operations after receiving the obstacle detection result feedback, and improves the driving safety of the unmanned vehicle in the coal mine. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0030] Figure 1 is a coal mine unmanned vehicle roadway environment perception method system flowchart provided by an embodiment of the present application;

[0031] Figure 2 is a flowchart of converting point cloud data to bird's eye view attributes provided by an embodiment of the present application;

[0032] Figure 3 is a flowchart of converting point cloud data to bird's eye view attributes provided by an embodiment of the present application;

[0033] Figure 4 is a fusion flowchart of target detection results at the same end provided by an embodiment of the present application;

[0034] Figure 5 is a fusion flowchart of target detection results at the same end provided by an embodiment of the present application;

[0035] Figure 6 This is a schematic diagram of the division of underground vehicle track areas in a coal mine, provided in an embodiment of the present invention.

[0036] Figure 7 This is the corresponding driving strategy for underground coal mine vehicles based on target detection results, provided in the embodiments of the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] It should be noted that the structures, proportions, sizes, etc., shown in the accompanying drawings of this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed in the specification, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportional relationships, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should fall within the scope of the technical content disclosed in the present invention. It should be noted that in this specification, relational terms such as "first" and "second" are only used to distinguish one entity from several other entities, and do not necessarily require or imply any actual relationship or order between these entities.

[0039] Point cloud data is sparse, resulting in poor detection of distant, small, or shape-incomplete targets. Visual technologies have poor adaptability to the underground coal mine environment, and obstacle detection is significantly affected by lighting and dust. Relying on a single sensor is insufficient to meet the accuracy requirements of target detection in unmanned vehicle perception systems in underground coal mines. Furthermore, relying solely on laser detection systems or image acquisition sensors installed on the vehicle end is limited by installation height, restricting the ability to detect distant obstacles. Current multimodal detection models are complex, computationally expensive, and difficult to train and tune. Therefore, this invention provides a method for perceiving the tunnel environment of unmanned vehicles in underground coal mines, fusing point cloud data and image data, and integrating detection results from both the vehicle end and the tunnel end for comprehensive obstacle detection. This effectively improves the comprehensiveness of target detection and driving safety for unmanned vehicles in underground coal mines.

[0040] like Figure 1 The flowchart shown illustrates a method for environmental perception in underground coal mine roadways for unmanned vehicles, comprising the following steps:

[0041] S1: Obtain point cloud data of the vehicle end and the roadway end collected by a laser detection system and image data of the vehicle end and the roadway end collected by an image acquisition device.

[0042] In this embodiment, the laser detection system and the image acquisition device are installed on the underground unmanned vehicle at the vehicle end, and a plurality of laser detection systems and image acquisition devices are installed around the driving lane of the underground unmanned vehicle at the roadway end. The point cloud data and the image data of the vehicle end and the roadway end are collected simultaneously, and the point cloud data and the image data of the vehicle end and the roadway end are processed respectively. The image acquisition device of the present application is a camera, and the present application is not limited thereto.

[0043] S2: Encode the point cloud data of the same end to obtain regular point cloud data, then perform attribute extraction based on the regular point cloud data to obtain voxel features of the point cloud data, compress the voxel features to obtain point cloud attributes of the point cloud data in the bird's eye view space.

[0044] Optionally, before the step S2, the point cloud data is further preprocessed by mean filtering to remove noise.

[0045] In this embodiment, due to the complexity of the underground environment of the coal mine, the point cloud data collected by the laser detection system inevitably contains a certain degree of noise. Therefore, the mean filtering algorithm is used to pre-process the point cloud data to improve the quality of the point cloud data. Other noise reduction methods can also be used, and the present application is not limited thereto.

[0046] Optionally, the attribute extraction based on the regular point cloud data to obtain the voxel features of the point cloud data comprises: processing the regular point cloud data into voxel form by using a sparse convolution layer to obtain three-dimensional voxel information; and extracting voxel features in the three-dimensional voxel information by using a sparse 3D convolution network to obtain the voxel features of the point cloud data.

[0047] In this embodiment, as shown in Figure 2 Fig. 1 shows a flowchart of converting point cloud data into bird's eye view attributes. The unordered massive point cloud data is encoded into regular point cloud data by a voxel encoder, and then processed and converted into voxel form by using a sparse convolution layer to obtain three-dimensional voxel information with target obstacle information, thereby reducing the complexity of the point cloud data in the vertical direction. After the point cloud data is converted into voxel form, a sparse 3D convolution network is used to capture local spatial structure and voxel features, thereby obtaining voxel features of the point cloud data. Finally, after the voxel features are compressed, the point cloud BEV (Bird's Eye View) features in the bird's eye view space are obtained.

[0048] S3: processing the graphic data of the same end to obtain primary graphic attributes and corresponding depth attributes, combining the primary graphic attributes and the depth attributes to obtain a three-dimensional view cone of the graphic data, and processing the three-dimensional view cone to obtain graphic attributes of the graphic data in the bird's eye view space.

[0049] Optionally, the graphic data of the same end is input into a preset depth prediction network after processing to obtain multi-scale primary graphic attributes and corresponding depth attributes, including: constructing a ResNet101+FPN attribute extraction network to extract image information of different levels in the graphic data, and fusing the image information of different levels to obtain the primary graphic attributes; constructing a depth prediction network to extract multi-scale image information in the graphic data, and calculating the depth attributes corresponding to the multi-scale image information by using static weights.

[0050] Optionally, the three-dimensional view cone is processed to obtain the graphic attributes of the graphic data in the bird's eye view space, including: establishing a mapping relationship between the three-dimensional view cone space and the voxel space to obtain a graphic voxel network attribute of the three-dimensional view cone in a point cloud coordinate system; and compressing the three-dimensional view cone along the z-axis direction in the three-dimensional view cone space to obtain the graphic attributes of the graphic data in the bird's eye view space.

[0051] In this embodiment, as shown in Figure 3 The flowchart for converting the graphic data to the bird's eye view features is shown in FIG. 1. The collected graphic data is subjected to feature extraction by using a ResNet101 attribute extraction network, and an FPN attribute pyramid structure is introduced into the ResNet101 attribute extraction network to construct a ResNet101+FPN attribute extraction network. Different levels of features are fused to obtain multi-scale features with rich image information. After the image data is subjected to the ResNet101+FPN attribute extraction network, the primary graphic attributes are obtained.

[0052] In this embodiment, the depth prediction network is composed of two 3x3 convolution layers Conv and one 1x1 convolution layer Conv. The first two 3x3 convolution layers Conv are mainly used to extract multi-scale image information, and the depth attributes corresponding to the image information are obtained by combining the image information through static weights. The 1x1 convolution layer Conv can obtain a predicted discrete depth map, that is, the distance between the obstacle and the unmanned vehicle in the underground lane driving scene.

[0053] The depth information and the graphic attribute of each pixel point included in the graphic data are combined, and the stereoscopic vision information of the corresponding graphic data is obtained according to the graphic attribute and the depth information of each pixel point, which is referred to as a three-dimensional view cone attribute. Further, in order to unify the bird's eye view attribute corresponding to the graphic data and the point cloud data in the bird's eye view space to the same coordinate system, a mapping relationship between the three-dimensional view cone space and the voxel space is established, the three-dimensional view cone attribute is converted into a voxel feature in the point cloud coordinate system according to the mapping relationship, and finally the graphic attribute of the graphic data in the bird's eye view space is obtained by compressing along the Z-axis direction in the three-dimensional space.

[0054] S4: concatenating the point cloud attribute and the graphic attribute of the same end based on the spatial dimension of the bird's eye view space, and obtaining a target detection result after processing; the detection result includes a target detection result of the vehicle end and a target detection result of the tunnel end.

[0055] Optionally, the target detection result obtained after processing includes: processing the concatenated bird's eye view through a region proposal network to generate an obstacle target candidate box; extracting key point information based on the concatenated bird's eye view, transmitting the extracted key point information to the obstacle target candidate box, and performing obstacle detection to obtain the target detection result.

[0056] In this embodiment, as shown in the fusion flowchart of the target detection result, Figure 4 concatenating the point cloud attribute and the graphic attribute of the same end based on the spatial dimension of the bird's eye view space, and obtaining a target detection result after processing; the detection result includes a target detection result of the vehicle end and a target detection result of the tunnel end.

[0057] In this embodiment, in order to enable the bird's eye view attribute to learn rich information, a channel attention mechanism and a spatial attention mechanism are introduced to amplify important information of local targets of the bird's eye view attribute. Then, the region proposal network is used to process the optimized bird's eye view attribute to generate a target candidate box for obstacle target detection.

[0058] In this embodiment, in order to enhance the understanding of the network to the space, the key point self-attention module of the point cloud is used to extract the characteristics of the key points in the point cloud (including the bird's eye view space coordinate information of the key points and the corresponding bird's eye view attribute information), the extracted key point information is transmitted to the target candidate box generated by the region proposal network, the attribute of the more accurate target candidate box is obtained, and finally the key point information in the target candidate box is input into the target detection head to obtain the detection result of the obstacle target (including the obstacle target category, size, pose parameter, etc.).

[0059] In this embodiment, the point cloud key point self-attention module is a multi-layer perceptron composed of two linear layers and an activation layer. The query matrix, key matrix and value matrix are constructed through linear transformation. The corresponding feature vector can be calculated by combining the position information of each point in the point cloud data, so as to obtain the characteristics of the point cloud key points in the point cloud data.

[0060] S5: The target detection results of the vehicle end and the roadway end are fused, obstacle detection is performed according to the fused target detection results, and obstacle information is obtained according to the detection results.

[0061] In this embodiment, as shown in Figure 5 The detection result fusion of the vehicle end and the roadway end is equivalent to decision layer fusion. The target detection of the roadway end mainly detects the obstacle situation far away from the unmanned vehicle during the driving process in the underground roadway. The target detection of the vehicle end mainly identifies small obstacles around the unmanned vehicle that cannot be detected by the roadway end. The optimal global decision is obtained by complementing each other between the results of the vehicle end and the roadway end, so as to realize the overall identification of the obstacles around the underground vehicle.

[0062] Optionally, the method further comprises: feeding back the obstacle information to the underground unmanned vehicle, and controlling the underground unmanned vehicle to adjust the driving strategy according to the feedback information.

[0063] In this embodiment, the final fusion detection result of the vehicle end and the roadway end is fed back to the underground unmanned vehicle. The unmanned vehicle performs corresponding operations according to the feedback result. The specific driving strategy is as shown in Figure 6 、 Figure 7 When the target is a pedestrian, the distance between the pedestrian and the unmanned vehicle is judged. When the distance is less than D1, region a belongs to the warning area, and the unmanned vehicle executes the driving strategy of honking and stopping. When the distance is greater than D1 and less than D2, region b belongs to the pre-warning area, and the unmanned vehicle executes the driving strategy of honking and slowing down. When the distance is greater than D2 and less than D3, region c belongs to the safety area, and the unmanned vehicle executes the driving strategy of continuing to drive at a constant speed. When the target is an object, the distance between the object and the unmanned vehicle is judged. When the distance is less than D1, region a belongs to the warning area, and the unmanned vehicle executes the driving strategy of changing lane. When the distance is greater than D1 and less than D2, region b belongs to the pre-warning area, and the unmanned vehicle executes the driving strategy of slowing down. When the distance is greater than D2 and less than D3, region c belongs to the safety area, and the unmanned vehicle executes the driving strategy of continuing to drive at a constant speed.

[0064] The coal mine underground unmanned vehicle roadway environment perception method provided by the present application fuses point set data of a laser detection system and graphic data of a camera, solves the limitation of a single sensor in target detection, and fuses the detection results of the vehicle end sensor and the detection results of the roadway end sensor at the decision layer, realizes information exchange between the vehicle end and the roadway end, expands the perception range in the vehicle driving process, effectively improves the comprehensiveness of the coal mine underground vehicle unmanned target detection and the safety of driving, further unifies the bird's eye view attribute to solve the data alignment problem of different modalities, optimizes the fusion perception algorithm, improves the accuracy of target detection in the coal mine underground vehicle unmanned technology, and finally feeds back the target detection result to the vehicle, so that the unmanned vehicle performs corresponding operations according to the predetermined driving strategy, and improves the driving safety of the coal mine underground vehicle.

[0065] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A coal mine underground unmanned vehicle roadway environment sensing method, characterized in that, The method comprises the following steps: S1: acquiring point cloud data of a vehicle end and a tunnel end collected by a laser detection system and image data of the vehicle end and the tunnel end collected by an image acquisition device; S2: encoding the point cloud data of the same end to obtain regular point cloud data, then performing attribute extraction based on the regular point cloud data to obtain voxel features of the point cloud data, compressing the voxel features to obtain point cloud attributes of the point cloud data in a bird's eye view space; The attribute extraction based on the regular point cloud data to obtain the voxel features of the point cloud data comprises: processing the regular point cloud data into voxel form by using a sparse convolution layer to obtain three-dimensional voxel information; extracting voxel features in the three-dimensional voxel information by using a sparse 3D convolution network to obtain the voxel features of the point cloud data; S3: processing the image data of the same end to obtain primary image attributes and corresponding depth attributes, and finally combining the primary image attributes and the depth attributes to obtain a three-dimensional frustum of the image data, and processing the three-dimensional frustum to obtain image attributes of the image data in the bird's eye view space; The processing of the image data of the same end to obtain the primary image attributes and the corresponding depth attributes comprises: constructing a ResNet101+FPN attribute extraction network to extract image information of different levels in the image data, and fusing the image information of different levels to obtain the primary image attributes; constructing a depth prediction network to extract multi-scale image information in the image data, and calculating depth attributes corresponding to the multi-scale image information by using static weights; The processing of the three-dimensional frustum to obtain the image attributes of the image data in the bird's eye view space comprises: establishing a mapping relationship between a three-dimensional frustum space and a voxel space to obtain image voxel network attributes of the three-dimensional frustum in a point cloud coordinate system; compressing the three-dimensional frustum along a z-axis direction in the three-dimensional frustum space to obtain the image attributes of the image data in the bird's eye view space; S4: concatenating the point cloud attributes and the image attributes of the same end based on a spatial dimension of the bird's eye view space, and obtaining a target detection result after processing; the detection result comprises a target detection result of the vehicle end and a target detection result of the tunnel end; The obtaining of the target detection result after processing comprises: processing the concatenated bird's eye view by using a region proposal network to generate an obstacle target candidate frame; extracting key point information based on the concatenated bird's eye view, transmitting the extracted key point information to the obstacle target candidate frame, and performing obstacle detection to obtain the target detection result; S5: fusing the target detection result of the vehicle end and the target detection result of the tunnel end, performing obstacle detection based on the fused target detection result, and obtaining obstacle information according to the detection result; wherein, the target detection of the tunnel end mainly detects the obstacle situation at a distance far away from the unmanned vehicle during the driving process in the underground lane, and the target detection of the vehicle end identifies small obstacles around the unmanned vehicle that cannot be detected by the tunnel end, and the global optimal decision is obtained by complementing the results of the vehicle end and the tunnel end.

2. The coal mine underground unmanned vehicle roadway environment perception method according to claim 1, characterized in that, Before the step S2, the method further comprises: The point cloud data is preprocessed by mean filtering method.

3. The coal mine underground unmanned vehicle roadway environment perception method according to claim 1, characterized in that, The method further comprises: The obstacle information is fed back to the unmanned vehicle in the well, and the driving strategy of the unmanned vehicle in the well is adjusted according to the feedback information.

Images