Underground coal mine unmanned vehicle roadway environment sensing method

Abstract

The invention belongs to the field of application of an underground coal mine driverless technology, and provides a roadway environment sensing method for an underground coal mine driverless vehicle in order to solve the problem that multi-modal fusion sensing of current multiple sensors is unstable in an underground coal mine environment. Point cloud data of laser detection systems installed at a vehicle end and a roadway end and graphic data of an image acquisition device are fused, the limitation of a single sensor on obstacle detection in the underground unmanned driving process is solved, meanwhile, a detection result of the vehicle end and a detection result of the roadway end are fused on a decision-making level, and the detection accuracy of the vehicle end and the roadway end is improved. According to the method, information exchange between a vehicle end and a roadway end is realized, the sensing range in the vehicle driving process is expanded, the comprehensiveness of unmanned target detection of the underground coal mine vehicle and the driving safety are effectively improved, further, data in different modes are aligned based on aerial view space, a fusion sensing algorithm is optimized, and the accuracy of target detection is improved. And the accuracy of target detection in the coal mine parallel running vehicle unmanned driving technology is improved.

Description

Technical Field

[0001] The present invention belongs to the field of application of unmanned driving technology in underground coal mines, and specifically relates to a method for sensing the tunnel environment of an unmanned driving vehicle in underground coal mines. Background Art

[0002] There is a lot of dust and mist in coal mines and the lighting conditions are poor. In order to achieve unmanned driving, the vehicle must first be able to autonomously perceive the surrounding environment. The accuracy of target detection around the vehicle is a prerequisite for achieving unmanned driving environment perception, and it is also the key to emergency obstacle avoidance and safe driving during vehicle movement.

[0003] At present, the sensors used in the unmanned driving perception system of coal mine underground vehicles include laser detection systems, millimeter wave radars and cameras. Due to the special environment of coal mine underground, it is difficult for a single sensor to guarantee the accuracy of target detection results. Multi-modal fusion perception based on multiple sensors is the current mainstream solution, but the relatively complex structure of the fusion algorithm is an unstable factor in practical applications, and the optimization algorithm model needs to be solved urgently. Summary of the invention

[0004] In order to solve at least one of the above-mentioned technical problems existing in the prior art, the present invention provides a method for sensing the tunnel environment of an unmanned vehicle in an underground coal mine.

[0005] The present invention is implemented by the following technical solution: A method for sensing the tunnel environment of an unmanned vehicle in a coal mine, comprising the following steps:

[0006] S1: Acquire the point cloud data of the vehicle end and the lane end collected by the laser detection system and the graphic data of the vehicle end and the lane end collected by the image acquisition device;

[0007] S2: Encoding the point cloud data at the same end to obtain regular point cloud data, then extracting attributes based on the regular point cloud data to obtain voxel features of the point cloud data, compressing the voxel features to obtain point cloud attributes of the point cloud data in the bird's-eye view space;

[0008] S3: After processing the graphic data at the same end, primary graphic attributes and corresponding depth attributes are obtained, and finally the primary graphic attributes and the depth attributes are combined to obtain a three-dimensional cone of the graphic data, and the three-dimensional cone is processed to obtain graphic attributes of the graphic data in a bird's-eye view space;

[0009] S4: The point cloud attributes and the graphic attributes at the same end are connected in series based on the spatial dimension of the bird's-eye view space, and a target detection result is obtained after processing; the detection result includes the target detection result at the vehicle end and the target detection result at the lane end;

[0010] S5: Fusing the target detection results at the vehicle end and the target detection results at the lane end, performing obstacle detection based on the fused target detection results, and obtaining obstacle information based on the detection results.

[0011] Preferably, before step S2 encodes the point cloud data at the same end, it also includes:

[0012] The point cloud data is preprocessed by using a mean filtering method to remove noise.

[0013] Preferably, in step S2, the attribute extraction is performed on the rule-based point cloud data to obtain the voxel features of the point cloud data, including:

[0014] Use sparse convolutional layers to process regular point cloud data into voxel form to obtain three-dimensional voxel information;

[0015] The voxel features in the three-dimensional voxel information are extracted by using a sparse 3D convolutional network to obtain the voxel features of the point cloud data.

[0016] Preferably, after processing the graphic data of the same end in step S3, primary graphic attributes and corresponding depth attributes are obtained, including:

[0017] Constructing a ResNet101+FPN attribute extraction network to extract image information of different levels in the graphic data, and fusing the image information of different levels to obtain primary graphic attributes;

[0018] A depth prediction network is constructed to extract multi-scale image information from the graphic data, and depth attributes corresponding to the multi-scale image information are calculated using static weights.

[0019] Preferably, in step S3, the three-dimensional view cone is processed to obtain graphic attributes of the graphic data in the bird's-eye view space, including:

[0020] Establish the mapping relationship between the 3D view cone space and the voxel space, and obtain the graphic voxel network properties of the 3D view cone in the point cloud coordinate system;

[0021] In the three-dimensional visual cone space, the three-dimensional visual cone is compressed along the z-axis direction to obtain graphic attributes of the graphic data in the bird's-eye view space.

[0022] Preferably, the target detection result obtained after processing in step S4 includes:

[0023] The concatenated bird's-eye view image is processed through the region proposal network to generate candidate boxes for obstacle targets.

[0024] Key point information is extracted based on the concatenated bird's-eye view, the extracted key point information is transmitted to the obstacle target candidate frame, and obstacle detection is performed on it to obtain a target detection result.

[0025] Preferably, it also includes:

[0026] The obstacle information is fed back to the underground unmanned vehicle, and the underground unmanned vehicle is controlled to adjust the driving strategy according to the feedback information.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention provides a method for sensing the tunnel environment of an unmanned vehicle in an underground coal mine. By fusing the point cloud data of the laser detection system installed at both ends of the vehicle and the tunnel and the graphic data of the image acquisition device, the limitation of obstacle detection by a single sensor in the process of unmanned driving in the underground coal mine is solved. At the same time, the detection result of the vehicle end is fused with the detection result of the tunnel end at the decision-making layer, so as to realize the information exchange between the vehicle end and the tunnel end, expand the perception range during vehicle driving, and effectively improve the comprehensiveness of target detection and driving safety of unmanned vehicles in underground coal mines. Furthermore, based on the bird's-eye view space, the data under different modes are aligned, the fusion perception algorithm is optimized, and the accuracy of target detection in the unmanned driving technology of vehicles in underground coal mines is improved, so that the unmanned vehicle can perform corresponding operations more accurately after receiving the feedback of the obstacle detection result, thereby improving the driving safety of unmanned vehicles in underground coal mines. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 This is a schematic diagram of a system flow of a method for sensing a tunnel environment of an unmanned vehicle in an underground coal mine provided by an embodiment of the present invention;

[0031] Figure 2 It is a schematic diagram of a process of converting point cloud data into bird's-eye view attributes provided by an embodiment of the present invention;

[0032] Figure 3 is a schematic diagram of a process for converting graphic data into bird's-eye view attributes provided by an embodiment of the present invention;

[0033] Figure 4 is a fusion flow chart of target detection results on the same end provided by an embodiment of the present invention;

[0034] Figure 5 is a flow chart of the fusion of vehicle-side and lane-side target detection results provided by an embodiment of the present invention;

[0035] Figure 6 It is a schematic diagram of the division of vehicle track areas in a coal mine provided by an embodiment of the present invention;

[0036] Figure 7 It is a corresponding driving strategy for a coal mine underground vehicle according to a target detection result provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In conjunction with the drawings in the embodiments of the present invention, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other implementation methods obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0038] It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification so that people familiar with this technology can understand and read them. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in proportional relationship or adjustment of size should fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention. It should be noted that in this specification, relational terms such as first and second are only used to distinguish one entity from several other entities, and do not necessarily require or imply any actual relationship or order between these entities.

[0039] Point cloud data is sparse, and the results are poor when detecting distant, small or shape-missing targets. Visual technology has poor adaptability in the application environment of coal mines. Obstacle target detection is greatly affected by light and dust. It is difficult to meet the accuracy of target detection in the unmanned driving perception system of coal mine vehicles with only a single sensor; at the same time, only a laser detection system or image acquisition sensor is installed on the vehicle side for perception, and the ability to detect long-distance obstacles is limited by the installation height; the current multimodal detection model is relatively complex, the calculation cost is high, and the training process is difficult to adjust parameters. Therefore, the present invention provides a method for sensing the environment of a coal mine unmanned vehicle lane, which realizes the fusion of point cloud data and graphic data, and can also fuse the detection results of the vehicle side and the lane side, and perform obstacle detection in all aspects, effectively improving the comprehensiveness of unmanned driving target detection and driving safety of coal mine vehicles.

[0040] like Figure 1 As shown, a flow chart of a method for sensing the tunnel environment of an unmanned vehicle in a coal mine is provided, including the following steps:

[0041] S1: Acquire the point cloud data of the vehicle end and the lane end collected by the laser detection system and the graphic data of the vehicle end and the lane end collected by the image acquisition device.

[0042] In this embodiment, a laser detection system and an image acquisition device are installed at the underground unmanned vehicle end, and several laser detection systems and image acquisition devices are installed around the driving lane of the underground unmanned vehicle at the tunnel end, and point cloud data and graphic data of the vehicle end and the tunnel end are collected at the same time, and the point cloud data and graphic data of the vehicle end and the tunnel end are processed separately. The image acquisition device of the present application is a camera, but the present invention is not limited to this.

[0043] S2: Encode the point cloud data at the same end to obtain regular point cloud data, then extract attributes based on the regular point cloud data to obtain voxel features of the point cloud data, compress the voxel features to obtain point cloud attributes of the point cloud data in the bird's-eye view space.

[0044] Optionally, before step S2 encodes the point cloud data at the same end, it also includes: using a mean filtering method to perform denoising preprocessing on the point cloud data.

[0045] In this embodiment, due to the complexity of the underground environment of the coal mine, the point cloud data collected by the laser detection system will inevitably have a certain degree of noise. Therefore, this application uses a mean filtering algorithm to perform denoising preprocessing on the point cloud data to improve the quality of the point cloud data. Other noise reduction methods may also be used, and the present invention is not limited to this.

[0046] Optionally, attribute extraction is performed based on regular point cloud data to obtain voxel features of the point cloud data, including: using a sparse convolutional layer to process the regular point cloud data into a voxel form to obtain three-dimensional voxel information; using a sparse 3D convolutional network to extract voxel features in the three-dimensional voxel information to obtain voxel features of the point cloud data.

[0047] In this embodiment, Figure 2 The figure shows a flow chart of the conversion of point cloud data into bird's-eye view attributes. The disordered massive point cloud data is encoded into regular point cloud data through a voxel encoder, and then converted into voxel form using a sparse convolutional layer to obtain three-dimensional voxel information with target obstacle information, thereby reducing the complexity of the point cloud data in the vertical direction. After the point cloud data is converted into voxel form, a sparse 3D convolutional network is used to capture the local spatial structure and voxel features, thereby obtaining the voxel features of the point cloud data. Finally, the voxel features are compressed to obtain the point cloud BEV (Bird's Eye View) features in the bird's-eye view space.

[0048] S3: After processing the graphic data at the same end, primary graphic attributes and corresponding depth attributes are obtained, and finally the primary graphic attributes and the depth attributes are combined to obtain a three-dimensional view cone of the graphic data, and then the three-dimensional view cone is processed to obtain graphic attributes of the graphic data in the bird's-eye view space.

[0049] Optionally, the graphic data at the same end is processed and input into a preset depth prediction network to obtain multi-scale primary graphic attributes and corresponding depth attributes, including: constructing a ResNet101+FPN attribute extraction network to extract image information of different levels in the graphic data, and fusing the image information of different levels to obtain primary graphic attributes; constructing a depth prediction network to extract multi-scale image information in the graphic data, and using static weights to calculate the depth attributes corresponding to the multi-scale image information.

[0050] Optionally, the three-dimensional view cone is processed to obtain graphic properties of the graphic data in the bird's-eye view space, including: establishing a mapping relationship between the three-dimensional view cone space and the voxel space to obtain the graphic voxel network properties of the three-dimensional view cone in the point cloud coordinate system; in the three-dimensional view cone space, compressing the three-dimensional view cone along the z-axis direction to obtain the graphic properties of the graphic data in the bird's-eye view space.

[0051] In this embodiment, Figure 3 The figure shows a process diagram for converting graphic data into bird's-eye view features. Feature extraction is performed on the collected graphic data. The ResNet101 attribute extraction network is used, and the FPN attribute pyramid structure is introduced into it to construct a ResNet101+FPN attribute extraction network. Features at different levels are integrated to obtain multi-scale features with rich image information. After the image data passes through the ResNet101+FPN attribute extraction network, primary graphic attributes are obtained.

[0052] In this embodiment, the depth prediction network is composed of two 3×3 convolutional layers Conv and one 1×1 convolutional layer Conv. The first two 3×3 convolutional layers Conv are mainly used to extract multi-scale image information, and combine the image information to obtain the depth attributes of the corresponding image information through static weights. The 1×1 convolutional layer Conv can obtain the predicted discrete depth map, that is, the distance between the obstacle and the unmanned vehicle in the underground lane driving scene.

[0053] Combined with the depth information and graphic attributes of each pixel contained in the graphic data, the stereoscopic visual information of the corresponding graphic data is obtained according to the graphic attributes and depth information of each pixel, which is called the three-dimensional cone attribute. Furthermore, in order to unify the bird's-eye view attributes corresponding to the graphic data and the point cloud data into the same coordinate system in the bird's-eye view space, a mapping relationship between the three-dimensional cone space and the voxel space is established. According to the mapping relationship, the three-dimensional cone attributes are converted into voxel features in the point cloud coordinate system, and finally the graphic attributes of the graphic data in the bird's-eye view space are obtained by compressing along the Z axis in the three-dimensional space.

[0054] S4: The point cloud attributes and the graphic attributes at the same end are connected in series based on the spatial dimension of the bird's-eye view space, and the target detection results are obtained after processing; the detection results include the target detection results at the vehicle end and the target detection results at the lane end.

[0055] Optionally, the target detection result is obtained after processing, including: processing the concatenated bird's-eye view through a region proposal network to generate an obstacle target candidate frame; extracting key point information based on the concatenated bird's-eye view, transmitting the extracted key point information to the obstacle target candidate frame, and performing obstacle detection on it to obtain the target detection result.

[0056] In this embodiment, Figure 4 The figure shows the fusion flow chart of the target detection results. The point cloud attributes and graphic attributes of the vehicle side and the lane side are concatenated based on the bird's-eye view space, and fused using a 3×3 convolution to obtain the fused bird's-eye view attributes of the vehicle side and the lane side.

[0057] In this embodiment, in order to learn rich information from the bird's-eye view attributes, the channel attention mechanism and the spatial attention mechanism are introduced to amplify the important information of the local target of the bird's-eye view attributes. Then, the region proposal network is used to process the optimized bird's-eye view attributes to generate target candidate boxes for obstacle target detection.

[0058] In this embodiment, in order to enhance the network's understanding of space, the point cloud key point self-attention module is used to extract the characteristics of the key points in the point cloud (including the bird's-eye view spatial coordinate information of the key points and the corresponding bird's-eye view attribute information), and the extracted key point information is transmitted to the target candidate box generated by the region proposal network to obtain more accurate attributes of the target candidate box. Finally, the key point information in the target candidate box is input into the target detection head to obtain the detection results of the obstacle target (including obstacle target category, size, posture parameters, etc.).

[0059] In this embodiment, the point cloud key point self-attention module is composed of a multilayer perceptron consisting of two linear layers and one activation layer. The query matrix, key matrix and value matrix are constructed through linear transformation. Combined with the position information of each point in the point cloud data, the corresponding feature vector can be calculated to obtain the characteristics of the point cloud key points in the point cloud data.

[0060] S5: Fusing the target detection results at the vehicle end and the target detection results at the lane end, performing obstacle detection based on the fused target detection results, and obtaining obstacle information based on the detection results.

[0061] In this embodiment, Figure 5 The figure shows the fusion flow chart of the target detection results at the vehicle end and the tunnel end. The fusion of the detection results at the vehicle end and the tunnel end is equivalent to the fusion of the decision layer. The target detection at the tunnel end mainly detects obstacles that are far away from the unmanned vehicle during driving in the underground lane. The target detection at the vehicle end is mainly to identify smaller obstacles that cannot be detected at the tunnel end around the unmanned vehicle. The results of the vehicle end and the tunnel end complement each other to obtain the global optimal decision, thereby realizing the comprehensive identification of obstacles around vehicles in coal mines.

[0062] Optionally, the method further includes: feeding back the obstacle information to an underground unmanned vehicle, and controlling the underground unmanned vehicle to adjust a driving strategy according to the feedback information.

[0063] In this embodiment, the final fusion detection results of the vehicle end and the tunnel end are fed back to the underground unmanned vehicle, and the unmanned vehicle performs corresponding operations according to the feedback results. The specific driving strategy is as follows: Figure 6 , Figure 7 As shown, firstly, according to the obstacle target category, size, posture parameters and other information of the detection result, it is determined whether the obstacle is a pedestrian or an object. When the target is a pedestrian, the distance between the pedestrian and the unmanned vehicle is determined. When the distance is less than D1, area a belongs to the warning area, and the unmanned vehicle executes the driving strategy of honking and stopping; when the distance is greater than D1 and less than D2, area b belongs to the early warning area, and the unmanned vehicle executes the driving strategy of honking and slowing down; when the distance is greater than D2 and less than D3, area c belongs to the safety area, and the unmanned vehicle executes the driving strategy of continuing to drive at a constant speed; when the target is an object, the distance between the object and the unmanned vehicle is determined. When the distance is less than D1, area a belongs to the warning area, and the unmanned vehicle executes the driving strategy of changing lanes and detouring; when the distance is greater than D1 and less than D2, area b belongs to the early warning area, and the unmanned vehicle executes the driving strategy of slowing down; when the distance is greater than D2 and less than D3, area c belongs to the safety area, and the unmanned vehicle executes the driving strategy of continuing to drive at a constant speed.

[0064] The present invention provides a method for sensing the tunnel environment of an unmanned vehicle in an underground coal mine, which fuses the point set data of a laser detection system and the graphic data of a camera, thereby solving the limitation of a single sensor in target detection; at the same time, the detection results of the vehicle-end sensor and the detection results of the tunnel-end sensor are fused at the decision-making layer, thereby realizing information exchange between the vehicle-end and the tunnel-end, expanding the perception range during vehicle driving, and effectively improving the comprehensiveness of target detection and driving safety of unmanned vehicles in underground coal mines; further, the bird's-eye view attributes are unified to solve the problem of data alignment of different modes, optimize the fusion perception algorithm, and improve the accuracy of target detection in the unmanned driving technology of vehicles in coal mines; finally, after the target detection results are fed back to the vehicle, the unmanned vehicle performs corresponding operations according to a predetermined driving strategy, thereby improving the driving safety of vehicles in underground coal mines.

[0065] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for sensing the tunnel environment of an unmanned vehicle in a coal mine, characterized in that: The steps include: S1: Acquire the point cloud data of the vehicle end and the lane end collected by the laser detection system and the graphic data of the vehicle end and the lane end collected by the image acquisition device; S2: Encoding the point cloud data at the same end to obtain regular point cloud data, then extracting attributes based on the regular point cloud data to obtain voxel features of the point cloud data, compressing the voxel features to obtain point cloud attributes of the point cloud data in the bird's-eye view space; S3: After processing the graphic data at the same end, primary graphic attributes and corresponding depth attributes are obtained, and finally the primary graphic attributes and the depth attributes are combined to obtain a three-dimensional cone of the graphic data, and the three-dimensional cone is processed to obtain graphic attributes of the graphic data in a bird's-eye view space; S4: The point cloud attributes and the graphic attributes at the same end are connected in series based on the spatial dimension of the bird's-eye view space, and a target detection result is obtained after processing; the detection result includes the target detection result at the vehicle end and the target detection result at the lane end; S5: Fusing the target detection results at the vehicle end and the target detection results at the lane end, performing obstacle detection based on the fused target detection results, and obtaining obstacle information based on the detection results.

2. The method for sensing the tunnel environment of an unmanned vehicle in a coal mine according to claim 1, characterized in that: Before step S2 encodes the point cloud data at the same end, it also includes: The point cloud data is preprocessed by using a mean filtering method to remove noise.

3. The method for sensing the tunnel environment of an unmanned vehicle in an underground coal mine according to claim 1, characterized in that: In step S2, the point cloud data is subjected to attribute extraction based on the rule to obtain voxel features of the point cloud data, including: Use sparse convolutional layers to process regular point cloud data into voxel form to obtain three-dimensional voxel information; The voxel features in the three-dimensional voxel information are extracted by using a sparse 3D convolutional network to obtain the voxel features of the point cloud data.

4. The method for sensing the tunnel environment of an unmanned vehicle in a coal mine according to claim 1, characterized in that: In step S3, after processing the graphic data of the same end, primary graphic attributes and corresponding depth attributes are obtained, including: Constructing a ResNet101+FPN attribute extraction network to extract image information of different levels in the graphic data, and fusing the image information of different levels to obtain primary graphic attributes; A depth prediction network is constructed to extract multi-scale image information from the graphic data, and depth attributes corresponding to the multi-scale image information are calculated using static weights.

5. The method for sensing the tunnel environment of an unmanned vehicle in a coal mine according to claim 1, characterized in that: In step S3, the three-dimensional view cone is processed to obtain graphic attributes of the graphic data in the bird's-eye view space, including: Establish the mapping relationship between the 3D view cone space and the voxel space, and obtain the graphic voxel network properties of the 3D view cone in the point cloud coordinate system; In the three-dimensional visual cone space, the three-dimensional visual cone is compressed along the z-axis direction to obtain graphic attributes of the graphic data in the bird's-eye view space.

6. The method for sensing the tunnel environment of an unmanned vehicle in a coal mine according to claim 1, characterized in that: The target detection result obtained after processing in step S4 includes: The concatenated bird's-eye view image is processed through the region proposal network to generate candidate boxes for obstacle targets. Key point information is extracted based on the concatenated bird's-eye view, the extracted key point information is transmitted to the obstacle target candidate frame, and obstacle detection is performed on it to obtain a target detection result.

7. The method for sensing the tunnel environment of an unmanned vehicle in a coal mine according to claim 1, characterized in that: Also includes: The obstacle information is fed back to the underground unmanned vehicle, and the underground unmanned vehicle is controlled to adjust the driving strategy according to the feedback information.

Images