A method for identifying a road surface obstacle in a mine

Abstract

The application provides a mine area road surface driving obstacle identification method, and relates to the technical field of data processing. The method comprises the following steps: acquiring a mine area road surface image and laser radar point cloud data corresponding to the mine area road surface image; identifying driving obstacles in the mine area road surface image according to a multi-task road surface detection and segmentation model; acquiring target laser radar point cloud data of the driving obstacles according to a preset calibration relationship between an image acquisition device and a laser radar, and then determining the shape and position of the driving obstacles; the above-mentioned model comprises a feature extraction structure, a feature purification structure and a multi-task prediction structure, the feature extraction structure is used for extracting features in the mine area road surface image and performing feature enhancement processing, and outputs first features, the feature purification structure is used for performing feature purification processing on the first features and outputting second features, and the multi-task prediction structure is used for performing prediction identification on the second features and outputting detection and segmentation results of the driving obstacles.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a method for identifying an object obstructing driving on a mine road. BACKGROUND

[0002] The mine environment has the characteristics of complex terrain and various obstacles. In order to ensure the safety of automatic driving of the mine car, the mine car is required to accurately identify the obstacles in the mine road.

[0003] However, the target objects such as rockfall, ruts and potholes in the mine have variable shapes, different sizes, and similar road colors, which are not easy to distinguish, making it difficult to identify them, thereby causing great safety hazards to the automatic driving of the mine car. SUMMARY

[0004] Therefore, the present application provides a method for identifying an object obstructing driving on a mine road, which realizes accurate identification of the object obstructing driving on the mine road that is not easy to detect and output of the shape and position of the object obstructing driving.

[0005] In a first aspect, the present application provides a method for identifying an object obstructing driving on a mine road, comprising: acquiring a mine road image through an image acquisition device and acquiring laser radar point cloud data corresponding to the mine road image through a laser radar; identifying the object obstructing driving in the mine road image according to a constructed multi-task road detection and segmentation model; acquiring target laser radar point cloud data corresponding to the object obstructing driving according to a preset calibration relationship between the image acquisition device and the laser radar and the laser radar point cloud data; determining the shape and position of the object obstructing driving in a three-dimensional space according to the target laser radar point cloud data; wherein the multi-task road detection and segmentation model comprises a feature extraction structure, a feature purification structure and a multi-task prediction structure, the feature extraction structure is used to extract features in the mine road image and perform feature enhancement processing on the extracted features to output first features, the feature purification structure is used to perform feature purification processing on the first features to output second features, and the multi-task prediction structure is used to perform prediction identification on the second features to output detection and segmentation results of the object obstructing driving.

[0006] According to the above method of the present application, the following additional technical features can be provided:

[0007] In the above technical solution, the object obstructing driving can include at least one of the following: rockfall, ruts and potholes.

[0008] In any of the technical solutions described above, optionally, the feature extraction structure comprises a backbone network, a first feature enhancement module, a second feature enhancement module, a first fusion module, a second fusion module, and a splicing module; wherein the backbone network generates a plurality of first layer features according to the mine area road surface image; the first feature enhancement module performs feature enhancement processing on a first deepest layer feature in the plurality of first layer features, and outputs a first enhanced feature; the first fusion module fuses the plurality of first layer features respectively using the first enhanced feature, and outputs a plurality of second layer features; the second feature enhancement module performs feature enhancement processing on a second deepest layer feature in the plurality of second layer features, and outputs a second enhanced feature; the second fusion module fuses the plurality of second layer features respectively using the second enhanced feature, and outputs a plurality of third layer features; and the splicing module splices and merges the plurality of third layer features, and outputs the first feature.

[0009] In any of the above technical solutions, optionally, the plurality of first layer features include a first sub-feature, a second sub-feature, a third sub-feature, and a fourth sub-feature, wherein the fourth sub-feature is the first deepest layer feature, the plurality of second layer features include a fifth sub-feature, a sixth sub-feature, and a seventh sub-feature, wherein the fifth sub-feature is the second deepest layer feature, and the plurality of third layer features include an eighth sub-feature, a ninth sub-feature, and a tenth sub-feature; the feature extraction structure further includes a plurality of convolution modules, the first fusion module includes a first sub-fusion module, a second sub-fusion module, and a third sub-fusion module, and the second fusion module includes a fourth sub-fusion module, a fifth sub-fusion module, and a sixth sub-fusion module; wherein the first sub-feature, the second sub-feature, the third sub-feature, and the fourth sub-feature are respectively subjected to convolution processing by a convolution module, the first feature enhancement module performs feature enhancement processing on the fourth sub-feature after convolution processing, and outputs a first enhanced feature; the first sub-fusion module performs fusion processing on the first enhanced feature and the fourth sub-feature after convolution processing, and outputs the fifth sub-feature; the second sub-fusion module performs fusion processing on the fifth sub-feature after sampling operation and the third sub-feature after convolution processing, and outputs the sixth sub-feature; the third sub-fusion module performs fusion processing on the sixth sub-feature after sampling operation and the second sub-feature after convolution processing, and outputs the seventh sub-feature; the fifth sub-feature, the sixth sub-feature, and the seventh sub-feature are respectively subjected to convolution processing by a convolution module, the second feature enhancement module performs feature enhancement processing on the fifth sub-feature after convolution processing, and outputs a second enhanced feature, the fourth sub-fusion module performs fusion processing on the second enhanced feature and the fifth sub-feature after convolution processing, and outputs the eighth sub-feature; the fifth sub-fusion module performs fusion processing on the eighth sub-feature after sampling operation and the sixth sub-feature after convolution processing, and outputs the ninth sub-feature; the sixth sub-fusion module performs fusion processing on the ninth sub-feature after sampling operation and the seventh sub-feature after convolution processing, and outputs the tenth sub-feature; the ninth sub-feature and the tenth sub-feature are respectively subjected to convolution processing by a convolution module, and the splicing module splices and combines the eighth sub-feature, the ninth sub-feature and the tenth sub-feature after convolution processing, and outputs the first feature.

[0010] In any of the above technical solutions, optionally, the first feature enhancement module or the second feature enhancement module includes a third fusion module and a first shortcut connection structure, a first dilated convolution layer, and a second dilated convolution layer connected in parallel, wherein the dilation rate of the first dilated convolution layer is not equal to the dilation rate of the second dilated convolution layer.

[0011] In any of the above technical solutions, optionally, the first feature enhancement module or the second feature enhancement module includes a fourth fusion module and a second shortcut connection structure, a standard convolution layer, a third dilated convolution layer, and a fourth dilated convolution layer connected in parallel, wherein the dilation rate of the third dilated convolution layer is not equal to the dilation rate of the fourth dilated convolution layer.

[0012] In any of the above technical solutions, optionally, the feature purification structure comprises a third shortcut connection structure, a channel attention module, a spatial attention module, a fifth fusion module, a sixth fusion module, and a seventh fusion module; the third shortcut connection structure directly transmits the first feature, the channel attention module performs feature purification processing on the first feature in the channel dimension and outputs a first purified feature, the spatial attention module performs feature purification processing on the first feature in the spatial dimension and outputs a second purified feature, the fifth fusion module performs fusion processing on the first feature transmitted by the third shortcut connection structure and the first purified feature and outputs a first purified fusion feature, the sixth fusion module performs fusion processing on the first feature transmitted by the third shortcut connection structure and the second purified feature and outputs a second purified fusion feature, and the seventh fusion module performs fusion processing on the first purified fusion feature and the second purified fusion feature and outputs the second feature.

[0013] In any of the above technical solutions, optionally, the target laser radar point cloud data corresponding to the driving-obstructing object is acquired according to the preset calibration relationship between the image acquisition device and the laser radar and the laser radar point cloud data, comprising: projecting the laser radar point cloud data to the mine road surface image according to the preset calibration relationship between the image acquisition device and the laser radar, to acquire a mapping relationship between the laser radar point cloud data and the pixel points of the mine road surface image; and acquiring the target laser radar point cloud data corresponding to the driving-obstructing object based on the mapping relationship.

[0014] In any of the above technical solutions, optionally, the mine road surface image is acquired by the image acquisition device, comprising: acquiring a mine image by the image acquisition device, and cropping the mine image to obtain the mine road surface image in the mine image.

[0015] In any of the above technical solutions, optionally, before the mine road surface image is acquired by the image acquisition device and the laser radar point cloud data corresponding to the mine road surface image is acquired by the laser radar, the method further comprises: acquiring a road surface sample image by the image acquisition device, and labeling the shape, position and category of a sample object in the road surface sample image; and training a model according to the labeled road surface sample image to construct a multi-task road surface detection and segmentation model.

[0016] In a second aspect, an embodiment of the present application provides a device for identifying an object obstructing driving on a mine road surface, comprising: a data acquisition module configured to acquire a mine road surface image by using an image acquisition device and to acquire laser radar point cloud data corresponding to the mine road surface image by using a laser radar; an identification module configured to identify an object obstructing driving in the mine road surface image according to a constructed multi-task road surface detection and segmentation model; and a post-processing module configured to acquire target laser radar point cloud data corresponding to the object obstructing driving according to a preset calibration relationship between the image acquisition device and the laser radar and the laser radar point cloud data, and to determine a shape and a position of the object obstructing driving in a three-dimensional space according to the target laser radar point cloud data; wherein the multi-task road surface detection and segmentation model comprises a feature extraction structure, a feature purification structure, and a multi-task prediction structure, the feature extraction structure is configured to extract features in the mine road surface image, to perform feature enhancement processing on the extracted features, and to output first features, the feature purification structure is configured to perform feature purification processing on the first features, and to output second features, and the multi-task prediction structure is configured to perform prediction identification on the second features, and to output detection and segmentation results of the object obstructing driving.

[0017] In a third aspect, an embodiment of the present application provides a chip, which comprises at least one processor and a communication interface, the communication interface is coupled to the at least one processor, and the at least one processor is configured to run a program or an instruction to implement steps of the method for identifying an object obstructing driving on a mine road surface according to the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a terminal, which comprises the device for identifying an object obstructing driving on a mine road surface according to the second aspect.

[0019] In a fifth aspect, an embodiment of the present application provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement steps of the method for identifying an object obstructing driving on a mine road surface according to the first aspect when executed on the processor.

[0020] In a sixth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is configured to implement steps of the method for identifying an object obstructing driving on a mine road surface according to the first aspect when executed on a processor.

[0021] In the embodiment of the present application, the mine road surface image and the laser radar point cloud data collected in synchronization are acquired, and then the mine road surface image is detected and semantically segmented according to the constructed multi-task road surface detection and segmentation model to identify the driving obstacle object therein. Then, the three-dimensional spatial shape and position of the driving obstacle object are estimated by fusing the laser radar and the image according to a post-processing method. Specifically, the feature extraction structure in the multi-task road surface detection and segmentation model has the functions of feature extraction and feature enhancement, can extract semantic features of different levels and perform semantic enhancement on the extracted semantic features, thereby obtaining rich context semantic information, enhancing the saliency of special target features such as rockfall, ruts and potholes that are not easy to detect, and improving the recognition ability of the multi-task road surface detection and segmentation model for the special target features. The feature purification structure in the multi-task road surface detection and segmentation model can purify the first features output by the feature extraction structure in different dimensions such as channels and spaces, improve the attention of the multi-task road surface detection and segmentation model to key areas and targets, and thus facilitate the accurate extraction of special target features such as rockfall, ruts and potholes that are not easy to detect, and further improve the recognition ability for the special target features. The multi-task prediction structure in the multi-task road surface detection and segmentation model is responsible for target prediction on the second features output by the feature purification structure, and outputs the detection and semantic segmentation results of the driving obstacle object.

[0022] In the embodiment of the present application, the multi-task road surface detection and segmentation model with the functions of feature enhancement and feature purification is used to identify the category of the driving obstacle object in the mine road surface image, and identify the shape and position of the driving obstacle object according to the mapping relationship between the laser radar point cloud data and the mine road surface image, so as to finally realize the accurate identification of the driving obstacle object that is not easy to detect on the mine road surface, and ensure the accuracy of the automatic driving decision planning of the mine car.

[0023] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the embodiments of the present application can be implemented according to the content of the description, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0024] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 A flowchart of a mine road surface driving obstacle object identification method according to an embodiment of the present application is shown;

[0026] Figure 2A structural schematic diagram of a multi-task road surface detection and segmentation model of an embodiment of the present application is shown.

[0027] Figure 3 A structural schematic diagram of a basic module of a backbone network of an embodiment of the present application is shown.

[0028] Figure 4 A structural schematic diagram of a feature enhancement module of an embodiment of the present application is shown.

[0029] Figure 5 A structural schematic diagram of a feature enhancement module of an embodiment of the present application is shown.

[0030] Figure 6 A structural block diagram of an obstacle identification device for a mine road surface of an embodiment of the present application is shown.

[0031] Figure 7 A structural schematic diagram of a chip of an embodiment of the present application is shown.

[0032] Figure 8 A structural schematic diagram of a terminal of an embodiment of the present application is shown.

[0033] Figure 9 A structural schematic diagram of a computer device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present application will be described clearly below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present application.

[0035] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of a kind and are not limited in number, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the objects before and after are in an "or" relationship.

[0036] The mine road surface obstacle identification method, mine road surface obstacle identification device, chip, terminal, computer device, and computer readable storage medium provided by the embodiments of the present application will be described in detail below with reference to the drawings, through specific embodiments and their application scenarios.

[0037] The embodiment of the present application provides a method for identifying an object impeding driving on a mine road surface, as shown in the method, the method comprises the steps of: Figure 1

[0038] In step 101, a mine road surface image is acquired through an image acquisition device, and laser radar point cloud data corresponding to the mine road surface image is acquired through a laser radar.

[0039] The mine road surface image is acquired through an image acquisition device (for example, a camera), and the laser radar point cloud data corresponding to the mine road surface image is acquired through a laser radar, wherein the acquisition time of the mine road surface image and the laser radar point cloud data is synchronized, for example, the image acquisition device and the laser radar can be time-synchronized through a soft synchronization mode. The soft synchronization mode is to match different sensors by using a timestamp, and generally, the data of each sensor is unified to the data of a sensor with a longer scanning period, for example, if the scanning period of the laser radar is longer than that of the camera, the matching is performed based on the sampling frequency of the laser radar, and each sampling time of the sensor is recorded on a unified time sequence, when the laser radar completes a sampling, the image of the nearest neighbor time is found, and thus the time matching of the laser radar point cloud data and the mine road surface image data is completed.

[0040] In step 102, an object impeding driving in the mine road surface image is identified according to a constructed multi-task road surface detection and segmentation model.

[0041] The multi-task road surface detection and segmentation model comprises a feature extraction structure, a feature purification structure and a multi-task prediction structure, the feature extraction structure is used for extracting features in the mine road surface image, and performing feature enhancement processing on the extracted features to output first features, the feature purification structure is used for performing feature purification processing on the first features to output second features, and the multi-task prediction structure is used for performing prediction identification on the second features to output detection and segmentation results of the object impeding driving.

[0042] The object impeding driving is a special target feature which is not easy to detect, and the categories thereof include but are not limited to rockfall, ruts, water pits and the like.

[0043] The mine road surface image is detected and semantically segmented according to the constructed multi-task road surface detection and segmentation model to identify the object impeding driving therein. Specifically, the feature extraction structure in the multi-task road surface detection and segmentation model has the functions of feature extraction and feature enhancement, can extract semantic features of different levels and perform semantic enhancement on the extracted semantic features, thereby obtaining rich context semantic information, enhancing the saliency of the special target features such as rockfall, ruts and water pits which are not easy to detect, and improving the identification capability of the multi-task road surface detection and segmentation model for the special target features.​

[0044] The feature purification structure in the multi-task road surface detection and segmentation model can purify the first features output by the feature extraction structure in different dimensions such as channels and spaces, improve the attention of the multi-task road surface detection and segmentation model to key areas and targets, and thus facilitate the accurate extraction of target features such as rockfalls, ruts and potholes that are not easy to detect, and further improve the recognition ability of special target features.

[0045] The multi-task prediction structure in the multi-task road surface detection and segmentation model is responsible for predicting and identifying the second features output by the feature purification structure, and outputs the detection and semantic segmentation results of the objects that hinder driving, that is, the categories of the objects that hinder driving in the mine road surface image.

[0046] It should be noted that multi-task means that the prediction structure includes two outputs, one is a target detection branch to predict the target, which is manifested by surrounding with a rectangular frame; and the other is a semantic segmentation branch to predict the target, which is manifested by surrounding with an envelope. Through this structure, it can be suitable for identifying objects of different categories, for example, rockfalls and potholes are suitable for being circled with a rectangular frame, while ruts cannot be well circled with a rectangular frame, so ruts are suitable for being circled with an envelope closed area.

[0047] Step 103, according to the preset calibration relationship between the image acquisition device and the laser radar, and the laser radar point cloud data, target laser radar point cloud data corresponding to the object that hinders driving is obtained.

[0048] Step 104, according to the target laser radar point cloud data, the shape and position of the object that hinders driving in the three-dimensional space are determined.

[0049] Step 105, the category, shape and position of the object that hinders driving are output.

[0050] Further, according to the extrinsic calibration relationship between the image acquisition device and the laser radar, target laser radar point cloud data corresponding to the object that hinders driving is obtained based on the laser radar point cloud data, and the shape and position of the object that hinders driving in the three-dimensional space are obtained according to the target laser radar point cloud data, wherein the three-dimensional space corresponds to the mine space, and the shape of the object that hinders driving includes shape and volume.

[0051] Specifically, for the track obstacle, the target laser radar point cloud data is post-processed by point cloud clustering and point cloud convex hull search to obtain the obstacle image region envelope point cloud, so as to determine the shape and spatial position of the obstacle; or for the rockfall and water pit obstacle, the target laser radar point cloud data is fitted by a bounding box to obtain the spatial rectangular bounding box of the obstacle, and the shape and spatial position of the obstacle are determined. The point cloud clustering adopts the Euclidean clustering algorithm, which uses the Euclidean distance between points in the point cloud for clustering. When the Euclidean distance between points is less than a set threshold, it is considered as a class. The point cloud convex hull search adopts the Graham scan algorithm, which continuously adds new points to the convex hull and removes points that affect the convexity, and finally forms a convex hull.

[0052] According to the mapping relationship between the laser radar point cloud data and the mine road surface image, the shape and position of the obstacle are identified, and finally the precise identification of the obstacle on the mine road surface that is not easy to detect is realized, which can ensure the accuracy of the automatic driving decision planning of the mine car.

[0053] In an embodiment of the present application, the step of obtaining the target laser radar point cloud data corresponding to the obstacle according to the preset calibration relationship between the image acquisition device and the laser radar and the laser radar point cloud data specifically comprises: projecting the laser radar point cloud data to the mine road surface image according to the preset calibration relationship between the image acquisition device and the laser radar, obtaining the mapping relationship between the laser radar point cloud data and the pixel points of the mine road surface image; and obtaining the target laser radar point cloud data corresponding to the obstacle based on the mapping relationship.

[0054] In this embodiment, the laser radar point cloud data is projected onto the obtained mine road surface image according to the preset calibration relationship between the image acquisition device and the laser radar to obtain the mapping relationship between the laser radar point cloud data and the pixel points of the mine road surface image. Specifically, by completing the coordinate conversion of the laser radar world coordinate system→the camera coordinate system→the image plane coordinate system→the pixel coordinate system, the projection of the 3D laser radar point cloud data to the 2D mine road surface image is realized, and the mapping relationship between the laser radar point cloud data and the 2D mine road surface image is obtained. Then, based on the mapping relationship, the detection and segmentation results of the obstacle are back-projected to the three-dimensional point cloud space to obtain the target laser radar point cloud data corresponding to the obstacle.

[0055] In this way, the target laser radar point cloud data of the obstacle in the mine road surface image can be accurately obtained, providing a data basis for determining the shape and position of the obstacle.

[0056] In an embodiment of the present application, the step of acquiring the mine area road surface image and the laser radar point cloud data specifically comprises: acquiring the mine area image through the image acquisition device, and cropping the mine area image to obtain the mine area road surface image in the mine area image; and acquiring the laser radar point cloud data through the laser radar, and performing soft synchronization on the laser radar point cloud data to obtain the laser radar point cloud data synchronized with the acquisition time of the mine area road surface image.

[0057] In this embodiment, the mine area image is acquired in real time through the image acquisition device. Since the rockfall, rut and water pit only appear on the ground, the real-time acquired mine area image needs to be cropped to remove the redundant part and only keep the road surface part, so as to reduce the subsequent identification difficulty and improve the running speed.

[0058] In the embodiment of the present application, cropping the mine area image means identifying the mine area image, dividing it into a road surface area and other background areas, and then segmenting the circumscribed rectangle of the road surface area to obtain the mine area road surface image. In this way, the interference of the background area can be excluded, and the identification of the objects on the road surface can be more focused.

[0059] After the laser radar point cloud data is acquired, it is time-synchronized with the mine area road surface image to ensure the accuracy of the subsequent determination of the laser radar point cloud data of the driving obstacle in the mine area road surface image, and further ensure the accuracy of the shape and position identification of the driving obstacle.

[0060] In an embodiment of the present application, before acquiring the mine area road surface image and the corresponding laser radar point cloud data, it further comprises: acquiring the road surface sample image through the image acquisition device, and labeling the shape, position and category of the sample object in the road surface sample image; training the model according to the labeled road surface sample image to construct the multi-task road surface detection and segmentation model.

[0061] In this embodiment, the image acquisition device is used to acquire the image in the scene containing rockfall, rut and water pit in the mine area to obtain the road surface sample image set containing rockfall, rut and water pit. Further, the shape, position and category of the rockfall, rut and water pit in the road surface sample image are labeled by using a labeling tool. Finally, based on the deep learning model, the model is trained by using the labeled road surface sample image to construct the multi-task road surface detection and segmentation model for identifying the special target features such as rockfall, rut and water pit.

[0062] In the above manner, the accurate multi-task road surface detection and segmentation model is constructed to facilitate the detection and semantic segmentation of the special target features such as rockfall, rut and water pit.

[0063] The following will be described in detail Figures 2 to 5 , the multi-task road surface detection and segmentation model in the embodiment of the present application detects the driving obstacle.

[0064] The structure of the multi-task road surface detection and segmentation model is as shown in Figure 2 The multi-task road surface detection and segmentation model includes a feature extraction structure, a feature purification structure, and a multi-task prediction structure.

[0065] The feature extraction structure is a multi-scale feature extraction structure, which can extract features of different scales of mine road surface images, and perform feature enhancement and fusion to effectively extract features such as rockfall, ruts, and water pits. The feature extraction structure includes a backbone network, a first feature enhancement module, a second feature enhancement module, a first fusion module, a second fusion module, a splicing module, and a plurality of convolution modules. The first fusion module includes a first sub-fusion module, a second sub-fusion module, and a third sub-fusion module. The second fusion module includes a fourth sub-fusion module, a fifth sub-fusion module, and a sixth sub-fusion module. The convolution module can be a 3x3 convolution module.

[0066] The backbone network is built by a plurality of basic modules. Each basic module is a residual structure, and the number of basic modules is 33. As shown in Figure 3 One basic module includes two 1x1 convolution units and one 3x3 convolution unit. The mine road surface image is generated into a plurality of first layer features by the backbone network, including a first sub-feature a1, a second sub-feature a2, a third sub-feature a3, and a fourth sub-feature a4. The first sub-feature a1, the second sub-feature a2, the third sub-feature a3, and the fourth sub-feature a4 output by the backbone network are then calculated by the 3x3 convolution module, and then wait for subsequent processing.

[0067] In order to improve the perception ability of rockfall, ruts, and water pits, the feature enhancement module is proposed in the embodiments of the present application. As shown in Figure 2 The first feature enhancement module performs feature enhancement processing on the deepest layer feature in the plurality of first layer features, i.e., the fourth sub-feature a4, and outputs a first enhanced feature. The first enhanced feature and the fourth sub-feature a4 processed by the 3x3 convolution are added and fused by the first sub-fusion module to obtain a fifth sub-feature b3.

[0068] The fifth sub-feature b3 is subjected to a sampling operation (e.g., an upsampling operation) to expand the size by one time and then passed down, and the third sub-feature a3 processed by the 3x3 convolution is added and fused by the second sub-fusion module to obtain a sixth sub-feature b2. The sixth sub-feature b2 is subjected to a sampling operation (e.g., an upsampling operation) to expand the size by one time and then passed down, and the second sub-feature a2 processed by the 3x3 convolution is added and fused by the third sub-fusion module to obtain a seventh sub-feature b1. The fifth sub-feature b3, the sixth sub-feature b2, and the seventh sub-feature b1 are the plurality of second layer features.

[0069] Similarly, the fifth sub-feature b3, the sixth sub-feature b2, and the seventh sub-feature b1 can be used to generate the eighth sub-feature c3, the ninth sub-feature c2, and the tenth sub-feature c1. Specifically, the fifth sub-feature b3, the sixth sub-feature b2, and the seventh sub-feature b1 are each calculated using a 3×3 convolution module. The second feature enhancement module performs feature enhancement processing on the deepest feature among the multiple second-layer features, namely the fifth sub-feature b3, and outputs a second enhanced feature. This second enhanced feature is then added to and fused with the sixth sub-feature b2 after the 3×3 convolution processing through the fourth sub-fusion module to obtain the eighth sub-feature c3.

[0070] The eighth sub-feature c3 is subjected to a sampling operation (e.g., upsampling or downsampling) to double its size and then passed down. It is then fused with the sixth sub-feature b2, which has undergone 3×3 convolution, through the fifth sub-fusion module to obtain the ninth sub-feature c2. The ninth sub-feature c2 is subjected to a sampling operation (e.g., upsampling or downsampling) to double its size and then passed down. It is then fused with the seventh sub-feature b1, which has undergone 3×3 convolution, through the sixth sub-fusion module to obtain the tenth sub-feature c1.

[0071] After obtaining the eighth sub-feature c3, the ninth sub-feature c2, and the tenth sub-feature c1, the ninth sub-feature c2 and the tenth sub-feature c1 are respectively subjected to 3×3 convolution processing so that they have the same size as the eighth sub-feature c3. Then, they are spliced ​​and merged with the eighth sub-feature c3 through the splicing module C to output the first feature d.

[0072] The structures of the first feature enhancement module and the second feature enhancement module can be the same.

[0073] In one embodiment of this application, such as Figure 4 As shown, the feature enhancement module includes a three-parallel branch structure and a fusion structure. The fusion structure is the third fusion module. The three parallel branches are a first shortcut connection structure, a first dilated convolutional layer with a kernel size of 3×3 and an dilation rate of 2 (r=2), and a second dilated convolutional layer with a kernel size of 3×3 and a dilation rate of 4 (r=4). The first shortcut connection structure directly transmits the input information, ensuring feature integrity. The other two branches expand the receptive field of the convolution to different degrees through dilated convolutions with different dilation rates, improving the ability to recognize special targets.

[0074] The feature enhancement module receives the deepest features (i.e., the fourth sub-feature a4 or the fifth sub-feature b3), processes the input deepest features by using three parallel branches, the first branch is a first shortcut connection structure which directly transmits the input features, the second branch is a first dilated convolution layer with a dilation rate of 2 which expands the convolution receptive field by one time without changing the size of the input features, and the third branch is a second dilated convolution layer with a dilation rate of 4 which expands the convolution receptive field by two times without changing the size of the input features. After the input deepest features are processed by the three branches, the third fusion module is used to additively fuse the results output by the three branches to obtain a fusion result, i.e., a first enhanced feature or a second enhanced feature.

[0075] In another embodiment of the present application, as shown in Figure 5 the feature enhancement module includes four parallel branch structures and one fusion structure, the fourth fusion module is a fusion structure, and the four parallel branches are a second shortcut connection structure, a standard convolution layer with a convolution kernel size of 3x3, a third dilated convolution layer with a convolution kernel size of 3x3 and a dilation rate of 2 (i.e., r=2), and a fourth dilated convolution layer with a convolution kernel size of 3x3 and a dilation rate of 4 (i.e., r=4). The added standard convolution layer improves the richness and completeness of the feature hierarchy, which can further improve the accuracy of the multi-task road surface detection and segmentation model.

[0076] Figure 4 The feature enhancement module shown in the figure obtains the original features and the dilated convolution features under different receptive fields through the shortcut connection branch and the dilated convolution branch, respectively, improves the ability of the multi-task road surface detection and semantic segmentation model in understanding the context semantics, and is beneficial to analyzing and identifying special targets such as rockfalls, ruts, and potholes. Figure 5 The feature enhancement module shown in the figure further adds a standard convolution branch to perform convolution calculation on the original features, so that the multi-task road surface detection and semantic segmentation model can obtain more rich feature levels and build a more complete hierarchical system, and further enhances the detection capability of the model on special targets. Therefore, under the condition that the computational amount is allowed, the feature enhancement module shown in Figure 5 can be used to improve the accuracy of the multi-task road surface detection and segmentation model. In actual application, the first and second feature enhancement modules can be used simultaneously as shown in Figure 4 , or the structures shown in Figure 5 can be used simultaneously, or the first feature enhancement module can use the structure shown in Figure 4 , and the second feature enhancement module can use the structure shown in Figure 5 . The specific application of the feature enhancement module is not limited in the present application.

[0077] In the embodiments of the present application, the feature purification structure uses the channel attention module and the spatial attention module to respectively perform feature purification processing on the features of the rockfall, the rut and the water pit in the channel and spatial (including length and width) dimensions, so that the feature information is more prominent and can be more accurately detected and semantically segmented.

[0078] As shown in Figure 2 The feature purification structure receives the first feature d, processes it through three branches, the three branches are respectively the channel attention module, the spatial attention module and the third shortcut connection structure, the third shortcut connection structure directly transmits the first feature d, the channel attention module performs feature purification processing on the first feature d in the channel dimension and outputs the first purified feature, and the spatial attention module performs feature purification processing on the first feature d in the spatial dimension and outputs the second purified feature.

[0079] Then, the fifth fusion module is used to multiply the output results of the channel attention module and the third shortcut connection structure to obtain the first purified fusion feature d1, and the sixth fusion module is used to multiply the output results of the spatial attention module and the third shortcut connection structure to obtain the second purified fusion feature d2. Finally, the seventh fusion module is used to add the first purified fusion feature d1 and the second purified fusion feature d2 to output the final purified fusion feature, that is, the second feature.

[0080] The multi-task prediction structure is responsible for target prediction on the second feature output by the feature purification structure, and outputs the target detection result and the semantic segmentation result.

[0081] In summary, the multi-task road surface detection and semantic segmentation model of the mine area road rockfall, rut and water pit based on deep learning of the present application mainly includes three parts, which are respectively the feature extraction structure, the feature purification structure and the multi-task prediction structure. The feature extraction structure extracts the features of the input mine area road surface image, enhances the expression ability of the features through the feature enhancement module, and improves the prominence of the target features of the rockfall, the rut and the water pit in the mine area road surface image; the feature purification structure is responsible for purifying the extracted features, uses the spatial attention module and the channel attention module to extract the feature information of the rockfall, the rut and the water pit, and obtains high-quality features of the rockfall, the rut and the water pit; the multi-task prediction structure predicts and identifies the purified features to generate the detection and semantic segmentation results of the rockfall, the rut and the water pit and other driving-obstructing objects. And based on the laser radar point cloud data corresponding to the driving-obstructing objects, the three-dimensional spatial shape and position of the driving-obstructing objects are estimated by fusing the laser radar and the image according to the post-processing method. Finally, the precise identification of the driving-obstructing objects that are not easy to detect on the mine area road surface is realized, and the accuracy of the automatic driving decision planning of the mine car is ensured.

[0082] As a specific implementation of the above-mentioned mine road surface obstacle identification method, the embodiment of the present application provides a mine road surface obstacle identification device. Figure 6 As shown in the figure, the mine road surface obstacle identification device 600 comprises a data acquisition module 601, an identification module 602 and a post-processing module 603.

[0083] The data acquisition module 601 is configured to acquire mine road surface images through an image acquisition device and acquire laser radar point cloud data corresponding to the mine road surface images through a laser radar; the identification module 602 is configured to identify the obstacles in the mine road surface images according to a multi-task road surface detection and segmentation model that has been constructed; and the post-processing module 603 is configured to acquire target laser radar point cloud data corresponding to the obstacles according to a preset calibration relationship between the image acquisition device and the laser radar and the laser radar point cloud data, and determine the shape and position of the obstacles in three-dimensional space according to the target laser radar point cloud data; wherein the multi-task road surface detection and segmentation model comprises a feature extraction structure, a feature purification structure and a multi-task prediction structure, the feature extraction structure is configured to extract features in the mine road surface images and perform feature enhancement processing on the extracted features to output first features, the feature purification structure is configured to perform feature purification processing on the first features to output second features, and the multi-task prediction structure is configured to perform prediction identification on the second features to output detection and segmentation results of the obstacles.

[0084] In this embodiment, mine road surface images and laser radar point cloud data that are collected synchronously in time are acquired, and then the mine road surface images are detected and semantically segmented according to a multi-task road surface detection and segmentation model that has been constructed, so as to identify the obstacles therein.

[0085] Specifically, the feature extraction structure in the multi-task road surface detection and segmentation model has the functions of feature extraction and feature enhancement, can extract semantic features of different levels and perform semantic enhancement on the extracted semantic features, so as to acquire rich context semantic information, enhance the saliency of special target features such as rockfalls, ruts and potholes that are not easy to detect, and improve the identification ability of the multi-task road surface detection and segmentation model for the special target features. The feature purification structure in the multi-task road surface detection and segmentation model can perform feature purification on the first features output by the feature extraction structure in different dimensions such as channels and spaces, improve the attention of the multi-task road surface detection and segmentation model to key areas and targets, and thus be conducive to realizing accurate extraction of special target features such as rockfalls, ruts and potholes that are not easy to detect, and further improving the identification ability for the special target features. The multi-task prediction structure in the multi-task road surface detection and segmentation model is responsible for target prediction on the second features output by the feature purification structure, and outputs detection and semantic segmentation results of the obstacles.

[0086] Further, according to an extrinsic calibration relationship between the image acquisition device and the laser radar, target laser radar point cloud data corresponding to the driving-obstructing object is acquired based on laser radar point cloud data, and the target laser radar point cloud data is processed by clustering to obtain the shape and position of the driving-obstructing object in a three-dimensional space, wherein the three-dimensional space corresponds to a mine space, and the shape of the driving-obstructing object includes shape and volume.

[0087] According to the multi-task road surface detection and segmentation model with feature enhancement and feature purification functions, the embodiments of the present application can identify the driving-obstructing object in the mine road surface image, and identify the shape and position of the driving-obstructing object according to the mapping relationship between the laser radar point cloud data and the mine road surface image, so as to accurately identify the driving-obstructing object on the mine road surface, and ensure the accuracy of the automatic driving decision planning of the mine car.

[0088] Further, the driving-obstructing object includes at least one of the following: rockfall, ruts, and water pits.

[0089] Further, the feature extraction structure includes a backbone network, a first feature enhancement module, a second feature enhancement module, a first fusion module, a second fusion module, and a splicing module; wherein the backbone network generates a plurality of first layer features according to the mine road surface image; the first feature enhancement module performs feature enhancement processing on the first deepest layer feature in the plurality of first layer features, and outputs a first enhanced feature; the first fusion module fuses the plurality of first layer features using the first enhanced feature, and outputs a plurality of second layer features; the second feature enhancement module performs feature enhancement processing on the second deepest layer feature in the plurality of second layer features, and outputs a second enhanced feature; the second fusion module fuses the plurality of second layer features using the second enhanced feature, and outputs a plurality of third layer features; and the splicing module splices and merges the plurality of third layer features, and outputs the first feature.

[0090] Further, the first layer features include a first sub-feature, a second sub-feature, a third sub-feature, and a fourth sub-feature, wherein the fourth sub-feature is the first deepest layer feature, the second layer features include a fifth sub-feature, a sixth sub-feature, and a seventh sub-feature, wherein the fifth sub-feature is the second deepest layer feature, and the third layer features include an eighth sub-feature, a ninth sub-feature, and a tenth sub-feature; the feature extraction structure further includes a plurality of convolution modules, the first fusion module includes a first sub-fusion module, a second sub-fusion module, and a third sub-fusion module, and the second fusion module includes a fourth sub-fusion module, a fifth sub-fusion module, and a sixth sub-fusion module; wherein the first sub-feature, the second sub-feature, the third sub-feature, and the fourth sub-feature are respectively subjected to convolution processing by a convolution module, the first feature enhancement module performs feature enhancement processing on the fourth sub-feature after convolution processing, and outputs a first enhanced feature; the first sub-fusion module performs fusion processing on the first enhanced feature and the fourth sub-feature after convolution processing, and outputs the fifth sub-feature; the second sub-fusion module performs fusion processing on the fifth sub-feature after sampling operation and the third sub-feature after convolution processing, and outputs the sixth sub-feature; the third sub-fusion module performs fusion processing on the sixth sub-feature after sampling operation and the second sub-feature after convolution processing, and outputs the seventh sub-feature; the fifth sub-feature, the sixth sub-feature, and the seventh sub-feature are respectively subjected to convolution processing by a convolution module, the second feature enhancement module performs feature enhancement processing on the fifth sub-feature after convolution processing, and outputs a second enhanced feature, the fourth sub-fusion module performs fusion processing on the second enhanced feature and the fifth sub-feature after convolution processing, and outputs the eighth sub-feature; the fifth sub-fusion module performs fusion processing on the eighth sub-feature after sampling operation and the sixth sub-feature after convolution processing, and outputs the ninth sub-feature; the sixth sub-fusion module performs fusion processing on the ninth sub-feature after sampling operation and the seventh sub-feature after convolution processing, and outputs the tenth sub-feature; the ninth sub-feature and the tenth sub-feature are respectively subjected to convolution processing by a convolution module, and the splicing module splices and combines the eighth sub-feature, the ninth sub-feature and the tenth sub-feature after convolution processing, and outputs the first feature.

[0091] Further, the first feature enhancement module or the second feature enhancement module includes a third fusion module and a first shortcut connection structure, a first dilated convolution layer, and a second dilated convolution layer connected in parallel, wherein the dilation rate of the first dilated convolution layer is not equal to the dilation rate of the second dilated convolution layer.

[0092] Further, the first feature enhancement module or the second feature enhancement module includes a fourth fusion module and a second shortcut connection structure, a standard convolution layer, a third dilated convolution layer, and a fourth dilated convolution layer connected in parallel, wherein the dilation rate of the third dilated convolution layer is not equal to the dilation rate of the fourth dilated convolution layer.

[0093] Further, the feature purification structure comprises a third shortcut connection structure, a channel attention module, a spatial attention module, a fifth fusion module, a sixth fusion module, and a seventh fusion module; the third shortcut connection structure directly transmits the first feature; the channel attention module performs feature purification processing on the first feature in the channel dimension and outputs a first purified feature; the spatial attention module performs feature purification processing on the first feature in the spatial dimension and outputs a second purified feature; the fifth fusion module performs fusion processing on the first feature transmitted by the third shortcut connection structure and the first purified feature and outputs a first purified fusion feature; the sixth fusion module performs fusion processing on the first feature transmitted by the third shortcut connection structure and the second purified feature and outputs a second purified fusion feature; and the seventh fusion module performs fusion processing on the first purified fusion feature and the second purified fusion feature and outputs the second feature.

[0094] Further, the post-processing module 603 is specifically configured to project the laser radar point cloud data to the mine road surface image according to a preset calibration relationship between the image acquisition device and the laser radar, and obtain a mapping relationship between the laser radar point cloud data and the pixels of the mine road surface image; and obtain the target laser radar point cloud data corresponding to the obstacle object based on the mapping relationship.

[0095] Further, the data acquisition module 601 is specifically configured to acquire the mine image through the image acquisition device, and crop the mine image to obtain the mine road surface image in the mine image.

[0096] Further, the data acquisition module 601 is further configured to acquire the road surface sample image through the image acquisition device; the device further comprises: a labeling module configured to label the shape, position and category of the sample object in the road surface sample image; and a model construction module configured to train the model according to the labeled road surface sample image and construct the multi-task road surface detection and segmentation model.

[0097] The mine road surface obstacle to driving object identification device 600 in the embodiments of the present application can be a computer device, or a component in a computer device, such as an integrated circuit or a chip. The computer device can be a terminal, or other devices other than a terminal. For example, the computer device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted computer device, a Mobile Internet Device (MID), a robot, an Ultra-Mobile Personal Computer (UMPC), a netbook, or a Personal Digital Assistant (PDA), and can also be a server, a Network Attached Storage (NAS), a Personal Computer (PC), and the like, and the embodiments of the present application are not limited in this regard.

[0098] The mine road surface obstacle to driving object identification device 600 in the embodiments of the present application can be a device with an operating system. The operating system can be an Android operating system, an ios operating system, or other possible operating systems, and the embodiments of the present application are not limited in this regard.

[0099] The mine road surface obstacle to driving object identification device 600 provided by the embodiments of the present application can implement the mine road surface obstacle to driving object identification method embodiments Figure 1 The processes implemented by the mine road surface obstacle to driving object identification method embodiments are not repeated here to avoid repetition.

[0100] The embodiments of the present application further provide a chip, as shown in Figure 7 The chip 700 includes at least one processor 701 and a communication interface 702, the communication interface 702 is coupled to the at least one processor 701, the at least one processor 701 is configured to run programs or instructions, implement various processes of the above-mentioned mine road surface obstacle to driving object identification method embodiments, and achieve the same technical effects, and the details are not repeated here to avoid repetition.

[0101] It should be understood that the chip mentioned in the embodiments of the present application can also be referred to as a system-level chip, a system chip, a chip system, or a system-on-chip chip, and the like.

[0102] Preferably, the chip 700 further includes a memory 703, which stores the following elements: executable modules or data structures, or a subset thereof, or an expanded set thereof.

[0103] In the embodiments of the present application, the memory 703 can include read-only memory and random access memory, and provide instructions and data for the processor 701. A part of the memory 703 can also include non-volatile random access memory (NVRAM).

[0104] In the embodiments of the present application, the processor 701, the communication interface 702 and the memory 703 are coupled together through the bus system 704. Among them, the bus system 704 can include not only a data bus, but also a power bus, a control bus and a status signal bus, etc. For the convenience of description, all kinds of buses are marked as the bus system 704. Figure 7

[0105] The method described in the above embodiments of the present application can be applied in the processor 701 or implemented by the processor 701. The processor 701 can be an integrated circuit chip with processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 701 or the instruction in the form of software. The processor 701 described above can be a general processor (for example, a microprocessor or a conventional processor), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices or discrete hardware components, and the processor 701 can implement or execute the methods, steps and logical block diagrams disclosed in the embodiments of the present application.

[0106] The embodiments of the present application also provide a terminal, as shown in the figure, the terminal 800 includes the above-mentioned mine road surface obstacle driving object identification device 600. Figure 8

[0107] The terminal 800 described above can execute the mine road surface obstacle driving object identification method described in the above embodiments through the mine road surface obstacle driving object identification device 600. It can be understood that the implementation mode of the terminal 800 controlling the mine road surface obstacle driving object identification device 600 can be set according to the actual application scene, and the embodiments of the present application are not limited specifically.

[0108] ​​The aforementioned terminal 800 includes, but is not limited to, vehicles, vehicle-mounted terminals, vehicle-mounted controllers, vehicle-mounted modules, vehicle-mounted components, vehicle-mounted chips, vehicle-mounted units, vehicle-mounted radar, or vehicle-mounted cameras, and other sensors. Vehicles can implement the methods provided in this application through these vehicle-mounted terminals, controllers, modules, components, chips, units, radar, or cameras. Vehicles in this application include passenger cars and commercial vehicles. Common commercial vehicle models include, but are not limited to, pickup trucks, mini-trucks, light trucks, minivans, dump trucks, cargo trucks, tractors, trailers, special-purpose vehicles, and mining vehicles. Mining vehicles include, but are not limited to, mining trucks, wide-body trucks, articulated trucks, excavators, electric shovels, and bulldozers. This application does not further limit the type of intelligent vehicle; any type of vehicle is within the scope of protection of this application.

[0109] The terminal in this embodiment serves as a control or adjustment system for non-electric variables. It enables the vehicle to perform category identification of obstructing objects in mining area road images through a multi-task road detection and segmentation model with feature enhancement and feature purification functions during operation. Furthermore, it identifies the shape and position of obstructing objects based on the mapping relationship between lidar point cloud data and mining area road images. Ultimately, it achieves accurate identification of obstructing objects that are difficult to detect on mining area roads, ensuring the accuracy of autonomous driving decision-making and planning for mining trucks.

[0110] This application also provides a computer device, such as... Figure 9 As shown, the computer device 900 includes a processor 901 and a memory 902. The memory 902 stores a program or instruction that can run on the processor 901. When the program or instruction is executed by the processor 901, it implements the various steps of the above-described method embodiment for identifying objects obstructing driving on the road surface in the mining area, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0111] It should be noted that the computer device 900 in this application embodiment includes the aforementioned mobile computer device and non-mobile computer device.

[0112] The memory 902 can be used to store software programs and various data. The memory 902 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 902 can include a volatile memory or a non-volatile memory, or the memory 902 can include both a volatile memory and a non-volatile memory. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 902 in the embodiments of the present application includes but is not limited to these and any other suitable types of memory.

[0113] The processor 901 can include one or more processing units; optionally, the processor 901 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 901.

[0114] The embodiments of the present application also provide a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement each process of the above-mentioned method for identifying road obstacles in a mining area, and can achieve the same technical effects. To avoid repetition, details are not described here.

[0115] The embodiment of the present application further provides a computer program product, which is stored in a storage medium, and is executed by at least one processor to implement each process of the above-mentioned method for identifying the road blocking object in a mine area, and can achieve the same technical effects. To avoid repetition, details are not described herein.

[0116] It should be noted that, in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article, or apparatus that comprises a list of elements does not only include those elements, but also includes other elements not expressly listed, or further includes elements inherent in such process, method, article, or apparatus. Without more limitations, an element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element. In addition, it should be pointed out that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing functions in the order shown or discussed, but can also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method can be performed in an order different from that described, and various steps can also be added, omitted, or combined. In addition, the features described with reference to certain examples can be combined in other examples.

[0117] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative, not restrictive, and those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims.

Claims

1. A method of identifying a travel obstructing object on a mine site road surface, characterized by, The method comprises the following steps: acquiring a mine area road surface image through an image acquisition device, and acquiring laser radar point cloud data corresponding to the mine area road surface image through a laser radar; identifying an obstacle object in the mine area road surface image according to a multi-task road surface detection and segmentation model constructed; acquiring target laser radar point cloud data corresponding to the obstacle object according to a preset calibration relationship between the image acquisition device and the laser radar and the laser radar point cloud data; determining the shape and position of the obstacle object in a three-dimensional space according to the target laser radar point cloud data; wherein the multi-task road surface detection and segmentation model comprises a feature extraction structure, a feature purification structure and a multi-task prediction structure, the feature extraction structure is used to extract features in the mine area road surface image, and the extracted features are subjected to feature enhancement processing, outputting first features, the feature purification structure is used to purify the first features, outputting second features, and the multi-task prediction structure is used to predict and identify the second features, outputting detection and segmentation results of the obstacle object; the feature extraction structure comprises a backbone network, a first feature enhancement module, a second feature enhancement module, a first fusion module, a second fusion module and a splicing module; wherein the backbone network generates a plurality of first layer features according to the mine area road surface image; the first feature enhancement module subjects a first deepest layer feature in the plurality of first layer features to feature enhancement processing, outputting a first enhanced feature; the first fusion module fuses the plurality of first layer features using the first enhanced feature, outputting a plurality of second layer features; the second feature enhancement module subjects a second deepest layer feature in the plurality of second layer features to feature enhancement processing, outputting a second enhanced feature; the second fusion module fuses the plurality of second layer features using the second enhanced feature, outputting a plurality of third layer features; and the splicing module splices and combines the plurality of third layer features, outputting the first features; the feature purification structure comprises a third shortcut connection structure, a channel attention module, a spatial attention module, a fifth fusion module, a sixth fusion module and a seventh fusion module; wherein the third shortcut connection structure directly transmits the first features, the channel attention module subjects the first features to feature purification processing in a channel dimension, outputting first purified features, the spatial attention module subjects the first features to feature purification processing in a spatial dimension, outputting second purified features, the fifth fusion module fuses the first features transmitted by the third shortcut connection structure and the first purified features, outputting first purified fusion features, the sixth fusion module fuses the first features transmitted by the third shortcut connection structure and the second purified features, outputting second purified fusion features, and the seventh fusion module fuses the first purified fusion features and the second purified fusion features, outputting the second features.

2. The method according to claim 1, characterized in that, The driving-obstructing object includes at least one of the following: falling rocks, ruts, and water pools.

3. The method of claim 1, wherein, The first layer features include a first sub-feature, a second sub-feature, a third sub-feature, and a fourth sub-feature, wherein the fourth sub-feature is the first deepest layer feature; the second layer features include a fifth sub-feature, a sixth sub-feature, and a seventh sub-feature, wherein the fifth sub-feature is the second deepest layer feature; and the third layer features include an eighth sub-feature, a ninth sub-feature, and a tenth sub-feature. The feature extraction structure further includes a plurality of convolution modules, the first fusion module includes a first sub-fusion module, a second sub-fusion module, and a third sub-fusion module, and the second fusion module includes a fourth sub-fusion module, a fifth sub-fusion module, and a sixth sub-fusion module. The first sub-feature, the second sub-feature, the third sub-feature, and the fourth sub-feature are respectively subjected to convolution processing by one of the convolution modules, the first feature enhancement module performs feature enhancement processing on the fourth sub-feature after convolution processing, and outputs the first enhanced feature. The first sub-fusion module performs fusion processing on the first enhanced feature and the fourth sub-feature after convolution processing, and outputs the fifth sub-feature. The second sub-fusion module performs fusion processing on the fifth sub-feature after sampling operation and the third sub-feature after convolution processing, and outputs the sixth sub-feature. The third sub-fusion module performs fusion processing on the sixth sub-feature after sampling operation and the second sub-feature after convolution processing, and outputs the seventh sub-feature. The fifth sub-feature, the sixth sub-feature, and the seventh sub-feature are respectively subjected to convolution processing by one of the convolution modules, the second feature enhancement module performs feature enhancement processing on the fifth sub-feature after convolution processing, and outputs the second enhanced feature. The fourth sub-fusion module performs fusion processing on the second enhanced feature and the fifth sub-feature after convolution processing, and outputs the eighth sub-feature. The fifth sub-fusion module performs fusion processing on the eighth sub-feature after sampling operation and the sixth sub-feature after convolution processing, and outputs the ninth sub-feature. The sixth sub-fusion module performs fusion processing on the ninth sub-feature after sampling operation and the seventh sub-feature after convolution processing, and outputs the tenth sub-feature. The ninth sub-feature and the tenth sub-feature are respectively subjected to convolution processing by one of the convolution modules, and the concatenation module concatenates and merges the eighth sub-feature, the ninth sub-feature after convolution processing, and the tenth sub-feature, and outputs the first feature.

4. The method of claim 1, wherein The first feature enhancement module or the second feature enhancement module includes a third fusion module, and a first shortcut connection structure, a first dilated convolution layer, and a second dilated convolution layer connected in parallel, wherein the dilation rate of the first dilated convolution layer is not equal to the dilation rate of the second dilated convolution layer.

5. The method of claim 1, wherein The first feature enhancement module or the second feature enhancement module comprises a fourth fusion module, a second shortcut connection structure, a standard convolution layer, a third hollow convolution layer and a fourth hollow convolution layer connected in parallel, wherein the dilation rate of the third hollow convolution layer is not equal to the dilation rate of the fourth hollow convolution layer.

6. The method according to any one of claims 1 to 5, characterized in that, According to the preset calibration relationship between the image acquisition device and the laser radar, the laser radar point cloud data, the target laser radar point cloud data corresponding to the driving obstacle object is obtained, comprising: According to the preset calibration relationship between the image acquisition device and the laser radar, the laser radar point cloud data is projected to the mine area road surface image, and the mapping relationship between the laser radar point cloud data and the pixel points of the mine area road surface image is obtained; Based on the mapping relationship, the target laser radar point cloud data corresponding to the driving obstacle object is obtained.

7. The method according to any one of claims 1 to 5, characterized in that, The mine area road surface image is obtained by the image acquisition device, comprising: The mine area image is collected by the image acquisition device, and the mine area road surface image in the mine area image is obtained by cutting the mine area image.

8. The method according to any one of claims 1 to 5, characterized in that, Before the mine area road surface image is obtained by the image acquisition device and the laser radar point cloud data corresponding to the mine area road surface image is obtained by the laser radar, further comprising: The road surface sample image is collected by the image acquisition device, and the shape, position and category of the sample object in the road surface sample image are labeled; According to the labeled road surface sample image, the model is trained to construct the multi-task road surface detection and segmentation model.

Images