A safe navigation method for mine auxiliary transport vehicles based on real-time scene analysis

By using sensors such as lidar, RGB camera and UWB tags to collect data in the mine environment, and designing technologies such as image and point cloud branch feature extraction modules, real-time analysis and safe navigation control of the mine environment are achieved, and the mine environment is solved The problem of limited perception capabilities of autonomous driving improves the safety and navigation efficiency of vehicles.

CN118548896BActive Publication Date: 2024-11-08CHINA UNIV OF MINING & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410743272.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-11
Publication Date
2024-11-08
Estimated Expiration
2044-06-11

AI Technical Summary

Technical Problem

Factors such as dust, high humidity and dark light in the mine environment limit the perception ability of end-to-end autonomous driving technology, affecting the reliability and safety of the autonomous driving system.

Method used

A safe navigation method for mine assisted transport vehicles based on real-time scene analysis was designed. By installing lidar, RGB camera and UWB tags, combined with IMU sensors, data is collected and preprocessed, and image and point cloud branch feature extraction modules and images are used. Feature conversion module, BEV scene analysis module and multi-branch continuous control module realize real-time analysis and safe navigation control of the mine environment.

Benefits of technology

It improves the safety control capabilities of mine vehicles in autonomous driving environments, ensures that vehicles can accurately navigate and drive safely in complex mine environments, and overcomes the perception limitations of traditional autonomous driving technology in uneven dust and light environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118548896B_ABST
    Figure CN118548896B_ABST
Patent Text Reader

Abstract

A safe navigation method for mine auxiliary transportation vehicles based on real-time scene analysis. First, the present invention designs a feature extraction module of image and point cloud branches, and introduces the CBAM method to focus on extracting key features in the perception information; secondly, in view of the feature mismatch problem between image and point cloud information due to modal differences, an image feature conversion module is designed to achieve the unification and fusion of feature space, and enhance the global perception of the mine environment; thirdly, in view of the challenge of the lack of GPS positioning information in the mine tunnel, the UWB positioning module is deployed in the key area of ​​vehicle control, and the overall topological map of the tunnel is constructed to achieve accurate planning of the vehicle navigation path; finally, in view of the safety and stability requirements of the tunnel working conditions, a continuous control module with multiple branches is designed, multi-step prediction is performed through the corresponding navigation instructions, and the safety of the execution control of the underground special auxiliary transportation vehicle is ensured through the underlying control detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a mine transport vehicle navigation method, in particular to a mine auxiliary transport vehicle safety navigation method based on real-time scene analysis, and belongs to the technical field of mine vehicle navigation. Background Art

[0002] At present, safety accidents in the auxiliary transportation link of mines account for more than 40% of mine electromechanical accidents, which is one of the biggest shortcomings in the intelligent construction and safe production of mines. In view of this, the application of unmanned transportation technology in underground special equipment is not only the key to promoting the construction of intelligent mines, but also an effective way to solve the problem of safe production in mines.

[0003] With the rapid advancement of artificial intelligence technology, the application of deep learning in end-to-end autonomous driving technology has achieved remarkable results. End-to-end autonomous driving technology, with its high computing stability and logic, indicates that once it is put into practical use, it will greatly improve the efficiency of mine transportation.

[0004] However, due to the special characteristics of the mine environment, such as high dust, high humidity, low light and uneven tunnel surface, the perception ability of end-to-end autonomous driving technology is greatly challenged. These factors limit the sensor's ability to perceive the environment, thus affecting the reliability and safety of the autonomous driving system. Summary of the invention

[0005] The purpose of the present invention is to provide a safe navigation method for mine auxiliary transportation vehicles based on real-time scene analysis, which can realize real-time analysis of mine environment scenes and design safer and more effective navigation control strategies, thereby improving the safety control capability of mine vehicles for automatic driving in mine environments.

[0006] In order to achieve the above object, the present invention provides a mine auxiliary transportation vehicle safe navigation method based on real-time scene analysis, comprising the following steps:

[0007] S1: Install lidar, RGB camera, UWB tag and IMU sensor on the underground auxiliary transport vehicle to collect the autonomous driving data set of the transport vehicle in the mine tunnel scene, preprocess the autonomous driving data set to obtain preprocessed image / point cloud data, and manually annotate the scene analysis label for autonomous driving model training;

[0008] S2: In order to solve the problem of limited perception caused by factors such as high-density dust, humidity and uneven lighting in the mine tunnel environment, the present invention designs a feature extraction module for the image branch and a feature extraction module for the point cloud branch, respectively, to extract multi-scale image features from the pre-processed image / point cloud data, wherein the multi-scale image features include scene features related to driving around the vehicle;

[0009] S3: In order to solve the feature mismatch problem between image and point cloud information due to modality differences, the present invention adopts an image feature conversion module to map the extracted multi-scale image features to the same feature space as the point cloud features, and performs channel-by-channel cross-modality feature fusion to obtain multiple sets of fused features;

[0010] S4: In order to solve the problem of complex road conditions and difficult to understand environment in the mine tunnel environment, the present invention designs a BEV scene analysis module for analyzing multiple sets of fusion features and generating real-time scene analysis based on BEV space;

[0011] S5: In order to solve the problem that the lack of GPS signals in the mine tunnels cannot provide effective navigation guidance for special auxiliary transport vehicles, the present invention constructs a global topological map and a pre-planned vehicle navigation path by installing UWB positioning modules in key areas of the tunnels, thereby setting navigation instructions for each positioning node and generating navigation instructions necessary for vehicle navigation;

[0012] S6: In order to improve the safety and stability of special auxiliary transport vehicles in mine tunnels, the present invention designs a continuous control module with a multi-branch architecture, and according to the set navigation instructions, uses the corresponding control branch to continuously predict vehicle control using the GRU unit under the real-time scene analysis results and the input of the vehicle's historical driving information, thereby obtaining the corresponding predicted vehicle control instructions;

[0013] S7: To ensure the effectiveness of vehicle control in narrow lane environments, the present invention converts the generated predicted vehicle control instructions into vehicle prediction coordinates and maps them into a real-time scene analysis diagram to determine whether the vehicle's current output control is safe, and ultimately controls the vehicle to perform special auxiliary transportation tasks.

[0014] In step S2 of the present invention, the feature extraction module of the image branch and the feature extraction module of the point cloud branch are used to extract the features of the driving-related scenes around the vehicle. The feature extraction module of the image branch includes a ResNet34 image feature extraction network, a multi-scale feature aggregation network and an image parsing network, and the feature extraction module of the point cloud branch includes a point cloud feature extraction network, a multi-scale feature aggregation network and an image parsing network. The specific steps are as follows:

[0015] Step S21: The input of the ResNet34 image / point cloud feature extraction network is the preprocessed image / point cloud data. Its calculation process is to generate a multi-scale feature map of the image / point cloud by retaining the different scale feature maps extracted by the last four convolution groups in ResNet34;

[0016] Step S22: The multi-scale feature aggregation network is composed of CBAM (Convolutional Block Attention Module) and a convolutional network. Its input is the generated image / point cloud multi-scale feature map. Its calculation process is to weight the features through four groups of CBAM, unify the feature map scale and compress the feature dimension by using four groups of convolutional layers, and aggregate them into a unified scale image / point cloud feature map;

[0017] Step S23: The generated unified scale image / point cloud feature map is feature information of a group of key objects. Steps S21 and S22 are repeated four times to generate four groups of image / point cloud features of key objects in the driving scene. The image / point cloud features of the four groups of key objects are image / point cloud features of lanes, lane lines, dynamic and static obstacles, and background.

[0018] Step S24: The image parsing network is composed of a deconvolution network and an upsampling layer. As described in step S23, after generating four sets of image features, they are input into the image parsing network to generate an image parsing result from a first-person perspective.

[0019] The image feature conversion module for cross-modal feature fusion in step S3 of the present invention comprises the following steps:

[0020] Step S31: The image feature conversion module is composed of four groups of convolutional networks, and its input is four groups of image features. The calculation process is to convert the image feature map scale to the same scale as the point cloud feature through the upsampling layer, and use the convolution layer to convert the first-person perspective image feature to the same BEV feature space as the point cloud feature;

[0021] Step S32: After being converted, the four sets of image features are concatenated with the four sets of point cloud feature information in the channel dimension to generate four sets of cross-modal fused lane, lane line, dynamic and static obstacle and background feature information.

[0022] Step S4 of the present invention is used to generate a BEV scene analysis module for real-time scene analysis, and includes the following steps:

[0023] Step S41: The BEV scene parsing module is composed of four groups of deconvolution networks and upsampling layers. Its input is four groups of cross-modal fusion features. Its calculation process is to restore the four groups of cross-modal fusion features to the size of the data set label map through four groups of parsing networks respectively composed of deconvolution and upsampling, and obtain a four-channel parsing result;

[0024] Step S42: Concatenate the features of the four-channel analysis results in the channel dimension, use the softmax function to convert the values ​​at the same position in all channels into the probabilities corresponding to each category, and use the argmax method to find the category with the highest probability in each position to generate real-time scene analysis.

[0025] The path planning and positioning method for generating navigation instructions in step S5 of the present invention uses local UWB positioning and global path planning, and the specific steps are as follows:

[0026] Step S51: Deploy UWB positioning modules at the forks in the mine tunnels, coal mining faces and other key locations to form an overall topological map of the trackless mining vehicle operation scene. When performing navigation tasks, use the Dijkstra's algorithm to plan the global optimal path;

[0027] Step S52: using the UWB positioning module to obtain the positioning information of the vehicle in the topological map, and generating the navigation instructions of the vehicle according to the planned optimal path and the current position of the vehicle, the navigation instructions including the vehicle turning left, turning right, lane following, etc.;

[0028] Step S53: The vehicle repeats step S52 at each key node where the navigation direction needs to be determined and performs the task according to the specified navigation direction.

[0029] The continuous control module of the multi-branch architecture for predicting vehicle control in step S6 of the present invention, the speed prediction network, and the control prediction network, specifically include the following steps:

[0030] Step S61: The speed prediction network is composed of a fully connected network, and its input is the historical speed and throttle control information. The calculation process is to input the historical speed and throttle information into the fully connected network, so as to predict the vehicle speed after the current throttle control;

[0031] Step S62: The control prediction network is composed of a GRU unit, a convolutional network, and a fully connected network. Its input is the historical speed, throttle, steering information, the current predicted speed information, and the real-time scene analysis generated in step S42. Its calculation process is to extract the environmental features of the real-time scene analysis graph through the convolutional network as the latent vector of the GRU unit, and use the speed, throttle, and steering information as the input vector of the GRU unit to generate the control features of the next moment. After decoding by the fully connected network, the throttle and steering control of the next moment are predicted respectively;

[0032] Step S63: After generating the next moment control, repeat step S61 to predict the vehicle speed and repeat step S62 to predict the continuous control.

[0033] The bottom-level control judgment for judging whether the vehicle output is safe in step S7 of the present invention comprises the following steps:

[0034] Step S71: Calculate the vehicle's forward distance based on the predicted vehicle speed information and the fixed vehicle control interval;

[0035] Step S72: Calculate the vehicle deviation angle in proportion to the predicted vehicle steering control;

[0036] Step S73: Calculate the difference in the vehicle's coordinates based on the vehicle's forward distance and the vehicle's offset angle ;

[0037] Step S74: Mapping the calculated coordinate change difference to the real-time scene analysis map generated in step S42 to determine whether the predicted navigation control is safe and compliant;

[0038] Step S75: If the navigation control is determined to be compliant in step S74, continue to run; otherwise, emergency braking is performed.

[0039] The present invention collects an autonomous driving navigation data set in a mine tunnel, including preprocessed RGB images, laser point clouds, vehicle steering, vehicle throttle, vehicle speed, vehicle navigation direction, annotated first-person image analysis labels, and BEV scene analysis labels; the preprocessed RGB images and laser point clouds are input into a real-time scene perception module composed of S1, S2, and S3, and joint training is performed using the labels provided in the data set; the BEV scene analysis generated by the real-time scene perception module is input into a multi-branch continuous control module together with the navigation instructions, and supervised training is performed using the vehicle driving input provided in the data.

[0040] Compared with the prior art, the present invention first designs a feature extraction module for image and point cloud branches, and introduces the CBAM method to focus on extracting key features in the perception information; secondly, in order to solve the feature mismatch problem between image and point cloud information due to modal differences, an image feature conversion module is designed, and cross-modal feature fusion is performed through image feature space conversion, and scene analysis is performed with a more efficient fusion method, which can effectively generate real-time tunnel environment analysis maps in harsh tunnel environments with dust and uneven lighting, thereby clearly showing the perception and understanding of the surrounding environment, laying the foundation for subsequent autonomous driving navigation control; at the same time, the present invention deploys UWB positioning modules at key locations (such as coal mining faces, tunnel intersections, etc.), which can construct an overall topological map within the vehicle's working range, and based on The "Dijkstra algorithm" plans the optimal path for vehicle travel, so that in the absence of GPS positioning information, accurate navigation instructions can be given to the vehicle to successfully complete the transportation task, realizing the precise planning of the vehicle navigation path; finally, in view of the safety and stability requirements of the tunnel working conditions, the present invention adopts a multi-branch continuous control module, and sets corresponding control branches according to different navigation instructions to execute specific control strategies. In the process of special auxiliary transportation vehicles performing tasks, continuous vehicle control is generated by predicting the vehicle speed. At the same time, in view of the transportation safety and stability requirements in the mine environment, the present invention designs a special control detection to convert the vehicle control output into predicted coordinates and map them in the scene analysis diagram to evaluate whether there are potential collisions or safety hazards, thereby ensuring the stability of the vehicle transportation task. The present invention can realize real-time analysis of the mine environment scene and design a safer and more effective navigation control strategy, thereby improving the safety control capability of mine vehicles in automatic driving in the mine environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a structural diagram of the real-time scene perception module of the present invention;

[0042] Figure 2 It is a specific structural diagram of the image and point cloud branch feature extraction module of the present invention;

[0043] Figure 3 This is a structural diagram of the multi-branch continuous control module of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described below in conjunction with the accompanying drawings.

[0045] like Figure 1-Figure 3 As shown, a mine auxiliary transportation vehicle safe navigation method based on real-time scene analysis includes the following steps:

[0046] S1: Install lidar, RGB camera, UWB tag and IMU sensor on the underground auxiliary transport vehicle to collect the autonomous driving data set of the transport vehicle in the mine tunnel scene, preprocess the autonomous driving data set to obtain preprocessed image / point cloud data, and manually annotate the scene analysis label for autonomous driving model training;

[0047] S2: Designing a feature extraction module for an image branch and a feature extraction module for a point cloud branch respectively, and extracting multi-scale image features from the preprocessed image / point cloud data respectively, wherein the multi-scale image features include scene features around the vehicle and related to driving;

[0048] The feature extraction module of the image branch includes the ResNet34 image feature extraction network, the multi-scale feature aggregation network, and the image parsing network; the feature extraction module of the point cloud branch includes the point cloud feature extraction network, the multi-scale feature aggregation network, and an additional convolutional network;

[0049] The ResNet34 image / point cloud feature extraction network generates multi-scale feature maps of images / point clouds by retaining the different scale feature maps extracted by the last four convolution groups in ResNet34 (i.e. retaining the feature maps of the "64, 128, 256, 512" size channels), capturing the low-frequency to high-frequency features in the mine tunnel scene; its function is to target the special harsh environment of mine tunnels to obtain richer scene features for further processing of subsequent features.

[0050] The multi-scale feature aggregation network includes four CBAMs (Convolutional Block Attention Modules) and four convolutional networks. Its input is the generated image / point cloud multi-scale feature map. Its calculation process is to weight the features through four groups of CBAMs, unify the feature map scales and compress the feature dimensions using four groups of convolutional layers, and aggregate them into a unified scale image / point cloud feature map. Its function is to first use the CBAM network to weight the features of each scale, and then use the convolutional network to unify the feature scales and extract the weighted effective key information.

[0051] The generated unified scale image / point cloud feature map is the feature information of a set of key objects. This is repeated four times to generate a total of four sets of image / point cloud features of key objects in the driving scene. The image / point cloud features of the four sets of key objects are the image / point cloud features of lanes, lane lines, dynamic and static obstacles, and background.

[0052] The image parsing network includes a deconvolution network and an upsampling layer. The four sets of image features are input into the image parsing network to generate the image parsing results from the first-person perspective. Its function is to guide the image feature processing branch to pay more attention to the extraction of driving features in the acquired dense information by parsing the lanes, lane lines, and dynamic and static obstacle objects in the first-person perspective image features.

[0053] The additional convolutional network of the point cloud branch performs channel compression on the sparse point cloud features, which can improve the perceived feature quality and computational efficiency. Feature processing is performed on the image branch and the point cloud branch respectively. The specific process includes the following steps:

[0054] S21, the image branch extracts the feature information of the vehicle front image collected in the mine tunnel, as follows:

[0055] S211, input the image after the preprocessing step into the ResNet34 image feature extraction network to generate four sets of feature maps of different sizes;

[0056] S212, inputting the four groups of feature maps of different sizes into the multi-scale feature aggregation network of the image branch, and compressing and aggregating the four groups of feature maps of different sizes into a group of image perception feature blocks through feature weighting corresponding to the four groups of CBAMs and feature screening and scale conversion by the convolution network;

[0057] S213. A total of four multi-scale feature aggregation networks with the same structure are used to perform four S212 feature processing steps [four groups of feature maps of different sizes are input into the multi-scale feature aggregation network of the image branch together, and the four groups of feature maps of different sizes are compressed and aggregated into a group of image perception feature blocks through feature weighting corresponding to the four groups of CBAMs and feature screening and scale conversion by the convolution network], and a total of four groups of image perception feature blocks are generated, which respectively represent the key features of the lanes, lane lines, dynamic and static obstacles and background in the image perception information.

[0058] S22. The feature extraction process of the point cloud branch is similar to that of the image branch. The specific process includes the following steps:

[0059] S221, input the point cloud after the preprocessing step into the ResNet34 point cloud feature extraction network to generate four sets of feature maps of different sizes;

[0060] S222, using the multi-scale feature aggregation network of the point cloud branch, repeating steps S212 [combining the four groups of feature maps of different sizes into the multi-scale feature aggregation network of the image branch, and compressing and aggregating the four groups of feature maps of different sizes into a group of point cloud perception feature blocks through feature weighting corresponding to the four groups of CBAMs and feature screening and scale conversion by the convolution network] and S213 [using a total of four multi-scale feature aggregation networks of the same structure to perform four feature processing steps, generating a total of four groups of point cloud perception feature blocks, respectively representing the key features of lanes, lane lines, dynamic and static obstacles, and background in the point cloud perception information] to generate four groups of point cloud perception feature blocks, respectively representing the key features of lanes, lane lines, dynamic and static obstacles, and background in the point cloud perception information;

[0061] S223. Use the additional convolutional network of the point cloud branch to further compress the feature channels of the four groups of point cloud perception feature blocks.

[0062] S23, inputting the four groups of image perception feature blocks generated in step S21 into the corresponding four groups of deconvolution networks in the image parsing network, parsing the four types of objects, namely lanes, lane lines, dynamic and static obstacles, and background, through the upsampling layer combined in the deconvolution network, and restoring them to the first-person perspective image;

[0063] Through feature processing of image and point cloud branches, four groups of perceptual feature blocks based on two perceptual modalities were generated respectively, and the image parsing network was used to obtain the parsing results of the image perceptual feature blocks. By training the image branch, the image branch can be guided to extract more effective features from dense image information.

[0064] S3: Design an image feature conversion module to map the extracted multi-scale image features to the same feature space as the point cloud features, and perform channel-by-channel cross-modal feature fusion to obtain multiple sets of fused features; design a BEV scene parsing module, and use the BEV scene parsing module to parse multiple sets of fused features to generate real-time scene parsing based on the BEV space;

[0065] The function of the image feature conversion module of the present invention is to utilize the advantage of convolution in learning the spatial information in the feature map, convert the image features of the first-person perspective into the BEV feature space, and perform the scale conversion of the image features, so as to facilitate cross-modal fusion perception;

[0066] The image feature conversion module consists of four groups of convolutional networks. Its input is four groups of image features. Its calculation process is to convert the scale of the image feature map to the same scale as the point cloud feature through the upsampling layer. Its function is to use the advantage of convolution to learn the spatial information in the feature map, convert the image features from the first-person perspective to the BEV feature space, and perform scale conversion of the image features, so as to facilitate cross-modal fusion perception;

[0067] The BEV scene analysis module contains four sets of deconvolution networks; its function is to analyze cross-modal fusion features to achieve perception and understanding of the mine tunnel environment;

[0068] The specific process of image feature conversion, cross-modal fusion and BEV scene parsing includes the following steps:

[0069] S31, the input of the image feature conversion module is the four groups of image perception feature blocks generated in step S21, four groups of convolutional networks with the same structure are used to convert the four groups of image perception feature blocks, and the scale of each image feature block is converted to the scale of the point cloud feature block;

[0070] S32, splicing and fusing the four groups of image feature blocks and the four groups of point cloud feature blocks with the same scale in the channel dimension according to the four types of object features, to generate four groups of fused feature blocks;

[0071] S33, inputting the four groups of fused feature blocks into the four groups of deconvolution networks of the BEV scene analysis module respectively, and using the same steps as S23, parsing the four types of objects, namely, lanes, lane lines, dynamic and static obstacles, and background, into an image in the BEV two-dimensional space;

[0072] Leveraging the collected dataset , the networks used to generate the first-person-view image analysis and the BEV scene analysis in steps S21 and S22 are jointly trained to obtain a real-time scene perception module constructed by an image branch feature extraction module, a point cloud branch feature extraction module, an image feature conversion module, and a BEV scene analysis module.

[0073] The loss functions of the two parsing graphs are the same, both of which include the dynamic weighted cross entropy loss function , mean square error loss function based on image edge gradient And the intersection-over-union loss function .

[0074] S4: Using the UWB positioning modules installed in key areas of the lane and the pre-planned vehicle navigation path, the navigation instructions for each positioning node are set to generate the navigation instructions necessary for vehicle navigation;

[0075] Taking into account that the tunnel is a closed environment and the vehicle driving route is relatively fixed, the present invention deploys UWB positioning modules at the forks and coal mining faces in the mine tunnels to form an overall topological map of the trackless mining vehicle operation scene.

[0076] The specific steps for planning and decision-making based on the overall topology map and the local UWB positioning module are as follows:

[0077] S41. The present invention sets the cost of a vehicle traveling between two nodes in proportion to the distance between adjacent nodes in the overall topological map;

[0078] S42, when performing the navigation task, using Dijkstra's algorithm to plan the global optimal path from the starting node to the ending node;

[0079] S43, the vehicle starts from the starting node, uses the UWB positioning module deployed at the node to determine the position of the vehicle in the topological map, and uses the planned path to send a specified navigation instruction to the vehicle. The navigation instruction includes the vehicle turning left, turning right, lane following, etc. The specific method is as follows:

[0080] S431: If the current node of the vehicle is connected to multiple nodes, calculate the angle between the current vehicle's driving direction based on the IMU sensor and the planned next node direction. If the angle range is Issue the left lane following navigation command, if the angle range is Issue the right lane following navigation command, if the angle range is Issue a left turn navigation command. If the angle range is Issue a right turn navigation instruction;

[0081] S432: If the vehicle is not within the positioning range of the UWB positioning module, a lane following navigation instruction is issued;

[0082] The issued navigation instructions are combined with BEV scene analysis and input into the multi-branch continuous control module , used to generate vehicle coherence control.

[0083] S5: Design a continuous control module with a multi-branch architecture. According to the set navigation instructions, the corresponding control branch uses the GRU unit to continuously predict vehicle control under the input of real-time scene analysis results and historical vehicle driving information, thereby obtaining the corresponding predicted vehicle control instructions;

[0084] The multi-branch architecture continuous control module for predicting vehicle control in step S5 of the present invention comprises a speed prediction network and a control prediction network. The speed prediction network comprises a multi-layer fully connected network, which is used to implicitly learn the dynamic characteristics of the vehicle using historical vehicle throttle control and corresponding speed information, thereby predicting the driving speed of the current vehicle under throttle control;

[0085] The control prediction network includes a convolutional network, a GRU network, and a fully connected network. Its function is to use the generated real-time scene analysis and the predicted vehicle driving information to predict the future steering and throttle control of the vehicle.

[0086] The specific steps to predict vehicle speed and generate vehicle coherent control are as follows:

[0087] S51, according to the navigation instruction generated by S4, select the corresponding control branch for decision control, as follows:

[0088] S511, if the navigation command is a left lane following navigation command, then the first control branch in the multi-branch continuous control module is selected to specifically predict this type of control;

[0089] S512: If the navigation command is a right lane following navigation command, the second control branch in the multi-branch continuous control module is selected to specifically predict this type of control;

[0090] S513: If the navigation command is a lane following navigation command, the third control branch in the multi-branch continuous control module is selected to specifically predict this type of control;

[0091] S514, if the navigation command is a left turn navigation instruction, then the fourth control branch in the multi-branch continuous control module is selected to specifically predict this type of control;

[0092] S515, if the navigation command is a right turn navigation instruction, then the fifth control branch in the multi-branch continuous control module is selected to specifically predict this type of control;

[0093] Each control branch of the multi-branch continuous control module adopts the same "speed prediction-control prediction" network architecture.

[0094] S52, splicing the historical speed of the vehicle and the throttle information together and inputting them into the speed prediction network of the corresponding control branch to generate predicted speed information;

[0095] S53, splicing the predicted vehicle speed information with the historical vehicle speed information, together with the historical steering information and the historical throttle information as an input vector and inputting it into the GRU unit in the corresponding control branch control prediction network;

[0096] S54, inputting the generated BEV real-time scene analysis graph into the convolutional network of the corresponding control branch control prediction network, generating the perception feature of the environment as a latent vector input into the GRU unit in the corresponding control branch control prediction network;

[0097] S55, using the perception feature latent vector and the vehicle driving information input vector, after one calculation by the GRU unit, generating a control feature with the same dimension as the latent vector;

[0098] S56, inputting the control feature into the fully connected network of the corresponding control branch to generate predicted vehicle throttle and steering control respectively;

[0099] S57, splicing the predicted control with the historical vehicle control information, repeating the above steps, and predicting the continuous vehicle control under the specified navigation command;

[0100] Leveraging the collected dataset , train the multi-branch continuous control module of step S5, and set the speed, control, and steering to Loss function.

[0101] S6: Convert the predicted vehicle control instructions into predicted vehicle coordinates and map them to the real-time scene analysis diagram to determine whether the current output control of the vehicle is safe, and ultimately control the vehicle to perform special auxiliary transportation tasks.

[0102] The specific steps for detecting control behavior are as follows:

[0103] S61. Calculation of predictions Average vehicle speed:

[0104]

[0105] S62, fixed time interval according to control frequency , calculate the vehicle's forward distance:

[0106]

[0107] S63. Calculate the predicted Step vehicle steering average:

[0108]

[0109] S64, converting the vehicle steering average value into a vehicle offset angle:

[0110]

[0111] S65, combined with the vehicle's forward distance and the vehicle deviation angle , calculate the coordinate change of the vehicle under the control command:

[0112]

[0113] S66, after obtaining the predicted coherent control, The coordinates are mapped to the BEV scenario analysis diagram to determine whether there are potential hazards such as collisions. If the navigation control is safe, the spline interpolation method is used to calculate the transition smooth control instructions to maintain operation, otherwise emergency braking is performed.

[0114] An embodiment of the present invention is given below

[0115] Step S1: Collect the data set required for autonomous driving training by manually operating a special underground auxiliary transportation vehicle , dataset Including 3D laser point cloud data collected by LiDAR, RGB image data collected by RGB camera, and manually set navigation instructions, driving control instructions, and scene analysis tags;

[0116] The dataset The specific process of preprocessing and labeling includes the following steps:

[0117] S11, preprocessing each frame of laser point cloud data collected during vehicle transportation, specifically including the following steps:

[0118] S111. Crop the 3D laser point cloud data and keep the data of 16m in front of the vehicle and 8m on the left and right. ;

[0119] S112. Create two empty three-dimensional Numpy arrays: ;

[0120] S113, Yes Each laser point cloud data is separated and the laser point cloud data height Data less than 1.5 is stored Otherwise, it is stored in ;

[0121] S114. Create a A two-channel grid, each grid pixel represents Point cloud perception of square meter area;

[0122] S115, Basis The laser points Coordinates, the laser points are stored in the corresponding spatial areas of the two channel grids, that is, and ;

[0123] S116, sequentially detect the number of point clouds existing in each grid pixel, limit the number to a maximum of 4, and perform normalization processing on the network pixels to control the value of each pixel to be within ;

[0124] S117, obtain the two-channel point cloud data of each frame under the BEV perspective after preprocessing ;

[0125] S12, each frame of RGB image data collected by the RGB camera during the vehicle transportation process , this embodiment collects RGB images with a field of view angle of 120°, and performs image edge cropping to remove edge distortion data to obtain image data with a resolution of 160×120×3;

[0126] Preprocess the collected laser point cloud data to generate 256×256×2 two-channel point cloud data;

[0127] S13, the vehicle speed corresponding to each frame image and point cloud perception , Vehicle Steering , Vehicle throttle And navigation instructions Save as vehicle driving information ,Right now ,in, , , ;

[0128] S14, analyzing and labeling the pre-processed image data, wherein the lanes, lane lines, dynamic and static obstacles, and background categories are labeled as ,get ;

[0129] S15, perform analytical annotation of the BEV two-dimensional plane space on the driving area covered by the pre-processed point cloud data, wherein lanes, lane lines, dynamic and static obstacles, and background categories are respectively annotated as ,get ;

[0130] Summarize the data collected in the same frame, arrange the data sets frame by frame, and complete the data set collection, that is, .

[0131] S2. Combination Figure 1 and Figure 2 , the collected 160×120×3 size images and 256×256×2 size point clouds are input into the real-time scene perception module to generate image first-person perspective analysis and BEV scene analysis, which includes the following steps:

[0132] S21. Use ResNet34 to extract features from image and point cloud information respectively:

[0133] Input the 160×120×3 size image into the ResNet34 image feature extraction network to generate four sets of image feature maps of different sizes: "(64, 40, 30), (128, 20, 15), (256, 10, 8), and (512, 5, 4)";

[0134] Input the 256×256×2 point cloud data into the ResNet34 point cloud feature extraction network to generate four groups of point cloud feature maps of different sizes: "(64, 64, 64), (128, 32, 32), (256, 16, 16), and (512, 8, 8)";

[0135] S22, respectively send four groups of image feature maps of different sizes and four groups of point cloud feature maps of different sizes into the multi-scale feature aggregation network to obtain key features of lanes, lane lines, dynamic and static obstacles, and background, as follows:

[0136] S221, converting the four groups of image feature maps of different sizes into four groups of feature maps of different sizes of "(16, 24, 4), (32, 24, 4), (64, 24, 4), and (128, 24, 4)" through feature weighting of CBAM and feature screening and scale conversion of the convolutional network, and compressing and aggregating them into a group of image perception feature blocks (240, 24, 4), repeating the above feature screening and aggregation method four times, and obtaining a total of four groups of image perception feature blocks;

[0137] S222, converting the four groups of point cloud feature maps of different sizes into four groups of feature maps of different sizes of "(16, 16, 16), (32, 16, 16), (64, 16, 16), (128, 16, 16)" through feature weighting of CBAM and feature screening and scale conversion of the convolutional network, and compressing and aggregating them into a group of point cloud perception feature blocks (240, 16, 16), and then using an additional convolutional network to compress the point cloud perception feature blocks into (80, 16, 16), repeating the above feature screening and aggregation method four times, and obtaining a total of four groups of point cloud perception feature blocks;

[0138] S23, four groups of image perception feature blocks, and four groups of point cloud perception feature blocks correspond to the key features of lanes, lane lines, dynamic and static obstacles, and background in the scene respectively.

[0139] S24, input the generated four sets of image perception feature blocks into the corresponding four sets of deconvolution networks in the image parsing network, and parse the lanes, lane lines, dynamic and static obstacles, and background through the upsampling layer combined in the deconvolution network to generate a four-channel result with the same resolution as the original input, i.e., 160×120×4, and then use the application After the function calculates the probability for all positions, Method to find the category with the highest probability in each position , thereby converting the channel feature maps for different categories into a single label map :

[0140]

[0141] in, Indicated in Channel Predicted value under position;

[0142] S3, using the image feature conversion module to map the four groups of image perception feature blocks to the same BEV feature space as point cloud perception, and perform scale conversion to obtain four groups of (160, 16, 16) image perception feature blocks, thereby performing feature splicing on the feature blocks for the same key features in the image and point cloud to obtain four groups of (240, 16, 16) fusion feature blocks;

[0143] S4, input the four groups of (240, 16, 16) fusion feature blocks into the BEV scene parsing module to generate a four-channel result with a resolution of 160×160, and use Functions and The method finally calculates the real-time scene analysis results based on BEV space;

[0144] Leveraging the collected dataset ,right Figure 1 The real-time scene perception module shown is trained, and the specific loss function is set as follows:

[0145] The loss function is designed as three parts. The first is the dynamic weighted cross entropy loss function. , by introducing dynamic weights, the edge area of ​​each parsing result is dynamically calculated:

[0146]

[0147] in, is the dataset label; Represents 2D average pooling with a kernel size of 31, a stride of 1, and a padding of 15.

[0148] After calculating the dynamic weights for each segmentation edge, the first part of the scene parsing loss is calculated:

[0149]

[0150] in, and are the height and width of the generated image respectively; is the category probability output by the model; is the generated dynamic weight; is the category weight set by the present invention (the lane line category weight is set to 5.0, the obstacle category weight is set to 2.0, and the other category weights are 1.0);

[0151] The second is the mean square error loss function based on the image edge gradient , the Sobel operator is used to improve the accuracy of edge prediction:

[0152]

[0153] in, and The distribution represents the gradient operation in the horizontal and vertical directions using the Sobel operator;

[0154] and Represent the dataset labels and model output results respectively.

[0155] Finally, the intersection-over-union loss function , encouraging the model to generate predictions that are structurally coherent overall:

[0156]

[0157] The overall loss function used to train the real-time scene perception module is:

[0158]

[0159] in, , , They are the hyperparameters for balancing the three losses and are set to 0.5, 1.0, and 0.5 respectively.

[0160] S5. Use the UWB positioning module to build a global topological map of the lane, and set the cost of the vehicle traveling between two nodes in proportion to the distance between adjacent nodes. Use the "Dijkstra algorithm" to plan the vehicle's predefined driving trajectory, so as to obtain navigation instructions when the vehicle travels to each UWB positioning device, including "left lane follow, right lane follow, lane follow, turn left, turn right".

[0161] S6, such as Figure 3 As shown in Figure 2, the specific steps of using the multi-branch continuous control module to predict vehicle control are as follows:

[0162] S61. According to the navigation instruction, select a corresponding control branch to perform control prediction.

[0163] S62: The vehicle's historical speed at four moments ,accelerator And turn Information 0 together, we get:

[0164]

[0165]

[0166]

[0167] Will and Input speed prediction network to generate predicted speed information ;

[0168] S63: The predicted vehicle speed information Combined with the historical vehicle speed information, we get:

[0169]

[0170] Will Together with as well as They are sent together as input vectors to the GRU unit in the control prediction network;

[0171] S64, inputting the generated BEV real-time scene analysis graph into the convolutional network in the control prediction network, generating the perception features of the environment as latent vectors and inputting them into the GRU unit;

[0172] S65, using the perception feature latent vector and the vehicle driving information input vector, after a calculation of the GRU unit, a control feature with the same dimension as the latent vector is generated, thereby using the fully connected network to generate the predicted Vehicle throttle and steering control at all times;

[0173] S66, Repeat spliced ​​to the step of generating predictions, the final output is , and It is expressed as:

[0174]

[0175]

[0176]

[0177] S67. Using the collected data sets ,right Figure 3 The multi-branch continuous control module shown is trained, and the specific loss function is set as follows:

[0178]

[0179] in, , , are the labels of throttle, steering, and speed in the dataset, respectively. , , They are the throttle, steering and speed results predicted by the multi-branch continuous control module. The hyperparameters are set to 5, 20, and 1 respectively.

[0180] S7, BEV scenario analysis diagram describes the driving environment 16m in front of the vehicle and 8m on the left and right, converting the coherent control into the vehicle prediction coordinates Mapped to the graph, to determine whether the vehicle has potential safety risks such as collision. If the navigation control is safe, the spline interpolation method is used to calculate the transition smooth control instructions to keep running, otherwise emergency braking is taken. The present invention can realize the safe automatic driving of underground special auxiliary transportation vehicles in the mine tunnel environment and improve the auxiliary transportation efficiency.

[0181] The training of each of the above neural network modules uses the Adam gradient descent algorithm to update its learnable parameters.

[0182] The activation functions used in the convolutional network, deconvolutional network, and fully connected network in the above neural network modules are all Leaky-ReLU, the convolutional networks all contain Batch-Normalization, and the upsampling layers used in the deconvolutional network are all bilinear interpolation.

Claims

1. A mine auxiliary transport vehicle safe navigation method based on real-time scene analysis, characterized in that: The following steps are involved: S1: Install lidar, RGB camera, UWB tag, and IMU sensor on underground auxiliary transport vehicles to collect autonomous driving datasets of transport vehicles in mine tunnel scenes. The autonomous driving datasets are preprocessed to obtain preprocessed image / point cloud data, and scene parsing labels are manually annotated for autonomous driving model training. S2: Designing feature extraction modules for the image branch and the point cloud branch, respectively, to extract multi-scale image features from the preprocessed image / point cloud data. The multi-scale image features include driving-related scene features around the vehicle. S3: Design an image feature conversion module to map the extracted multi-scale image features to the same feature space as the point cloud features, and perform channel-by-channel cross-modal feature fusion to obtain multiple sets of fused features; S4: Design a BEV scene parsing module to parse multiple sets of fused features and generate real-time scene parsing based on the BEV space; S5: Using the UWB positioning modules installed in key areas of the lane and the pre-planned vehicle navigation path, the navigation instructions for each positioning node are set to generate the necessary navigation instructions for vehicle navigation; S6: Design a continuous control module with a multi-branch architecture. According to the set navigation instructions, the corresponding control branch uses the real-time scene analysis results and the GRU unit to continuously predict vehicle control under the input of the vehicle's historical driving information, thereby obtaining the corresponding predicted vehicle control instructions; S7: Convert the predicted vehicle control command into the predicted vehicle coordinates and map them to the real-time scene analysis graph to determine whether the vehicle's current output control is safe, and ultimately control the vehicle to perform special auxiliary transportation tasks; In step S2, the feature extraction module of the image branch includes a ResNet34 image feature extraction network, a multi-scale feature aggregation network, and an image parsing network. The feature extraction module of the point cloud branch includes a point cloud feature extraction network, a multi-scale feature aggregation network, and an image parsing network. The specific steps are as follows: Step S21: The input of the ResNet34 image feature extraction network is the preprocessed image data, and its calculation process is to generate a multi-scale feature map of the image by retaining the different scale feature maps extracted by the last four convolution groups in ResNet34; the input of the ResNet34 point cloud feature extraction network is the preprocessed point cloud data, and its calculation process is to generate a multi-scale feature map of the point cloud by retaining the different scale feature maps extracted by the last four convolution groups in ResNet34; Step S22: The multi-scale feature aggregation network is composed of CBAM and convolutional network. Its input is the generated image multi-scale feature map and point cloud multi-scale feature map. Its calculation process is to use four sets of CBAM to perform feature weighting, and then use four sets of convolutional layers to unify the feature map scale and compress the feature dimension to aggregate into a unified scale image feature map and point cloud feature map; Step S23: The generated unified-scale image feature map and point cloud feature map are feature information of a group of key objects. Steps S21 and S22 are repeated four times to generate a total of four groups of image / point cloud features of key objects in the driving scene. The four groups of image / point cloud features of key objects are image / point cloud features of lanes, lane lines, dynamic and static obstacles, and background. Step S24: The image parsing network is composed of a deconvolution network and an upsampling layer. As described in step S23, after generating four sets of image features, they are input into the image parsing network to generate the first-person perspective image parsing results. The image feature conversion module for cross-modal feature fusion in step S3 includes the following steps: Step S31: The image feature conversion module is composed of four sets of convolutional networks. Its input is four sets of image features. Its calculation process is to convert the image feature map scale to the same scale as the point cloud features through the upsampling layer, and use the convolution layer to convert the first-person perspective image features to the same BEV feature space as the point cloud features; Step S32: After the four sets of image features are converted, they are concatenated with the four sets of point cloud feature information in the channel dimension to generate four sets of lane, lane line, dynamic and static obstacle, and background feature information after cross-modal fusion.

2. A mine auxiliary transport vehicle safe navigation method based on real-time scene analysis according to claim 1, characterized in that: Step S4 is used to generate a BEV scene analysis module for real-time scene analysis, which includes the following steps: Step S41: The BEV scene parsing module includes four sets of deconvolution networks and upsampling layers. Its input is four sets of cross-modal fusion features. Its calculation process is to restore the four sets of cross-modal fusion features to the size of the dataset label map through four sets of parsing networks composed of deconvolution and upsampling, and obtain a four-channel parsing result; Step S42: The four-channel analysis results are concatenated in the channel dimension. The softmax function is used to convert the values ​​at the same position in all channels into probabilities corresponding to each category. The argmax method is used to find the category with the highest probability in each position to generate real-time scene analysis.

3. The mine auxiliary transport vehicle safe navigation method based on real-time scene analysis according to claim 1 is characterized in that: The path planning and positioning method for generating navigation instructions in step S5 uses local UWB positioning and global path planning, and the specific steps are as follows: Step S51: Deploy UWB positioning modules at the intersections and coal mining faces of the mine tunnels to form an overall topological map of the trackless mining vehicle operation scene. When performing the navigation task, the "Dijkstra algorithm" is used to plan the global optimal path; Step S52: Using the UWB positioning module to obtain the vehicle's positioning information in the topological map, and generating vehicle navigation instructions based on the planned optimal path and the vehicle's current location. The navigation instructions include turning left, turning right, and lane following. Step S53: The vehicle repeats step S52 at each key node where the navigation direction needs to be determined and performs the task according to the specified navigation direction.

4. The mine auxiliary transport vehicle safe navigation method based on real-time scene analysis according to claim 3 is characterized in that: The continuous control module of the multi-branch architecture for predicting vehicle control in step S6, the speed prediction network, and the control prediction network, are specifically configured as follows: Step S61: The speed prediction network includes a fully connected network, whose input is the historical speed and throttle control information. The calculation process is to input the historical speed and throttle information into the fully connected network to predict the vehicle speed after the current throttle control; Step S62: The control prediction network includes a GRU unit, a convolutional network, and a fully connected network. Its input is the historical speed, throttle, and steering information, the current predicted speed information, and the real-time scene analysis generated in step S42. The calculation process is to use the real-time scene analysis graph to extract environmental features through the convolutional network as the latent vector of the GRU unit, and use the speed, throttle, and steering information as the input vector of the GRU unit to generate the next moment control feature. After decoding through the fully connected network, the throttle and steering control at the next moment are predicted respectively; Step S63: After generating the next moment control, repeat step S61 to predict the vehicle speed and repeat step S62 to predict the continuous control.

5. The mine auxiliary transport vehicle safe navigation method based on real-time scene analysis according to claim 3 is characterized in that: The underlying control judgment in step S7 for determining whether the vehicle output is safe includes the following steps: Step S71: Calculating the vehicle's forward distance based on the predicted vehicle speed information and the fixed vehicle control interval; Step S72: Calculating the vehicle offset angle proportionally based on the predicted vehicle steering control; Step S73: Calculate the vehicle's coordinate change difference based on the vehicle's forward distance and the vehicle's offset angle. ; Step S74: Mapping the calculated coordinate change difference to the real-time scene analysis map generated in step S42 to determine whether the predicted navigation control is safe and compliant; Step S75: If the navigation control is determined to be compliant in step S74, continue to run; otherwise, take emergency braking.