A method and system for detecting drivable areas in underground mine tunnels based on visual semantic segmentation

Abstract

The present invention discloses a method and system for detecting drivable areas in underground mine tunnels based on visual semantic segmentation. The method includes: production of a scene dataset of drivable areas in underground mine tunnels; construction and training of a lightweight drivable area detection model MvSeg‑UR based on MobileVit; MvSeg‑UR system integration and model deployment; real-time feasible area detection and grid map feedback. The present invention improves the MobileVit backbone network structure, further reduces the amount of network parameters while increasing accuracy, and improves the calculation speed; in view of the problem of fuzzy and uneven edge detection in drivable areas, the present invention improves the edge loss function and proposes a soft edge loss function based on the Canny operator, so that the model training process is more sensitive to edge pixels, and the robustness and accuracy of the MvSeg‑UR model are improved. The present invention solves the problem of real-time detection of drivable areas in underground mining areas, provides a real-time and accurate drivable area detection method, and provides a bottom-up effect for the unmanned driving navigation method of underground mine trucks.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving visual perception, and in particular to an image feature extraction network based on Transformer and CNN and a visual semantic segmentation method, and specifically to a method for detecting a drivable area in an underground mine tunnel based on visual semantic segmentation. Background Art

[0002] In the transportation system of underground mines, mining trucks are one of the main modes of transportation. However, traditional mining trucks have problems such as high personnel costs, low production efficiency, and poor safety. Therefore, realizing unmanned mining trucks in underground mines has become an important development direction. Through the application of unmanned mining truck technology, the mine transportation system can be integrated into an organic whole that can be coordinated and dispatched, thereby reducing personnel costs and improving production efficiency.

[0003] The mine tunnel drivable area detection system based on visual semantic segmentation plays an important role in the realization of unmanned mining trucks. Through this system, the images of the mine tunnel can be analyzed and interpreted, and the boundaries between the drivable area and walls and obstacles can be accurately identified, providing key navigation information for unmanned mining trucks.

[0004] First, through visual semantic segmentation technology, the system can classify each pixel in the mine tunnel image and identify roads and walls. In this way, the area where the mining truck can drive can be accurately delineated, avoiding the risk of entering the prohibited area or colliding with obstacles.

[0005] Secondly, the system can update the drivable area information of the tunnel in real time and respond to changes in the internal environment of the tunnel in a timely manner. If new obstacles appear or road conditions change, the system can immediately detect and update the drivable area information to ensure that the unmanned mining truck always operates in a safe area, providing a backup for the SLAM-based prior global map navigation method.

[0006] In addition, the mine tunnel drivable area detection system based on visual semantic segmentation can also be integrated with other sensor data, such as lidar, inertial navigation, etc., to improve the accuracy and reliability of detection. Through the comprehensive analysis of multi-sensor data, the system can more comprehensively understand the environment of the mine tunnel and provide more accurate navigation guidance for unmanned mining trucks.

[0007] Therefore, the mine tunnel drivable area detection system based on visual semantic segmentation plays a key role in achieving safe operation and efficient transportation of unmanned mining trucks. It provides accurate navigation guidance for unmanned mining trucks by identifying and updating the information of drivable areas, ensuring the smooth operation of mine transportation. At the same time, the application of this system can also reduce human errors and accident risks, improve production efficiency and work safety, and bring significant improvements and innovations to underground mine transportation. Summary of the invention

[0008] The present invention proposes a method for detecting the drivable area of ​​underground mine tunnels based on visual semantic segmentation. The method provides accurate navigation guidance for unmanned mining trucks by identifying and updating the information of the drivable area, thereby ensuring the smooth progress of mine transportation.

[0009] The object of the present invention is achieved in that:

[0010] S01. Collect the original image data of the underground tunnel in real time through the vehicle-mounted camera;

[0011] S02. According to the labeling rules, the contours and categories of the drivable area and the background in the original image are labeled to form a label data set, the label data set and the original image are masked to obtain a mask image, the original image and the mask image are integrated into a mine tunnel data set in the cityscape format, and then divided into a training set, a validation set, and a test set;

[0012] S03. Performing data enhancement on the mask image and the original image in the mine tunnel data set, respectively, to obtain the mask image and the original image after the enhanced data, and normalizing the pixel values ​​of each channel of the original image;

[0013] S04. Construct an MvSeg-UR model for segmenting the drivable area. The MvSeg-UR model includes an encoding layer and a decoding layer. The output end of the decoding layer is connected to the classifier.

[0014] S05. The MvSeg-UR model is trained, verified and tested using the mask image and the original image in the mine tunnel data set after data enhancement to obtain a trained MvSeg-UR model;

[0015] S06. Use the trained MvSeg-UR model to detect the original image of the underground mine tunnel collected in real time, calculate the actual distance of each pixel point in the drivable area relative to the vehicle body according to the height difference between the vehicle-mounted camera and the ground of the drivable area, and convert it into a raster image to obtain a two-dimensional grid map of the drivable area.

[0016] In step S06, the mmdeploy model deployment tool is used to deploy the MvSeg-UR model on the Nvidia Orin-X platform. The deployed MvSeg-UR model is used to process the input video frames of the original image of the underground tunnel in real time to obtain the segmented mask image, and the mask image is converted into a raster image according to the 2D to 3D distance calculation formula.

[0017] The step S02 comprises:

[0018] Use LabelMe to annotate the outlines and categories of the drivable area and background area in the original image to form a label dataset in JSON format;

[0019] Use masks to perform pixel-level segmentation and annotation on the drivable area and background selected from the JSON format label dataset and the original image, mark different categories with different masks, and obtain a mask image;

[0020] Convert the mask image into a labeled dataset in the cityscape format;

[0021] Using the random partitioning method, the original images and mask images in the cityscape format label data set are divided into training set, validation set and test set in a ratio of 7:1:2.

[0022] The step S03 comprises:

[0023] Use cropping, rotation, flipping, translation, noise addition, brightness adjustment or one or more data enhancement methods in AutoAugment to perform data enhancement on the mask image and the original image respectively;

[0024] The original image is normalized. The normalization process includes: calculating the pixel average and standard deviation of all the RGB channels of the original image, and using the normalization formula to normalize each pixel value of the RGB channel of each original image.

[0025] The MvSeg-UR model constructed in step S04 is used to solve the problem of real-time detection of drivable areas in underground mining areas. The MvSeg-UR model uses the improved MobileViTv2 as the backbone network. MobileViTv2 has the advantages of CNN and Transformer, and after improvement, it has fewer parameters and meets the real-time requirements. In order to further improve the accuracy, the model introduces the Ds-FPN structure, which improves the network accuracy while only increasing the amount of calculation. Specific structure of the model Specifically, the MvSeg-UR network consists of a coding layer feature branch and a decoding layer branch connected in series, which are then connected in parallel with the global feature fusion branch, and finally the output end is connected to the classifier. The eight feature maps from left to right in the model correspond to the features Figure 1To feature map 8. In addition, the model segmentation loss function consists of a mask loss function and an edge loss function. Finally, the model is trained using appropriate hyperparameters and learning rate adjustment strategies, and the model effect is evaluated.

[0026] The step S04 comprises:

[0027] Construct the MvSeg-UR model loss function;

[0028] Based on the series connection of the encoding layer feature branch and the decoding layer branch fused with the DS-FPN module, and then in parallel with the global feature fusion branch, a multi-scale feature fusion network is built;

[0029] Establishing a deep supervision network based on the MvSeg-UR model loss function;

[0030] Build a prediction network based on the classifier;

[0031] The multi-scale feature fusion network, the deep supervision network and the prediction network are connected to form the MvSeg-UR model.

[0032] The image is first feature extracted through the encoding layer, and feature maps of different stages are obtained through the MobileViTv2 backbone network; then upsampling is performed at the decoding layer to restore the image to the original scale, and the feature map is spliced ​​with the feature map of the decoding layer. At this stage, DS-FPN (Dual-stream Feature Pyramid Networks) is used to perform feature fusion enhancement on the feature maps of different channels at each scale.

[0033] The encoding layer feature branch adopts the improved MobileViTv2 as the backbone network. Figure 1 With features Figure 2 ,feature Figure 3 ,feature Figure 4They are connected in series in sequence, and each feature map is downsampled using a MobileNetV2 block with stride=1 and a MobileNetV2 block with stride=2. Transformer does not have spatial inductive bias (translation equivariance and correlation between local pixels) like CNN, and a large amount of data is required to learn certain characteristics in the data, which makes it impossible to be well applied to edge devices. The information obtained by CNN in space is local, so it will restrict the performance of the CNN network structure to a certain extent. On the contrary, the self-attention mechanism of Transformer can obtain global information. For this reason, the present invention introduces an improved MobileViTv2 as the backbone network of the model. MobileViTv2 combines the advantages of CNN and Transformer, and has spatial inductive bias and global information understanding, so that the accuracy of the model is improved compared with the traditional CNN model. In order to further reduce the number of model parameters and improve the calculation speed and accuracy, the present invention improves MobileViTv2.

[0034] The improvements include:

[0035] By replacing all 3×3 convolutions with depth-wise separable convolution structures, the number of parameters is further reduced from 5.8M to 5.2M.

[0036] The patch embeddings of the feature map are divided into 9 groups. When the model performs attention calculation, the patch embeddings in the same group are only calculated with the patch embeddings in the same group. This reduces the time complexity of the model and improves the calculation speed.

[0037] DS-FPN (Dual-stream Feature Pyramid Networks) is introduced for feature enhancement to improve the model's perception field of view and enhance the ability to perceive boundaries. DS-FPN is used to fuse the features extracted by the ViT-based encoder. The present invention adds the Ds block to each pyramid-scale feature map, so that when the model performs attention calculation, PatchEmbeddings is divided into two channels, global Patch Embedding and local Patch Embedding, and the features of these two channels are fused at the end of each stage, thereby improving the model accuracy.

[0038] The decoding layer feature branch is composed of feature map 5, feature map 6, feature map 7, and feature map 8 connected in series in sequence, and a deconvolution layer with a convolution kernel size of 3*3 is used between each feature map to perform a 2-fold upsampling process;

[0039] The global feature fusion branch consists of the features Figure 1 Figure 8 and features Figure 2 Figure 7. Features Figure 3 Figure 6. Features Figure 4 It is constructed in series with characteristic diagram 5;

[0040] The loss function of the MvSeg-UR model includes the mask loss function and the edge loss function;

[0041] Among them, the mask loss function is:

[0042]

[0043] in, represents the kth mask prediction of the ith image, m i express The corresponding true mask label value; y k Indicates whether the kth mask resolution is selected as the output resolution, and l indicates the calculation of binary cross entropy loss;

[0044] The marginal loss function is:

[0045]

[0046] Among them, e i represents the edge of the ground truth mask, represents the edge of the predicted mask, the true mask label m i and predicted mask labels The soft edge map is obtained by the Canny operator, and then the edge map is converted into a binary edge map according to the threshold value to obtain the edge of the true value mask. i and the edge of the predicted mask

[0047] The total loss function is the weighted addition of two loss functions, defined as follows:

[0048] L total =λ1L mask +λ2L edge (3)

[0049] Where λ1 and λ2 represent the weights of the mask loss function and the soft edge loss function, respectively, which are set here to λ1 = 0.2 and λ2 = 0.8.

[0050] During the training process of step S05, the MvSeg-UR model is trained using the mask image and original image of the marked mine tunnel drivable area dataset. The training rounds are set to 300 rounds, the Adam optimizer is used, and the cosine annealing algorithm is used to adjust the learning rate. The trained model is evaluated using the validation set, and IoU (average intersection over union) and PixelAccuracy (pixel accuracy) are used as evaluation indicators.

[0051] The data sets used for deep learning training are generally divided into training sets, validation sets, and test sets. The requirements for training indicators refer to the relevant result parameters obtained by the training model on the validation set. For the mask label of a certain pixel, the positive example represents the category as a drivable area, and the negative example represents the category as background. For the prediction result of the pixel, the positive example represents the category as a drivable area, and the negative example represents the category as background. TP (True Positive) represents a true positive example, which is actually a positive example and the model predicts it to be a positive example; TN (True Negative) represents a true negative example, which is actually a negative example and the model predicts it to be a negative example; FP (False Positive) represents a false positive example, which is actually a negative example and the model predicts it to be a positive example; FN (False Negative) represents a false negative example, which is actually a positive example and the model predicts it to be a negative example.

[0052] Pixel accuracy represents the number of correctly classified samples divided by the total number of samples. The calculation formula is as follows:

[0053]

[0054] The average intersection-over-union ratio represents the average of the intersection and union ratios of the true value and the predicted value for all categories. It is defined as follows in the binary classification task:

[0055]

[0056] In the training set of 3200 images, the model converged after 180 rounds. The model parameter files of each round were saved and the model with the best performance on the validation set was selected for testing on the test set. After testing, the model obtained an IoU score of 0.9575 and a Pixel Accuracy of 0.9740 in the validation set of 600 images. This proves that the MvSeg-UR model proposed in the present invention meets the requirements of real-time and accuracy at the same time, and meets the operating conditions on edge computing devices.

[0057] After obtaining the corresponding weight file in the training process, step S05 converts the weight file into the ONNX intermediate representation format.

[0058] The ONNX Runtime inference engine was used for inference, and the trained model was deployed to the Nvidia Orin-X platform and integrated into the mine tunnel drivable area detection system. After testing, the model inference speed was 8FPS, meeting the real-time requirements. It is worth mentioning that during the model transplantation process, the difference in operators caused a certain loss of accuracy. On the Nvidia Orin-X platform, the Pixel Accuracy of the model was 0.9580.

[0059] The step S06 is specifically as follows:

[0060] The vehicle-mounted camera collects images of underground mine tunnels in real time, and publishes the image data to the Ros system through the can2ros node.

[0061] The image processing method is used to call the Opencv library to perform perspective transformation and dedistortion on the image, thereby reducing the distance calculation error caused by image distortion in subsequent rasterization image operations.

[0062] The trained MvSeg-UR model is used to read the original image of the underground mine tunnel collected in real time through ROS topic communication, and the MvSeg-UR model is used to perform real-time reasoning of the drivable area to obtain a mask image.

[0063] Since the camera is installed at a fixed height and the drivable area is on the ground, the height difference between the drivable area and the camera in the image is known. This means that the actual distance of each pixel point in the drivable area relative to the vehicle body is calculated based on the height difference between the on-board camera and the ground of the drivable area in the upper and lower boundaries of the image. The actual distance of the drivable area relative to the electric locomotive is determined. Therefore, the drivable area in the image is first gridded according to pixels, the actual distance information of the drivable area relative to the vehicle body is calibrated, the rasterized image is estimated, and the feedback is given to the automatic control system to provide real-time safety prompts and decision support.

[0064] A mine tunnel drivable area detection system based on visual semantic segmentation, comprising:

[0065] The original image acquisition module collects the original image data of the underground mine tunnel in real time through the vehicle-mounted camera;

[0066] The image formatting module annotates the contours and categories of the drivable area and background in the original image according to the annotation rules to form a label data set, masks the label data set and the original image to obtain a mask image, integrates the original image and the mask image into a mine tunnel data set in the cityscape format, and then divides it into a training set, a validation set, and a test set;

[0067] The data enhancement module performs data enhancement on the mask image and the original image in the underground tunnel data set, obtains the mask image and the original image after the enhanced data, and normalizes the pixel value of each channel of the original image;

[0068] Model building module, building the MvSeg-UR model for segmenting the drivable area. The MvSeg-UR model includes an encoding layer and a decoding layer. The output of the decoding layer is connected to the classifier.

[0069] A model training module, using the mask image and the original image in the underground mine tunnel data set after the data enhancement to train, verify and test the MvSeg-UR model to obtain a trained MvSeg-UR model;

[0070] The drivable area detection module uses the trained MvSeg-UR model to detect the original image of the underground mine tunnel collected in real time, calculates the actual distance of each pixel point in the drivable area relative to the vehicle body according to the height difference between the vehicle-mounted camera and the ground of the drivable area, and converts it into a raster image to obtain a two-dimensional grid map of the drivable area. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 This is a flowchart of detecting the drivable area in underground mine tunnels based on visual semantic segmentation proposed in the present invention.

[0072] Figure 2 This is a schematic diagram for visualizing detection data of the drivable area under a mine proposed according to the present invention.

[0073] Figure 3 Schematic diagram of the MvSeg-UR model proposed according to the present invention.

[0074] Figure 4 It is a structural diagram of a mine tunnel drivable area detection system based on visual semantic segmentation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The following will clearly and completely describe the technical solutions of various embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiment is only one embodiment of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0076] As used herein, the term "mining truck" means a mining truck.

[0077] The present invention will be described in more detail below with reference to the accompanying drawings:

[0078] An embodiment of the method of the present invention and its real-time situation are as follows Figure 4 As shown, the process is as follows:

[0079] (1) Create a dataset of drivable areas in underground mine tunnels.

[0080] (1.1) The camera is installed at the center axis of the front window of the mine car, 1.8 meters above the ground. The mine car travels normally along the mine road, and the on-board camera collects RGB images of different working conditions in the mine tunnel, such as Figure 2 As shown in (a), there are 4000 images in total.

[0081] (1.2) Formulate labeling rules, use the Labelme labeling tool to label the image, use polygonal boxes to distinguish the drivable area and background in the labeling rules, and store the labeling information in a json format file. Write a Python data processing script to convert the json format file exported by the Labelme tool and the original image into a mask image as the true label of the data set, where the pixel value 0 represents the background and the pixel value 1 represents the drivable area in the image.

[0082] Finally, the original image and JSON file are converted into a dataset in the cityscape format, such as Figure 2 As shown in the figure, the cityscape format dataset contains two folders, the image files are stored in the "leftImg8bit" folder, and the mask files are stored in the "gtFine" folder.

[0083] The format of the Cityscapes dataset is as follows:

[0084] Each image in the Cityscapes dataset is an RGB image of 2048×1024 pixels.

[0085] The semantic segmentation annotation layer (mask image) is a single-channel grayscale image, and the grayscale value of each pixel corresponds to its corresponding semantic category. In this embodiment, there are two category labels, including drivable area and background.

[0086] The annotation file is in XML format, which contains the coordinates of each pixel and the corresponding label information.

[0087] Each image in the Cityscapes dataset has a corresponding JSON file that contains the image's metadata and annotation information.

[0088] (1.3) Using the random partitioning method, the cityscape format data set is divided into training set, validation set and test set in a ratio of 7:1:2 to ensure that the distribution of each set is consistent.

[0089] The completed mine tunnel drivable area dataset is visualized as follows: Figure 2 (a) and Figure 2 (b) Figure 2 (a) represents the original image before annotation. Figure 2 (b) represents the mask visualization image after annotation, and the light-colored area is the marked drivable area under the mine.

[0090] (2) Before the data is input into the model, the collected mask image and original image data are enhanced and normalized respectively.

[0091] (2.1) Using the AutoAugment optimal data enhancement solution, first prepare 16 commonly used data enhancement operations, select 5 operations from the 16, randomly generate the probability of using the operation and the corresponding amplitude, and perform a series of random transformations on the original image and the corresponding mask label to expand the training data. For example, if you choose a rotation operation, you can randomly select the rotation angle and whether to flip the image. This operation is called a sub-policy, and a total of 5 sub-polices are generated; for each batch of images during the training process, one of the 5 sub-polices operations is randomly used; feedback is provided through the generalization ability of the model on the validation set, and the optimization method used is the reinforcement learning method; after 80 to 100 epochs, the network begins to learn effective sub-policies; then these 5 sub-policies are concatenated, that is, the combination strategy of array enhancement, and then the final training is carried out.

[0092] The optimal data enhancement scheme for AutoAugment varies depending on the dataset and task, but generally includes some common methods such as cropping, rotation, flipping, translation, adding noise, and brightness adjustment. These methods can help increase the diversity and generalization of data, thereby improving the performance of the model.

[0093] Generalization ability refers to the ability of a machine learning model to adapt to new samples. Specifically, when the model learns the inherent laws of the data during training, it can correctly classify or predict new samples.

[0094] Epoch represents the number of complete passes of the dataset through the entire neural network reinforcement learning.

[0095] (2.2) Calculate the mean and standard deviation of the pixel values ​​of each channel of the original image, and then use the normalization formula to map the pixel values ​​of each channel of the original image to between 0 and 1, that is, subtract the mean value and divide it by the standard deviation. This can prevent excessive pixel values ​​from having too much impact on training. This accelerates model convergence and ensures that the value ranges of different features are similar, so as to avoid certain features from having too much impact on model training.

[0096] The normalized calculation formula is as follows:

[0097]

[0098] Where X scale represents the normalized pixel value, x represents the pixel value before normalization, and x mean represents the mean value of the pixel value of a certain channel, and S represents the standard deviation of the pixel value of a certain channel.

[0099] The image resolution captured by the camera is 1920*1080. A series of random transformations are performed on the original image and the corresponding mask label to expand the training data. The obtained feature map is scaled to a size of 224*224. Finally, the image with a size of 224×224 pixels and the corresponding mask label are used as the input of the MvSeg-UR model.

[0100] (3) MvSeg-UR model construction and training.

[0101] (3.1) The MvSeg-UR model structure is as follows Figure 3 As shown, specifically, the MvSeg-UR network consists of a coding layer, a decoding layer, and a classifier. The coding layer and the decoding layer include a coding layer feature branch and a decoding layer feature branch respectively. The coding layer feature branch and the decoding layer feature branch are connected in series and then connected in parallel with the global feature fusion branch. Finally, the output end is connected to the classifier to form the MvSeg-UR network.

[0102] Specifically, the encoding layer feature branch consists of features Figure 1 With features Figure 2 ,feature Figure 3 ,feature Figure 4 The structure is connected in series, Figure 1 and Features Figure 2 Between, and features Figure 2 and Features Figure 3 A MobileNetV2 block with stride=1 and a MobileViT block are used for feature extraction, and a MobileNetV2 block with stride=2 is used for downsampling. Figure 3 and Features Figure 4 A MobileNetV2 block with stride = 2 is used for downsampling. stride refers to the step size; MV2 block refers to MobileNetV2 block.

[0103] The original image size is 224×224. After a convolution with a kernel size of 3×3, a stride of 2, and 16 channels, the size becomes 112×112. Then it passes through an MV2 block with a stride of 1 and 16 channels, and a MobileViT block with 16 channels to obtain the feature Figure 1 ;feature Figure 1 After two MV2 blocks, the first MV2 block has a step size of 2 and a number of channels of 24, and the size becomes 56×56. The second MV2 block has a step size of 1 and a number of channels of 24, and the feature Figure 2 ;

[0104] feature Figure 2 After one MV2 block, the stride is set to 2, the number of channels is 48, and the size becomes 28×28. After one MobileViT block, the number of channels is 48, the feature Figure 2 ;feature Figure 2 After two MV2 blocks, the first MV2 block has a step size of 2 and a channel number of 48, and the size becomes 28×28. The second MV2 block has a step size of 1 and a channel number of 48, and the feature Figure 3 ;

[0105] feature Figure 3 After two MV2 blocks, the first MV2 block has a step size of 2, a number of channels of 96, and a size of 14×14. The second MV2 block has a step size of 1, a number of channels of 96, and the feature Figure 4 .

[0106] The MobileViT block includes convolution (for extracting local feature representatives), Transformer module (for extracting global feature representatives), and residual connection fusion module. First, the feature map is used to extract local features of the image through an n×n convolution, and then the number of channels of the feature map is adjusted from C to d using a 1×1 convolution. Then the feature map is divided into 16 blocks (patches), each patch corresponds to an embedding, and all embeddings are divided into 9 groups. The Transformer module is used to extract global features between the same groups, and the number of channels of the feature map is adjusted back to C using a 1×1 convolution. Finally, the shortcut residual branch is used to simply concatenate the feature map and the original input map along the channel direction, and these features are fused through an n×n convolution to obtain the final global feature output.

[0107] The feature branch of the decoding layer is composed of feature map 5, feature map 6, feature map 7, and feature map 8 connected in series in sequence. A deconvolution layer with a convolution kernel size of 3×3 is used between each feature map to perform a 2x upsampling process;

[0108] Specifically, the Ds-FPN decoding layer contains feature maps of four scales. The feature map generated by the decoding layer is adjusted to a scale of 14×14 and 64 channels after global pooling and 1×1 convolution channels. Then, channel attention is used to focus on the features. Figure 4 After processing, the feature map scale is 14×14 and the number of channels is 32. The two feature maps are spliced ​​to finally obtain feature map 5, with a scale of 14×14 and 96 channels. Feature map 5 is adjusted to a feature map scale of 28×28 and 32 channels after a deconvolution layer with a convolution kernel size of 3×3 and a convolution channel of 1×1. Then, channel attention is used to adjust the feature map. Figure 3 After processing, the feature map scale is 28×28 and the number of channels is 16. The two feature maps are spliced ​​to finally obtain feature map 6, with a scale of 28×28 and a number of channels of 48. Feature map 6 is adjusted to a feature map scale of 56×56 and a number of channels of 16 after a deconvolution layer with a convolution kernel size of 3×3 and a convolution channel of 1×1. Then, channel attention is used to adjust the feature map. Figure 2 After processing, the feature map scale is 56×56 and the number of channels is 8. The two feature maps are spliced ​​to finally obtain feature map 7, with a scale of 56×56 and 24 channels. Feature map 7 is adjusted to a feature map scale of 112×112 and 12 channels after a deconvolution layer with a convolution kernel size of 3×3 and a convolution channel of 1×1. Then, channel attention is used to adjust the feature map. Figure 2 After processing, the feature map scale is 112×112 and the number of channels is 4. The two feature maps are concatenated to finally obtain feature map 8 with a scale of 112×112 and 16 channels.

[0109] The classifier (Segmention Head) uses a deconvolution layer with a convolution kernel size of 3×3 to scale the feature map to 224×224, and then uses a convolution layer with a convolution kernel size of 1×1 to adjust the channel to 2, corresponding to the two categories of segmentation, background and drivable area.

[0110] The global feature fusion branch consists of features Figure 1 Figure 8 and features Figure 2 Figure 7. Features Figure 3 Figure 6. Features Figure 4 It is constructed in series with characteristic diagram 5;

[0111] When the MvSeg-UR model performs attention calculations, Patch Embeddings in the same group are only calculated with Patch Embeddings in the same group. This reduces the time complexity of the model and improves the calculation speed.

[0112] All 3×3 convolutions in the MobileViTv2 network are replaced with depth-wise separable convolution structures, and the number of parameters is further reduced from the original 5.8M to 5.2M.

[0113] The image is first feature extracted through the encoding layer, and feature maps of different stages are obtained through the MobileViTv2 backbone network; then upsampling is performed at the decoding layer to restore the image to the original scale, and the feature map is spliced ​​with the feature map of the decoding layer. In the decoding stage, DS-FPN (Dual-stream Feature Pyramid Networks) is used to perform feature fusion enhancement on the feature maps of different channels at each scale.

[0114] In the decoding layer upsampling stage, the MvSeg-UR model introduces DS-FPN (Dual-stream Feature Pyramid Networks) for feature enhancement, improving the MvSeg-UR model's perception field of view to enhance its ability to perceive boundaries. DS-FPN is used to fuse the features extracted by the ViT-based encoder, adding the Ds block to each pyramid-scale feature map, so that when the MvSeg-UR model performs attention calculations, it divides the Patch Embeddings into two channels, the global Patch Embedding and the local Patch Embedding, and calculates them separately. The features of these two channels are fused at the end of each stage, and the accuracy of the MvSeg-UR model is improved.

[0115] In addition, the MvSeg-UR model loss function contains a mask loss function and an edge loss function. The mask loss function reveals the quality of pixel classification and is defined as follows:

[0116]

[0117] in, represents the kth mask prediction of the ith image, m i express The corresponding true mask label value; y k Indicates whether the k-th mask resolution is selected as the output resolution, and l indicates the calculation of the binary cross entropy loss.

[0118] In contrast, the edge losses produced by masks of different resolutions vary greatly, and the edge loss function can better reveal the quality of the mask edge, which is beneficial to the edge detection of the drivable area. The traditional edge detection function is sensitive to sharp points, while for the drivable area, the boundary of the mask is generally smoother. Therefore, the use of the traditional edge detection function will cause the problem of high difficulty in boundary alignment and poor accuracy. In order to solve this problem, the present invention improves the definition of edge loss and adopts an edge loss function based on the Canny operator, which is defined as follows:

[0119]

[0120] Among them, e i represents the edge of the ground truth mask, Represents the edge of the predicted mask. The edge of the true value mask e i and the edge of the predicted mask The calculation method is as follows: First, in the real mask label m i and predicted mask labels Apply the Canny operator to get the true mask label m i and predicted mask labels The soft edge map of the image is the edge map, which represents the strength and position of the edge in the image. The edge map is then converted into a binary edge map through a threshold. All edge points below the threshold are considered non-edges, while edge points above the threshold are considered edges. i and the edge of the predicted mask The soft edge loss function can reveal the quality of the mask from the perspective of segmentation boundaries. The Canny operator is used to detect whether a pixel is an edge.

[0121] The total loss function is the weighted addition of two loss functions, defined as follows:

[0122] L total =λ1L mask +λ2L edge (3)

[0123] Where λ1 and λ2 represent the weights of the mask loss function and the soft edge loss function, respectively, which are set here to λ1 = 0.2 and λ2 = 0.8.

[0124] (3.2) During the training process, the model is trained using the labeled mine tunnel drivable area dataset. The training rounds are set to 300, the Adam optimizer is used, and the cosine annealing algorithm is used to adjust the learning rate. The trained model is evaluated using the validation set, and IoU (average intersection over union) and Pixel Accuracy are used as evaluation indicators. The labeled mine tunnel drivable area dataset refers to the dataset divided into a training set, a validation set, and a test set. It contains two folders, the image files are stored in the "leftImg8bit" folder, and the mask files are stored in the "gtFine" folder.

[0125] The data sets used for deep learning training are generally divided into training sets, validation sets, and test sets. The training completion index requirements refer to the relevant result parameters obtained by the training model on the validation set. For the mask label of a certain pixel, the positive example represents the category as the drivable area, and the negative example represents the category as the background. Table 1 is the machine learning confusion matrix, in which TP (True Positive) represents the true positive example, which is actually a positive example and the model predicts it to be a positive example; TN (True Negative) represents the true negative example, which is actually a negative example and the model predicts it to be a negative example; FP (False Positive) represents the false positive example, which is actually a negative example and the model predicts it to be a positive example; FN (False Negative) represents the false negative example, which is actually a positive example and the model predicts it to be a negative example.

[0126] Pixel accuracy represents the number of correctly classified samples divided by the total number of samples. The calculation formula is as follows:

[0127]

[0128] The average intersection-over-union ratio represents the average of the intersection and union ratios of the true value and the predicted value for all categories. It is defined as follows in the binary classification task:

[0129]

[0130] Table 1 Confusion matrix

[0131]

[0132] In this implementation case, with 3200 training sets, the model converged after 180 rounds. The model parameter files of each round were saved and the model with the best performance on the validation set was selected for testing on the test set. After testing, the model obtained an IoU score of 0.9575 and a Pixel Accuracy of 0.9740 on the 600 validation sets.

[0133] Pixel Accuracy is a commonly used evaluation metric to measure the accuracy of image segmentation algorithms. In image segmentation tasks, Pixel Accuracy evaluates the accuracy of the algorithm by comparing the pixel classification results between the segmentation map output by the algorithm and the real annotated image. The calculation formula is the number of correctly predicted pixels divided by the total number of predicted pixels.

[0134] (4) Convert the trained model weight file into ONNX intermediate representation format. Use ONNX Runtime inference engine for inference, deploy the trained model to Nvidia Orin-X platform, and integrate it into the mine tunnel drivable area detection system. After testing, the model inference speed in this embodiment is 8FPS. During the model transplantation process, the difference in operators caused a certain loss of accuracy. On the Nvidia Orin-X platform, the Pixel Accuracy of the model is 0.9580.

[0135] (5) Use the MvSeg-UR model to process video frames in real time, obtain mask images, and convert them into raster images.

[0136] (5.1) The system collects real-time images of underground tunnels through the vehicle-mounted camera and publishes the image data to the Ros system through the can2ros node.

[0137] The role of the can2ros node is to convert data on the CAN bus into ROS messages, allowing the ROS system to receive and process data from the CAN bus. It is usually used in the field of automobiles and autonomous driving.

[0138] (5.2) Using traditional image processing methods and calling the Opencv library, the mine tunnel images are subjected to perspective transformation and dedistortion processing to reduce the distance calculation error caused by image distortion in subsequent rasterization image operations.

[0139] OpenCV is a cross-platform computer vision and machine learning software library released under the Apache 2.0 license (open source). OpenCV provides algorithms for perspective transformation, including:

[0140] cv2.getPerspectiveTransform(), used to obtain the perspective transformation matrix;

[0141] cv2.warpPerspective() is used to implement perspective transformation of images.

[0142] Before de-distortion processing, it is usually necessary to perform camera calibration to obtain the camera's internal parameters (such as camera matrix and distortion coefficient) and camera's external parameters (such as rotation matrix and translation vector). OpenCV provides algorithms for de-distortion processing, including:

[0143] cv2.undistort(), used to eliminate image distortion caused by lens distortion;

[0144] cv2.fisheye.undistortImage(), dedistortion function for fisheye lens;

[0145] cv2.opencv_contrib.omnidir.undistortImage() is a dedistortion function for omnidirectional cameras.

[0146] (5.3) The MvSeg-UR node reads the array mine tunnel image data through ROS topic communication, performs real-time inference of the drivable area through the MvSeg-UR model, and obtains the mask image.

[0147] (5.4) The actual distance of each pixel point in the drivable area relative to the electric locomotive is calculated based on the height difference between the camera and the ground of the drivable area, and the image is rasterized to obtain a two-dimensional grid map of the drivable area, which is fed back to the automatic control system to provide real-time safety prompts and decision support.

[0148] Based on the method provided by the present invention, the present invention also provides a mine tunnel drivable area detection system based on visual semantic segmentation, comprising:

[0149] The original image acquisition module collects the original image data of the underground mine tunnel in real time through the vehicle-mounted camera;

[0150] The image formatting module annotates the contours and categories of the drivable area and background in the original image according to the annotation rules to form a label data set, masks the label data set and the original image to obtain a mask image, integrates the original image and the mask image into a mine tunnel data set in the cityscape format, and then divides it into a training set, a validation set, and a test set;

[0151] The data enhancement module performs data enhancement on the mask image and the original image in the underground tunnel data set, obtains the mask image and the original image after the enhanced data, and normalizes the pixel value of each channel of the original image;

[0152] Model building module, building the MvSeg-UR model for segmenting the drivable area. The MvSeg-UR model includes an encoding layer and a decoding layer. The output of the decoding layer is connected to the classifier.

[0153] A model training module, using the mask image and the original image in the underground mine tunnel data set after the data enhancement to train, verify and test the MvSeg-UR model to obtain a trained MvSeg-UR model;

[0154] The drivable area detection module uses the trained MvSeg-UR model to detect the original image of the underground mine tunnel collected in real time, calculates the actual distance of each pixel point in the drivable area relative to the vehicle body according to the height difference between the vehicle-mounted camera and the ground of the drivable area, and converts it into a raster image to obtain a two-dimensional grid map of the drivable area.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting drivable areas in underground mine tunnels based on visual semantic segmentation, characterized in that: include: Step S01. Collecting original image data of underground tunnels in real time through the vehicle-mounted camera; Step S02. According to the annotation rules, the outlines and categories of the drivable area and background in the original image are annotated to form a label data set, the label data set and the original image are masked to obtain a mask image, the original image and the mask image are integrated and converted into an underground mine tunnel data set in the cityscape format, and then divided into a training set, a validation set, and a test set; Step S03. Perform data enhancement on the mask image and the original image in the underground tunnel data set, respectively, to obtain the mask image and the original image after the enhanced data, and perform normalization on the pixel values ​​of each channel of the original image; Step S04. Constructing an MvSeg-UR model for segmenting the drivable area. The MvSeg-UR model includes an encoding layer and a decoding layer. The output end of the decoding layer is connected to the classifier. Step S05. Using the mask image and the original image in the underground mine tunnel data set after data enhancement to train, verify and test the MvSeg-UR model, to obtain a trained MvSeg-UR model; The mask image and the original image in the training set after data enhancement are subjected to global feature extraction through the encoding layer of the MobileViTv2 network, and local features are extracted at different stages of the MobileViTv2 backbone network respectively. The global features are upsampled through the decoding layer to restore to the original scale before extracting the global features. The DS-FPN is used to perform feature fusion enhancement on different local features and decoding layer features, and then the classifier outputs the training results, and the corresponding weight files during the training process are saved; Each weight file is tested on the validation set after data enhancement and the hyperparameters of the MvSeg-UR model are adjusted. At the same time, the weight file with the highest segmentation accuracy is selected as the model parameters of the trained MvSeg-UR model. Input the data-augmented test set into the trained MvSeg-UR model to check the approximate generalization performance of the model; Step S06. Use the trained MvSeg-UR model to detect the original image of the mine tunnel collected in real time, calculate the actual distance of each pixel point in the drivable area relative to the vehicle body according to the height difference between the vehicle-mounted camera and the ground in the drivable area, and convert it into a raster image to obtain a two-dimensional grid map of the drivable area; The coding layer includes a coding layer feature branch, wherein the coding layer feature branch adopts an improved MobileViTv2 as a backbone network, and the coding layer feature branch is composed of feature map 1, feature map 2, feature map 3, and feature map 4 connected in series in sequence. A MobileNetV2 block with stride=1 and a MobileViT block are used between feature map 1 and feature map 2, and between feature map 2 and feature map 3 for feature extraction, and a MobileNetV2 block with stride=2 is used for downsampling. A MobileNetV2 block with stride=2 is used for downsampling between feature map 3 and feature map 4, and stride represents the step size. The MobileViT block includes convolution, Transformer module, and residual connection fusion module. When MobileViTblock extracts features, it first extracts local features from the image through an n×n convolution, and then uses a 1×1 convolution to adjust the number of channels of the feature map from C to d. Then, the feature map is divided into 16 patches, each patch corresponds to an embedding, and all embeddings are divided into 9 groups. The Transformer module is used between the same groups to extract global features and a 1×1 convolution is used to adjust the number of channels of the feature map back to C. Finally, the shortcut residual branch is used to simply concatenate the feature map and the original input map along the channel direction and fuse these features through an n×n convolution to obtain the final global feature output; The step S04 comprises: Construct the MvSeg-UR model loss function; Based on the series connection of the encoding layer feature branch and the decoding layer branch that integrates the DS-FPN and Dual-stream Feature Pyramid Networks modules, and then in parallel with the global feature fusion branch to build a multi-scale feature fusion network; Establishing a deep supervision network based on the MvSeg-UR model loss function; Build a prediction network based on the classifier; Connecting the multi-scale feature fusion network, the deep supervision network and the prediction network to form the MvSeg-UR model; The loss function of the MvSeg-UR model includes the mask loss function and the edge loss function; Among them, the mask loss function is: （1） in, represents the k-th mask prediction of the i-th image, express The corresponding true mask label value; Indicates whether the kth mask resolution is selected as the output resolution, Indicates the calculation of binary cross entropy loss; The marginal loss function is: （2） in, represents the edge of the ground truth mask, represents the edge of the predicted mask, the true mask label and predicted mask labels The soft edge map is obtained by the Canny operator, and then the edge map is converted into a binary edge map according to the threshold value to obtain the edge of the true value mask. and the edge of the predicted mask ; The total loss function is the weighted addition of two loss functions: （3） in and They represent the weights of the mask loss function and the soft edge loss function respectively.

2. The method for detecting drivable areas in underground mine tunnels based on visual semantic segmentation according to claim 1 is characterized in that: The step S02 comprises: Use LabelMe to annotate the outlines and categories of the drivable area and background area in the original image to form a label dataset in JSON format; Use masks to perform pixel-level segmentation and annotation on the drivable area and background selected from the JSON format label dataset and the original image, mark different categories with different masks, and obtain a mask image; Convert the mask image into a labeled dataset in the cityscape format; Using the random partitioning method, the original images and mask images in the cityscape format label data set are divided into training set, validation set and test set in a ratio of 7:1:

2.

3. The method for detecting drivable areas in underground mine tunnels based on visual semantic segmentation according to claim 1 is characterized in that: The step S03 comprises: Use cropping, rotation, flipping, translation, noise addition, brightness adjustment or one or more data enhancement methods in AutoAugment to perform data enhancement on the mask image and the original image respectively; The original image is normalized. The normalization process includes: calculating the pixel average and standard deviation of all the RGB channels of the original image, and using the normalization formula to normalize each pixel value of the RGB channel of each original image.

4. A system for detecting drivable areas in underground mine tunnels based on visual semantic segmentation, used to implement the method for detecting drivable areas in underground mine tunnels based on visual semantic segmentation according to claim 1, characterized in that: include: The original image acquisition module collects the original image data of the underground mine tunnel in real time through the vehicle-mounted camera; The image formatting module annotates the contours and categories of the drivable area and background in the original image according to the annotation rules to form a label data set, masks the label data set and the original image to obtain a mask image, integrates the original image and the mask image into a mine tunnel data set in the cityscape format, and then divides it into a training set, a validation set, and a test set; The data enhancement module performs data enhancement on the mask image and the original image in the underground tunnel data set, obtains the mask image and the original image after the enhanced data, and normalizes the pixel value of each channel of the original image; Model building module, building the MvSeg-UR model for segmenting the drivable area. The MvSeg-UR model includes an encoding layer and a decoding layer. The output of the decoding layer is connected to the classifier. A model training module, using the mask image and the original image in the underground mine tunnel data set after the data enhancement to train, verify and test the MvSeg-UR model to obtain a trained MvSeg-UR model; The drivable area detection module uses the trained MvSeg-UR model to detect the original image of the underground mine tunnel collected in real time, calculates the actual distance of each pixel point in the drivable area relative to the vehicle body according to the height difference between the vehicle-mounted camera and the ground of the drivable area, and converts it into a raster image to obtain a two-dimensional grid map of the drivable area.

Images