Mining underground monorail crane rail segmentation system and method based on U-Net

Abstract

The invention discloses a mining underground monorail crane rail segmentation system and method based on U-Net, and belongs to the technical field of mine intelligence and computer vision crossing. In order to solve the problems of poor track identification robustness and low precision caused by underground low illumination, high dust and strong light reflection, a segmentation system with U-Net as a core is constructed; an image is acquired through a mining explosion-proof RGB camera; training the model by using a data enhancement strategy of brightness degradation, noise injection and dust simulation; pixel-level segmentation is realized by using an encoder-decoder structure and jump connection; and optimizing boundary details through a composite loss function. The system only needs an RGB camera meeting the mining explosion-proof standard, does not need a laser radar, and can operate in real time on an embedded platform. Experiments show that a test set mIoU reaches 87.91%, performance attenuation under typical underground interference is extremely small, high robustness, high precision and strong engineering practicability are achieved, and a reliable visual perception basis is provided for intelligent navigation of the monorail crane.

Description

Technical Field

[0001] This invention relates to a method for track segmentation of a single-rail crane used in underground mining based on the U-Net deep learning model, belonging to the interdisciplinary field of intelligent mining and computer vision. Background Technology

[0002] In underground coal mining operations, suspended monorails are key equipment for transporting personnel and materials. However, their operation is highly dependent on manual operation. In the complex underground environment with low light, high dust, humidity, and a large amount of interference from metal structures, drivers are prone to accidents such as collisions and derailments due to visual fatigue or poor visibility.

[0003] The primary prerequisite for achieving automation or unmanned operation of monorail gantry cranes is building a reliable visual perception system capable of accurately identifying and locating the track. Traditional computer vision methods, such as edge detection methods based on Canny and Sobel operators, or line / curve fitting combined with Hough transform, heavily rely on clear and continuous edge features. In real-world underground scenarios, the track is often obscured by rust, oil, water stains, or dust, leading to blurred or even missing edge information. This renders these traditional methods ineffective and extremely unreliable.

[0004] In recent years, deep learning-driven semantic segmentation techniques have offered new solutions to this problem. Among them, U-Net, with its unique encoder-decoder symmetric structure and skip connection mechanism, has achieved great success in fields such as medical image segmentation. This structure effectively fuses global contextual information and local spatial details of an image, making it ideal for tasks requiring precise localization. However, directly applying U-Net to underground industrial scenarios with uneven lighting and dynamic interference still faces significant challenges. Current technologies lack a dedicated segmentation system specifically designed for the characteristics of mining monorail tracks and capable of effectively resisting typical underground interferences (such as dust, noise, and extreme lighting). Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a U-Net-based track segmentation system for underground monorail cranes in mines, characterized by a clear structure and convenient deployment. It also provides a highly efficient algorithm for segmenting the track of underground monorail cranes in mines, with strong anti-interference capabilities and precise segmentation boundaries.

[0006] The technical solution of this invention to solve the above problems is: a U-Net-based underground monorail track segmentation system, comprising an image acquisition module, a preprocessing module, a track segmentation core module, and a post-processing and decision module; the image acquisition module is used to acquire real-time images of underground roadways containing the monorail track; the preprocessing module is used to standardize the original images; the track segmentation core module uses a U-Net deep neural network model to generate pixel-level track segmentation probability maps; the post-processing and decision module is used to convert the probability maps into usable track area information and output it to the control system.

[0007] A method for track segmentation of a mine underground monorail crane based on U-Net includes the following steps:

[0008] Step 1: Image Acquisition and Dataset Construction: RGB images of the monorail track in the underground roadway are acquired by an industrial camera installed at the front of the monorail crane, and the acquired images are manually annotated at the pixel level to construct a training dataset containing track mask labels.

[0009] Step 2, Data Preprocessing and Augmentation: The original image is normalized in size and pixel value, and data augmentation strategies simulating typical downhole disturbances are introduced during the training phase, including brightness degradation, noise injection and dust occlusion simulation.

[0010] Step 3: U-Net Model Construction and Training: Build a U-Net neural network model with an encoder-decoder symmetric structure and a skip connection mechanism. Optimize and train the model using a composite loss function consisting of binary cross-entropy loss and Dice loss until the model converges.

[0011] Step 4, Model Deployment and Real-time Inference: The trained U-Net model is converted into a lightweight format and deployed to an in-vehicle embedded computing platform. During runtime, it receives real-time image streams and performs forward inference to generate track segmentation results.

[0012] Step 5: Post-processing and track information output: Thresholding and morphological optimization are performed on the probability map output by U-Net to generate a binary segmentation mask, and the track centerline or boundary coordinates are further extracted for use by the upper navigation and control system.

[0013] The above-mentioned U-Net-based method for track segmentation of underground monorail cranes in mines includes the following steps: In step one, image acquisition covers four typical underground working conditions: straight track, curved track, uphill track, and downhill track. The acquisition equipment uses an industrial-grade RGB network camera with wide dynamic range and 3D digital noise reduction capabilities, such as the Hikvision DS-2CD2820FWD model. The dataset is labeled using professional image labeling tools such as LabelMe to ensure the accuracy and consistency of the track area boundary labeling.

[0014] In the above-mentioned U-Net-based method for segmenting the track of a single-rail crane in a mine, in step two, the image size is uniformly normalized to 224×224 pixels, and the pixel value is normalized to the range of [0, 1]. The data enhancement specifically includes: (1) randomly reducing the brightness (V channel) in the HSV (hue, saturation, brightness) color space to simulate low illumination and overexposure; (2) superimposing Gaussian noise and salt-and-pepper noise to simulate sensor noise and particulate interference; (3) superimposing a semi-transparent dark brown or gray texture in a local area of ​​the image and applying Gaussian blur to realistically simulate the occlusion effect of dust covering on the visual features of the track.

[0015] The above-mentioned U-Net-based method for segmenting the track of a single-rail crane in underground mines, in step three, involves progressively downsampling the encoder path of the U-Net model to extract multi-scale features, progressively upsampling the decoder path through transposed convolution to restore spatial resolution, and using skip connections to stitch high-resolution feature maps from each encoder level to the corresponding decoder level to preserve track edge details; the composite loss function is defined as:

[0016]

[0017] in For real labels, For model prediction probability plots, The balancing factor is set to 0.5; training uses the Adam optimizer (an adaptive learning rate stochastic gradient descent optimization algorithm), with an initial learning rate of 1×10⁻⁶. -4 The batch size is 8, and the number of training rounds is 200.

[0018] In the above-mentioned U-Net-based method for segmenting the track of a single-rail crane in underground mines, step four involves exporting the trained PyTorch model to ONNX (Open Neural Network Exchange) format and further converting it to the TensorRT engine (NVIDIA's high-performance deep learning inference acceleration engine) to improve inference speed. The deployment platform is an embedded GPU device such as NVIDIA Jetson AGX Xavier, and the inference frame rate can reach more than 15 FPS, meeting the real-time requirements.

[0019] In the above-mentioned U-Net-based method for segmenting the track of a single-rail crane in underground mines, step five involves thresholding using a fixed threshold of 0.5 to convert the probability map into a binary mask; and morphological optimization using 3×3 structuring elements for closing operations to fill tiny holes in the segmentation results and smooth the track boundaries.

[0020] The beneficial effects of this invention are as follows:

[0021] 1. High robustness: Through targeted data augmentation strategies, the model exhibits minimal performance degradation (F1-Score decrease of <4%) under typical downhole disturbances such as dust occlusion, low illumination, and high noise, significantly outperforming traditional image processing methods.

[0022] 2. High segmentation accuracy: U-Net's skip connection mechanism effectively integrates shallow spatial details and deep semantic information, achieving high-precision segmentation with an mIoU of 87.91% and a recall rate of 92.90%, with clear and continuous boundaries;

[0023] 3. Lightweight and Real-time Performance: The model has only 1.95M parameters and low inference computation (8.25 GFLOPs), which can be efficiently deployed on resource-constrained vehicle-mounted embedded platforms to meet the real-time perception needs of mine sites.

[0024] 4. Strong engineering practicality: The system only needs to rely on intrinsically safe or explosion-proof RGB industrial cameras that meet mining explosion-proof standards (such as GB 3836.1 / 4), without the need to deploy expensive sensors such as LiDAR and depth cameras, which significantly reduces hardware costs and system complexity. At the same time, the solution is easy to integrate into the existing monorail control platform, providing a safe, reliable and scalable visual perception solution for the intelligent upgrade of underground monorails in mines. Attached Figure Description

[0025] Figure 1 This is a block diagram of the overall architecture of the system described in this invention;

[0026] Figure 2 This is a schematic diagram of the U-Net neural network structure used in this invention;

[0027] Figure 3 The image shows the track segmentation effect of the present invention in a typical downhole scenario (including dust and low light), where (a) is the original image, (b) is the manually annotated ground truth, and (c) is the segmentation result of the present invention. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0029] like Figure 1As shown, the U-Net-based track segmentation system for underground monorail cranes in mines, as described in this invention, includes an image acquisition module, a preprocessing module, a track segmentation core module, and a post-processing and decision-making module. The image acquisition module uses an industrial-grade RGB network camera (e.g., Hikvision DS-2CD2820FWD) that meets mining explosion-proof standards and is installed at the front end of the monorail crane vehicle. The preprocessing module runs on an onboard embedded computing unit (e.g., NVIDIA Jetson AGX Xavier) and is responsible for image standardization. The track segmentation core module is a U-Net deep neural network model deployed on this computing unit. The post-processing and decision-making module optimizes the segmentation results and outputs track area information for use by the upper-level navigation and control system.

[0030] A method for track segmentation of a mine underground monorail crane based on U-Net includes the following steps:

[0031] Step 1: Image Acquisition and Dataset Construction

[0032] To accurately simulate the operating conditions of a monorail crane in the actual underground working environment, the data acquisition work for this invention was specifically carried out in the monorail crane experimental workshop of a cooperative industrial partner. This experimental workshop replicates typical roadway structures, track layouts, and common environmental interference factors in underground mines, including low illumination, suspended dust, and metallic glare, to simulate the operating scenario of a monorail crane in a real mine.

[0033] During the data acquisition process, a Hikvision DS-2CD2820FWD high-definition network camera was installed at the center of the front of the monorail crane, ensuring that the field of view was directly facing the track area. The camera captured RGB images of the monorail track at a raw resolution of 1920×1080 pixels. The entire acquisition process covered four typical track configurations for monorail crane operation: straight sections, horizontal curves, uphill sections, and downhill sections, and was conducted under various artificially controlled lighting conditions (including normal lighting, partial obstruction, strong reflections, and low light scenes) to cover common visual interferences in underground operations. A total of 2,681 high-quality images were ultimately acquired. All images contained typical underground interference factors such as low light, overexposure, track metal reflections, and suspended dust, covering the main challenges in visual perception of monorail cranes.

[0034] Subsequently, the LabelMe open-source tool was used to perform pixel-level manual semantic annotation on all images. Specifically, two uniformly trained annotators independently completed the annotation of each image: the area belonging to the monorail track itself was marked as the foreground (label=1), and the remaining tunnel structures, equipment, and background areas were marked as the background (label=0), generating the corresponding binary mask image. Where H=1080 and W=1920.

[0035] The final dataset is denoted as Where N=2681, Let be the i-th original image. To ensure annotation quality, all annotation results are cross-checked by a third senior researcher. Regions with differing annotations are discussed and agreed upon before the final labels are determined. This dual-checking mechanism effectively guarantees the consistency and accuracy of the labeled data, providing reliable data support for subsequent model training.

[0036] Step 2: Data Preprocessing and Augmentation

[0037] To adapt to the U-Net model input, the original images are first preprocessed. Size normalization: all images and their corresponding masks are scaled to 224×224 pixels using bilinear interpolation;

[0038] Pixel value normalization: Normalize the RGB channels separately, using the following formula:

[0039]

[0040] in = [0.485, 0.456, 0.406], = [0.229, 0.224, 0.225] are the mean and standard deviation of the ImageNet dataset, respectively.

[0041] To improve the model's generalization ability in complex downhole environments, three types of targeted data augmentation strategies were introduced during the training phase to simulate typical disturbances:

[0042] (1) Brightness degradation: Insufficient or uneven lighting is a common problem in underground mines. Therefore, the brightness channel is scaled in the HSV color space using the following transformation formula:

[0043]

[0044] in, Represents the brightness value of the original image. The value is the transformed brightness. The brightness degradation intensity parameter has values ​​of {0.3, 0.6, 0.9}, corresponding to mild, moderate, and severe degradation levels, respectively, with maximum brightness reduction percentages of 21%, 42%, and 63%. This method effectively simulates real low-light conditions while preserving the relationship between image hue and saturation.

[0045] (2) Noise Injection: To simulate the sensor noise and particulate interference experienced by mining cameras in harsh environments, this invention injects Gaussian noise and salt-and-pepper noise into the image. Specifically, Gaussian noise is superimposed on the original image, with a standard deviation of... (Indicating noise intensity) set to ,in The noise intensity level parameter takes values ​​of {0.3, 0.6, 0.9} to generate a noise pattern from weak to strong; simultaneously, when When the density is greater than 0.7, additional salt-and-pepper noise is introduced, with a density ratio δ set to δ = 0.01 × It is used to simulate sensor failure or dense particle interference under extreme conditions.

[0046] (3) Dust Occlusion Simulation: To address the problem of severe visibility impairment caused by underground dust, this invention simulates the dust effect through a combination of color overlay, texture synthesis, and optical attenuation. First, a semi-transparent mask in a typical mining hue (dark brown [100, 90, 80]) is overlaid on the original image, with its opacity... (set as) ,in Similarly, values ​​are taken as {0.3, 0.6, 0.9}, with a maximum of 36%; based on this, programmed noise patterns and circular accumulation structures are further integrated to form natural dust textures, and at high interference levels (such as... Gaussian blur (convolution kernel size 5×5, standard deviation 1.0) and global contrast attenuation are applied at 0.9 to simulate visual blur and loss of detail caused by dense suspended particles.

[0047] After the above processing, the dataset is divided into a training set D_train, a validation set D_val, and a test set D_test in a 6:2:2 ratio.

[0048] Step 3: U-Net Model Construction and Training

[0049] This invention adopts the classic U-Net architecture, whose encoder-decoder structure is as follows: Figure 2 As shown. Let the input image be X∈R. 224×224×3 The model output is a segmentation probability map Ŷ ∈ [0,1] 224×224 .

[0050] The encoder path consists of three downsampling blocks, each containing two 3×3 convolutional layers and a 2×2 max pooling layer. The feature map sizes are 224×224×32, 112×112×64, and 56×56×128, respectively.

[0051] Bottleneck layer: Contains two 3×3 convolutional layers, with an output of 28×28×256.

[0052] Decoder path: It consists of 3 upsampling blocks. Each block is first upsampled through a 2×2 transposed convolutional layer, then concatenated with the feature map of the corresponding layer of the encoder (skip connection), and then passed through two 3×3 convolutional layers. The feature map size is gradually restored to 224×224×32.

[0053] Output layer: 1×1 convolutional layer + Sigmoid activation function, generating a single-channel probability map.

[0054] To mitigate the extreme class imbalance between tracks (foreground) and lanes (background), a composite loss function is employed:

[0055]

[0056] The binary cross-entropy loss is

[0057]

[0058] Where N represents the total number of pixels in a single image; i is the pixel index; g i This represents the true label value of the i-th pixel, which can be either 0 or 1. This represents the output probability that the model predicts the i-th pixel as the foreground, and its value ranges from (0, 1).

[0059] Dice coefficient is

[0060]

[0061] in, The smoothing factor has a value of 10. -6 This is used to avoid numerical instability caused by a denominator of zero.

[0062] The model is implemented using the PyTorch 1.12 framework, with Adam as the optimizer and an initial learning rate η = 1 × 10⁻⁶. -4 Batch size B = 8, training for 200 epochs. When the validation set loss does not decrease for 10 consecutive epochs, learning rate decay (multiplied by 0.5) is enabled. Finally, the model weights with the lowest validation loss are saved.

[0063] Step 4: Model Deployment and Real-time Inference.

[0064] To meet the real-time requirements of automotive embedded platforms, the trained PyTorch model is deployed in a lightweight manner:

[0065] (1) Model conversion: Export the model using ONNX as the intermediate format:

[0066] torch.onnx.export(model, dummy_input, "track_unet.onnx", opset_version=11)

[0067] (2) Inference acceleration: The ONNX model is converted into an optimization engine using NVIDIA TensorRT 8.6:

[0068] Trtexec --onnx=track_unet.onnx --saveEngine=track_unet.trt --fp16

[0069] Enable FP16 precision (half-precision floating-point, 16-bit floating-point format) to further improve inference speed.

[0070] (3) Deployment and Operation: On the Jetson AGX Xavier development board (Ubuntu 18.04 + JetPack 4.6), the TensorRT engine was loaded via C++ API to receive real-time video streams from the GStreamer pipeline and perform forward inference. The measured average inference time was 58 ms (approximately 17 FPS), which meets the sensing requirements of a monorail crane operating at low speeds (<2 m / s).

[0071] Step 5: Post-processing and track information output

[0072] The probability map output by U-Net needs to be converted into usable orbital region information:

[0073] (1) Thresholding: A binary mask is generated using a fixed threshold τ = 0.5.

[0074]

[0075] (2) Morphological optimization: Use 3×3 all-1 structuring elements to perform closing operation (dilation followed by erosion) to fill the tiny holes in the segmentation results and smooth the boundaries, eliminating isolated noise.

[0076] Ultimately, the binary mask Y_binary can be used directly for visualization, or further used to generate a sequence of track centerline coordinates through algorithms such as connected component analysis and skeleton extraction. This sequence can then be published through the topic / track_mask in the Robot Operating System (ROS) for use by the path tracking and obstacle avoidance modules.

[0077] Experimental verification:

[0078] The U-Net model was systematically evaluated on a test set of a self-made dataset. This test set includes four typical track scenarios: straight, horizontal curve, uphill, and downhill, and each scenario includes realistic underground disturbances such as low light, reflection, and dust.

[0079] Quantitative results show that the system achieves an average intersection-union ratio (mIoU, i.e., the average overlap rate between the predicted region and the real orbit region) of 87.91% and a recall rate (i.e., the proportion of correctly identified orbit pixels to all real orbit pixels) of 92.90% on the overall test set.

[0080] To visually demonstrate the model's segmentation performance in different scenarios, four representative sets of samples (one for each scenario) were selected from the test set, and comparison charts were drawn showing the original images, ground truth labels, and model prediction results, as shown in the attached diagram in the instruction manual. Figure 3 As shown. Figure 3 The four columns correspond to the straight section, horizontal curve section, uphill section, and downhill section, respectively. It can be seen that the segmentation mask output by the U-Net model can accurately fit the geometric contour of the track, with continuous and complete boundaries, and no obvious breaks, offsets, or large-area false detections. Especially in the horizontal curve section, although there are local edge missing parts of the track due to viewpoint obstruction or metallic reflection, the model can still maintain complete structural reconstruction, demonstrating good contextual understanding ability.

[0081] To further evaluate the model's robustness in harsh downhole environments, three types of artificial interference were applied to all test images: brightness degradation, Gaussian noise injection, and localized semi-transparent dust occlusion, with three intensity levels for each type of interference. Experiments showed that even under the most severe interference conditions (63% brightness reduction, Gaussian noise standard deviation σ = 22.5, and dust occlusion coverage reaching 36%), the model's F1-Score (the harmonic mean of precision and recall) decreased by only 0.30%, 3.78%, and 0.84%, respectively, indicating a relatively small performance degradation. This demonstrates the system's strong adaptability to typical downhole visual interference.

Claims

1. A track segmentation system for a mine underground monorail crane based on U-Net, characterized in that, The system includes an image acquisition module for real-time acquisition of images of underground roadways containing the monorail track, using an RGB industrial camera that meets mining explosion-proof standards; a preprocessing module for normalizing the size and pixel values ​​of the original images; a track segmentation core module using a U-Net deep neural network model to generate pixel-level track segmentation probability maps; and a post-processing and decision module for converting the probability maps into track region information and outputting it to the control system.

2. The system as described in claim 1, characterized in that, The U-Net deep neural network model includes an encoder path, a bottleneck layer, and a decoder path. The encoder path consists of three downsampling blocks, each containing two 3×3 convolutional layers and one 2×2 max-pooling layer, outputting feature maps of 112×112×32, 56×56×64, and 28×28×128 respectively. The bottleneck layer contains two 3×3 convolutional layers, outputting a 28×28×256 feature map. The decoder path consists of three upsampling blocks. Each upsampling block is first upsampled using a 2×2 transposed convolution, then concatenated with the feature map of the corresponding level of the encoder through skip connections, and subsequently passed through two 3×3 convolutional layers to gradually restore the spatial resolution. Finally, a 224×224 single-channel segmentation probability map is output through a 1×1 convolutional layer and a Sigmoid activation function.

3. The system as described in claim 1, characterized in that, During the U-Net model training phase, a composite loss function consisting of binary cross-entropy loss and Dice loss is used for optimization. The binary cross-entropy loss measures the overall classification error between the predicted probability and the true label, while the Dice loss improves the overlap of the orbital regions and alleviates the imbalance between positive and negative samples. The two are linearly weighted with equal weights. At the same time, data augmentation strategies simulating typical downhole interference are introduced during training, including: randomly adjusting the brightness channel in the HSV color space to simulate brightness degradation; superimposing Gaussian noise and salt-and-pepper noise to simulate sensor noise and particulate interference; and superimposing semi-transparent dark brown or gray textures in local image regions and applying Gaussian blur to simulate dust occlusion effects.

Images