A mine underground track line detection method based on machine vision
By deploying machine vision detection models in the mine environment, combining TensorRT accelerated inference and intelligent completion functions, the robustness and accuracy of traditional track line detection methods in complex environments is solved, efficient and accurate track line detection and turntwitch information provision, and the safety of locomotive operation is improved.
Patent Information
- Application Number
- CN202411762790.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-03
- Publication Date
- 2025-04-04
AI Technical Summary
The railway transportation environment under the mine is complex, and the traditional rail line detection method is poor in complex environments, especially under the influence of factors such as bending rails and light changes, which limits its practicality.
Using the mine underground track line detection method based on machine vision, the trained track line detection model is deployed to the embedded AI edge computing platform, and TensorRT accelerated inference is used to receive video streams from the front-end camera in real time for track line detection model inference, achieving rapid detection of track lines, and intelligently completing the missing track mask when necessary.
It significantly improves the efficiency and accuracy of track line detection, can accurately detect track lines in complex mine environments, provide turnout information, and intelligently complete the missing track mask under water accumulation, ensure the accuracy of track line detection and improve the safety of locomotive operation.
Smart Images
Figure CN119763061A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of mine technology, and in particular to a mine underground track line detection method based on machine vision. Background Art
[0002] The railway transportation environment under the mine is relatively complex, especially near the track. These areas are usually highly random and unpredictable, such as pedestrians, mine cars, and foreign objects such as falling rocks, which may pose a serious threat to the safe transportation of locomotives. In order to effectively solve the problem of foreign object intrusion, it is necessary to accurately detect the track, thereby providing a reliable basis for establishing the locomotive running limit. By extracting track information, the safe area for locomotive driving can be clarified, thereby avoiding the potential dangers caused by foreign objects to locomotive operation.
[0003] Traditional track line detection methods usually rely on the feature information of track lines and use clustering and other technologies to extract track lines. However, such methods have poor robustness in complex environments, especially low detection accuracy when facing curved tracks, and are easily affected by external environmental factors such as lighting changes, which limits their practicality. Summary of the invention
[0004] Based on the technical problems existing in the background technology, the present invention proposes a mine underground track line detection method based on machine vision, which significantly improves the detection efficiency and the safety of locomotive operation.
[0005] The present invention proposes a method for detecting underground mine track lines based on machine vision, comprising the following steps:
[0006] Step 1: Deploy the trained track line detection model to the embedded AI edge computing platform and use TensorRT to accelerate reasoning;
[0007] Step 2: Receive the video stream from the front-end camera in real time to perform track line detection model inference, obtain the predicted segmentation mask of the track, and realize rapid detection of the track line.
[0008] Furthermore, in step one, specifically: convert the pt weights of the trained track line detection model into a plain text wts file, load the weights in TensorRT, define the network structure of the track line detection model, and build the TensorRT engine.
[0009] Furthermore, in step 2, specifically: after receiving the video stream from the front-end camera, each frame image before being input into the track line detection model is resized and uniformly filled to the same pixel, and the adjusted image is used as the input of the track line detection model.
[0010] Furthermore, each frame image is resized and uniformly filled to 640×640 pixels, specifically: the image of the original ratio is scaled to obtain the primary adjusted image: the primary adjusted image is filled to the target size of 640×640 pixels.
[0011] Furthermore, in step 2, if the number of tracks inferred by the track line detection model is 1, the track area detected by the track line detection model is complete; if it is greater than 1, the track area detected by the track line detection model is missing.
[0012] Furthermore, if there is a missing track area, the missing track mask is automatically completed. The automatic completion process is as follows:
[0013] Extract the coordinate points of the front and rear track masks on the left and right sides;
[0014] Combine the left contour coordinate points of all track masks into a left coordinate point set, and combine the right contour coordinate points of all track masks into a right coordinate point set;
[0015] Use a quadratic function to perform curve fitting on the left coordinate point set and the right coordinate point set respectively, to obtain a fitting function of the left contour and a fitting function of the right contour;
[0016] Get the maximum and minimum values of the ordinates in the left coordinate sets of different orbital masks, and get the discontinuous ordinate points in the middle, and get the coordinate points of the left contour line of the missing part through the fitting function of the left contour;
[0017] Get the maximum and minimum values of the ordinates in the right coordinate sets of different orbital masks, and get the discontinuous ordinate points in the middle, and get the coordinate points of the right contour line of the missing part through the fitting function of the right contour;
[0018] Reverse the order of the coordinate points of the right contour line, and merge them with the coordinate points of the left contour line to form a complete polygonal contour, and fill the interior of the polygonal contour as a new mask area;
[0019] The new mask region and the detected track mask region are merged to form a final track region as a complete track region.
[0020] Furthermore, the training process of the track line detection model is as follows:
[0021] Collect and annotate the multi-scenario motion images of underground locomotives, and divide the annotated data set into a training set and a validation set;
[0022] Use the YOLO series segmentation model as a pre-trained model, and train the pre-trained model based on the training set to obtain a track line detection model;
[0023] Verify the training performance of the track line detection model based on the validation set.
[0024] Furthermore, the multi-scene motion diagram of the underground locomotive is collected, specifically:
[0025] Install the camera on the front of the locomotive and adjust the camera angle so that the track where the locomotive is currently located is in the center of the video screen;
[0026] The front video is obtained through the front camera of the locomotive, the video is framed, and the image frames containing the track are filtered out to construct a dataset for track line detection.
[0027] Furthermore, a polygon tool is used to annotate the data set, and only the track area where the current locomotive can travel is annotated, and the remaining track areas are not annotated, and the annotated label categories include straight track, turning track and track without turnout;
[0028] If a positioning switch appears in the track area that the current locomotive can travel, the track will be marked as a straight track;
[0029] If there is a reverse turnout in the track area where the current locomotive can travel, the track will be marked as a turning track;
[0030] If there is no turnout in the track area where the current locomotive can travel, the track is marked as a track without turnout;
[0031] If there are multiple turnouts on the traversable track at the same time, the nearest turnout will be used as the label of the track.
[0032] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned underground mine track line detection method when executing the computer program.
[0033] The advantage of the machine vision-based mine underground track line detection method provided by the present invention is that the trained track line detection model is deployed on the embedded AI edge computing platform, which has significant real-time and accuracy advantages. The method can detect the track area where the locomotive can currently travel and provide turnout information. At the same time, in view of the possible water-covered track in the mine environment, the missing track mask can be intelligently completed to further ensure the accuracy of track line detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 It is a schematic diagram of the process of the present invention;
[0035] Figure 2 An example diagram of the data annotation method;
[0036] Figure 3 This is a schematic diagram of the detection results of the track line. DETAILED DESCRIPTION
[0037] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.
[0038] like Figures 1 to 3 As shown, the present invention proposes a method for detecting underground mine track lines based on machine vision, comprising the following steps:
[0039] Step 1: Deploy the trained track line detection model to the embedded AI edge computing platform and use TensorRT to accelerate reasoning;
[0040] Use TensorRT to accelerate reasoning. Specifically, convert the pt weights of the trained track line detection model into a plain text wts file, load the weights in TensorRT, define the network structure of the track line detection model, and build the TensorRT engine.
[0041] Step 2: Receive the video stream from the front-end camera in real time to perform track line detection model inference, obtain the predicted segmentation mask of the track, and realize rapid detection of the track line.
[0042] The process of track line detection model inference is as follows: the image obtained by the camera is scaled and adjusted to the image size acceptable to the model; the prediction frames with low confidence are filtered out according to the confidence, and non-maximum suppression is applied to the filtered prediction frames to remove overlapping prediction frames and retain only the optimal prediction frames; the coordinates of the prediction frame are adjusted according to the size of the original size image and the scaling ratio during preprocessing to align it with the original size image; the segmentation mask of the corresponding target is selected according to the adjusted target frame; finally, the predicted segmentation mask of the track is obtained after combination.
[0043] The image captured by the camera is scaled and adjusted to the image size acceptable to the model. Specifically, each frame of the image is resized and uniformly filled to 640×640 pixels. Specifically:
[0044] If the video stream resolution is W and H, scale the original image to get the first adjusted image:
[0045]
[0046] Among them, Wm ,H m To adjust the width and height of the image for the first time, W and H are the width and height of the original scale image respectively;
[0047] In order to fill the first adjusted image to the target size, it is also necessary to calculate the width (P w ) or high (P h ):
[0048]
[0049] Thus, through the width (P w ) or high (P h ) to an image size that the model can accept.
[0050] In this embodiment, in a mine environment, the track may be covered with a large amount of water, which will cause the track area mask detected by the track line detection model to be missing. Therefore, the function of intelligently completing the missing mask is added to the post-processing module. This function is suitable for the scene where the middle area of the track is covered by water. That is, if the number of tracks inferred by the track line detection model is 1, the track area detected by the track line detection model is complete. If it is greater than 1, the track area detected by the track line detection model is missing. The automatic completion implementation process is specifically (a1) to (a7):
[0051] (a1) Extract the coordinate points of the front and rear track masks on the left and right sides;
[0052] That is, the coordinate points of the leftmost and rightmost pixel values of each row of the track mask are obtained, and finally the coordinate points of the left and right contours of the track mask are obtained;
[0053] (a2) combining the left contour coordinate points of all track masks into a left coordinate point set, and combining the right contour coordinate points of all track masks into a right coordinate point set;
[0054] (a3) using a quadratic function to perform curve fitting on the left coordinate point set and the right coordinate point set respectively, to obtain a fitting function of the left contour and a fitting function of the right contour;
[0055] Using the quadratic function y=ax 2 +bx+c fits the coordinate point sets on both sides. The fitting process is as follows:
[0056] The left (or right) coordinate point set is (x1, y1), (x2, y2), ..., (x n ,y n ), n is the total number of coordinate points on the left and right sides. For each point (x i ,y i), according to the quadratic equation It can be written as a matrix equation:
[0057]
[0058] The pseudo-inverse of the matrix is used to solve Xp=y to obtain the values of a, b, and c. Thus, the fitting function of the left contour and the fitting function of the right contour are obtained.
[0059] (a4) obtaining the maximum and minimum values of the ordinates in the left coordinate sets of different track masks, and obtaining the discontinuous ordinate points in the middle, and obtaining the coordinate points of the left contour line of the missing part through the fitting function of the left contour;
[0060] (a5) obtaining the maximum and minimum values of the ordinates in the right coordinate sets of different track masks, and obtaining the discontinuous ordinate points in the middle, and obtaining the coordinate points of the right contour line of the missing part through the fitting function of the right contour;
[0061] (a6) reversing the order of the coordinate points of the right contour line and merging them with the coordinate points of the left contour line to form a complete polygonal contour, and filling the interior of the polygonal contour as a new mask area;
[0062] (a7) The new mask region and the detected track mask region are merged to form a final track region as a complete track region.
[0063] It is understandable that when the track line detection model detects that there is no missing track mask, there is no need to complete it through (a1) to (a7).
[0064] Through step one to step two, this embodiment uses a single-stage neural network segmentation track line detection model to perform track line detection in the mine, and is deployed on an embedded AI edge computing platform, which has significant real-time and accuracy advantages. This method can detect the track area where the locomotive can currently travel and can provide turnout information. At the same time, in view of the possible situation of water covering the track in the mine environment, the missing track mask can be intelligently completed to further ensure the accuracy of track line detection. This comprehensive detection capability and real-time response capability provide a safer and more reliable guarantee for locomotive operation.
[0065] In this embodiment, the training process of the track line detection model is as follows (b1) to (b3):
[0066] (b1) Collect and annotate the multi-scene motion images of underground locomotives, and divide the annotated data set into a training set and a validation set;
[0067] A high-resolution wide dynamic camera is installed on the front of the locomotive. By adjusting the camera angle of view, the track where the locomotive is currently located is located in the center of the video screen. The front video is obtained through the front camera of the locomotive, the video is framed, and the image frames containing the track are filtered out to construct a data set for track line detection.
[0068] When annotating the images in the data set, use the polygon tool to annotate the data set and label different tracks accordingly. Note here that only the track area where the current locomotive can travel is annotated, and the remaining track areas are not annotated. Specifically, the annotated label categories include straight tracks, turning tracks, and tracks without switches; among them, if a positioning switch appears in the track area where the current locomotive can travel, the track is marked as a straight track; if a reverse switch appears in the track area where the current locomotive can travel, the track is marked as a turning track; if there is no switch in the track area where the current locomotive can travel, the track is marked as a track without switches; if multiple switches appear in the traversable track at the same time, the nearest switch is used as the label of the track.
[0069] The track line detection model trained by the above annotation method can identify the track area where the locomotive is currently traveling instead of all the track areas in the image. The annotated label file is converted into a label format that can be used for training, and the images and labels are divided into a training set and a validation set.
[0070] (b2) Using the YOLO series segmentation model as a pre-trained model, the pre-trained model is trained based on the training set to obtain a track line detection model;
[0071] During the training process of the track line detection model, specify the address and label name of the data set (i.e., training set), set the parameters used for training, and perform model training. The parameters used for model training are shown in the following table:
[0072] parameter batch Resolution Optimizer Loss Function Learning Rate value 200 640 Adam Cross Entropy Loss 0.01
[0073] (b3) Verify the training performance of the track line detection model based on the validation set.
[0074] Since the track line detection model is completed by segmentation, mAP50 is used as the evaluation index of track line detection. In the target segmentation task, mAP50 is one of the most commonly used evaluation criteria, which indicates the accuracy of the model in segmenting the target area under the condition of IoU of 0.5.
[0075] By training the track line detection model through (b1) to (b3) and regarding track line detection as an image segmentation problem, real-time and accurate detection of the track line can be better achieved, significantly improving the detection efficiency and the safety of locomotive operation. The trained track line detection model can be deployed on the edge computing platform, which has significant real-time and accuracy advantages.
[0076] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A mine underground track line detection method based on machine vision, characterized in that: The steps include: Step 1: Deploy the trained track line detection model to the embedded AI edge computing platform and use TensorRT to accelerate reasoning; Step 2: Receive the video stream from the front-end camera in real time to perform track line detection model inference, obtain the predicted segmentation mask of the track, and realize rapid detection of the track line.
2. The method for detecting underground mine track line based on machine vision according to claim 1, characterized in that: In step one, specifically: convert the pt weights of the trained track line detection model into a plain text wts file, load the weights in TensorRT, define the network structure of the track line detection model, and build the TensorRT engine.
3. The method for detecting underground mine track line based on machine vision according to claim 1, characterized in that: In step 2, specifically: after receiving the video stream from the front-end camera, each frame image before being input into the track line detection model is resized and uniformly filled to the same pixel, and the adjusted image is used as the input of the track line detection model.
4. The method for detecting underground mine track line based on machine vision according to claim 3 is characterized in that: Each frame image is resized and uniformly filled to 640×640 pixels. Specifically, the image of the original ratio is scaled to obtain the first adjusted image: the first adjusted image is filled to the target size of 640×640 pixels.
5. The method for detecting underground mine track line based on machine vision according to claim 1, characterized in that: In step 2, if the number of tracks inferred by the track line detection model is 1, the track area detected by the track line detection model is complete; if it is greater than 1, the track area detected by the track line detection model is missing.
6. The method for detecting underground mine track line based on machine vision according to claim 5, characterized in that: If there is a missing track area, the track mask of the missing part is automatically completed. The automatic completion process is as follows: Extract the coordinate points of the front and rear track masks on the left and right sides; Combine the left contour coordinate points of all track masks into a left coordinate point set, and combine the right contour coordinate points of all track masks into a right coordinate point set; Use a quadratic function to perform curve fitting on the left coordinate point set and the right coordinate point set respectively, to obtain a fitting function of the left contour and a fitting function of the right contour; Get the maximum and minimum values of the ordinates in the left coordinate sets of different orbital masks, and get the discontinuous ordinate points in the middle, and get the coordinate points of the left contour line of the missing part through the fitting function of the left contour; Get the maximum and minimum values of the ordinates in the right coordinate sets of different orbital masks, and get the discontinuous ordinate points in the middle, and get the coordinate points of the right contour line of the missing part through the fitting function of the right contour; Reverse the order of the coordinate points of the right contour line, and merge them with the coordinate points of the left contour line to form a complete polygonal contour, and fill the interior of the polygonal contour as a new mask area; The new mask region and the detected track mask region are merged to form a final track region as a complete track region.
7. The method for detecting underground mine track lines based on machine vision according to claim 1, characterized in that: The training process of the track line detection model is as follows: Collect and annotate the multi-scenario motion images of underground locomotives, and divide the annotated data set into a training set and a validation set; Use the YOLO series segmentation model as a pre-trained model, and train the pre-trained model based on the training set to obtain a track line detection model; Verify the training performance of the track line detection model based on the validation set.
8. The method for detecting underground mine track lines based on machine vision according to claim 1, characterized in that: Collect multi-scene motion pictures of underground locomotives, specifically: Install the camera on the front of the locomotive and adjust the camera angle so that the track where the locomotive is currently located is located in the center of the video screen; The front video is obtained through the front camera of the locomotive, the video is framed, and the image frames containing the track are filtered out to construct a dataset for track line detection.
9. The method for detecting underground mine track line based on machine vision according to claim 7, characterized in that: Use the polygon tool to label the dataset, only label the track area where the current locomotive can travel, and do not label the other track areas. The label categories include straight track, turning track and track without turnout. If a positioning switch appears in the track area that the current locomotive can travel, the track will be marked as a straight track; If there is a reverse turnout in the track area where the current locomotive can travel, the track will be marked as a turning track; If there is no turnout in the track area where the current locomotive can travel, the track is marked as a track without turnout; If there are multiple turnouts on the traversable track at the same time, the nearest turnout will be used as the label of the track.
10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for detecting underground mine track lines according to any one of claims 1 to 9 is implemented.