A rail foreign body detection method based on semi-automatic annotation and improved deep learning

Abstract

This invention discloses a semi-automatic annotation and improved deep learning-based method for detecting foreign objects on railway tracks. The method includes: semi-automatic annotation of a dataset based on thermal infrared human body segmentation results; improvement of the ResNet 50 neural network for all-weather outdoor railway track environments to identify different environmental scenarios; improvement of the YOLOv5 neural network based on scene recognition results to identify foreign objects on the tracks; extraction of railway track contours from images, combined with the location of the foreign object, to design an alarm strategy for foreign object intrusion on the tracks, and to perform tiered alarm judgments. By introducing deep learning scene classification combined with an improved target recognition network, this invention improves the effectiveness of foreign object identification on railway tracks in different environmental scenarios, enhances the accuracy of alarms for foreign object intrusion on railway tracks, and, combined with a scientific alarm strategy, reduces the possibility of train accidents caused by foreign object intrusion on railway tracks.

Description

Technical field:

[0001] The present invention relates to the fields of computer vision and pattern recognition and intelligent transportation systems (ITS), and in particular to a rail foreign body detection method based on semi-automatic labeling and improved deep learning. Background technology:

[0002] Traditional detection methods require installing multiple cameras along the rails, which is costly and has poor real-time performance.

[0003] In the traditional detection method, Duan Jiaying of China Academy of Railway Sciences (Duan Jiaying. Research on Wireless Sensor Network for High-speed Train Operation Environment Monitoring [D]. China Academy of Railway Sciences, 2017.) proposed to improve the existing railway monitoring equipment, add a wireless transmission module, realize the wireless transmission of information, and at the same time be able to receive monitoring information on the ground, and process, store and display it, effectively solving the problem of high wiring cost and inflexible detection of wired networks. Lin Yubing of North China University of Technology (Lin Yubing. Research on the Application of Three-dimensional Lidar Images in Railway Foreign Object Intrusion Detection [D]. North China University of Technology, 2018.) proposed to use three-dimensional laser radar for railway foreign object intrusion detection. By combining GL-1025 laser radar with a pan-tilt system to form a three-dimensional laser radar system for information collection, the three-dimensional point cloud data is converted into a format, three-dimensional modeling is performed, and finally the OTSU algorithm, K-means clustering and DBSCAN clustering are used for target detection. The experimental results show that the algorithm can automatically classify and detect targets, and has good detection effects in different scenarios. However, due to the huge amount of point cloud data and the requirements for hardware, it is very difficult to complete real-time detection. An active track status monitoring system (CN113484420A) is installed on the track. A railway power grid inspection robot (CN114194231A) adopts a robot inspection method, which provides a new innovative direction for foreign body detection on the rail.

[0004] Among the technical methods based on deep learning, Wang Lei (Wang Lei. Research and system design of high-precision pedestrian detection algorithm for railway driving safety [D]. Beijing Jiaotong University, 2019.) proposed a small-scale pedestrian detection algorithm. The algorithm was based on the Faster R-CNN network and optimized the network structure from the weak semantic segmentation loss function, channel radiation transformation and feature fusion. Liu Hui et al. (Liu Hui, Zhang Lishuai, Shen Yue, et al. Real-time detection method of pedestrians in orchards based on improved SSD [J]. Transactions of the Chinese Society of Agricultural Machinery, 2019, 50(4):8.) proposed a real-time pedestrian monitoring algorithm, introducing void convolution into SSD to expand the receptive field of the convolutional neural network. Rui Ting et al. (Rui Ting, Fei Jianchao, Zhou You, et al. Pedestrian detection based on deep convolutional neural network [J]. Computer Engineering and Applications, 2016, 52(13):5.) proposed an algorithm for detecting pedestrians by constructing a convolutional neural network with a multi-layer network structure. The algorithm has a high efficiency for pedestrian detection. Wang Ning (Wang Ning. Railway intrusion pedestrian classification algorithm based on feature fusion [D]. Beijing Jiaotong University.) proposed a railway pedestrian classification algorithm based on feature fusion, i.e., the fusion of convolutional neural network features and manual pedestrian features. He Wenyu et al. from Jiangxi University of Science and Technology (He Wenyu, Yang Jie, Zhang Tianlu. Track foreign body intrusion detection algorithm based on deep learning [J]. Computer Engineering and Design, 2020, 41(12):8.) proposed to combine traditional machine learning algorithms to perform basic image processing, delineate the intrusion area, and use the deep learning YOLO-V3 network model for multiple training. The method of detecting the demarcated intrusion area and whether there are foreign objects in the area through the training model makes it possible to detect foreign body intrusion in real time. However, this requires the installation of multiple cameras along the line, which is costly, and its mobility is insufficient, and the real-time advantage of the YOLOv3 algorithm is not fully utilized.

[0005] In summary, although the current foreign body detection methods based on deep learning technology have achieved initial results in other fields, there is still room for further improvement in the detection rate, detection accuracy, detection efficiency and robustness in the railway intelligent transportation system segment. Summary of the invention:

[0006] The purpose of the embodiments of the present invention is to provide a semi-automatic labeling and improved deep learning method for detecting foreign objects on the rails, aiming to promote the solution to the problems that the existing methods for detecting foreign objects on the rails are not real-time and robust, and are difficult to cope with the intrusion of foreign objects on the rails in various complex environments.

[0007] A rail foreign body detection method based on semi-automatic labeling and improved deep learning comprises the following steps:

[0008] (1) A human body segmentation algorithm based on thermal infrared is proposed, the segmentation results are mapped to visible light images, the visible light human body annotation box is quickly obtained, and a YOLOv5 training dataset is established;

[0009] (2) A scene classification algorithm based on the improved ResNet 50 model is proposed to guide the human recognition method of the human recognition model in different scenarios;

[0010] (3) Using the target detection algorithm of the proposed improved YOLOv5 model, the classification model is trained offline to detect foreign objects on the rails and realize foreign object detection;

[0011] (4) A rail contour extraction algorithm based on traditional algorithms and improved deep learning is proposed to achieve rail segmentation and contour extraction in a single image;

[0012] (5) A foreign object alarm strategy algorithm is proposed based on the track segmentation results and the location of the foreign object, which guides the analysis of the alarm level of the entire image and realizes the alarm level judgment.

[0013] Furthermore, the step (1) of quickly obtaining the visible light human body annotation frame is obtained by the following steps: by selecting a thermal infrared camera and a visible light camera with the same focal length, in a scene where the scene temperature is below 25 degrees Celsius, the thermal infrared and visible light videos are recorded simultaneously; on this basis, by taking advantage of the fact that the thermal infrared human body is easily segmented by the Otsu algorithm below 25 degrees Celsius, the Otsu algorithm is used to automatically complete the segmentation of the thermal infrared human body image, and then the segmentation result is mapped to the visible light image, thereby directly obtaining a large number of visible light human body annotation frame results; finally, a small amount of manual correction is performed on a small number of possible segmentation errors, and the visible light human body annotation frame is semi-automatically obtained to establish a YOLOv5 training data set.

[0014] Furthermore, the scene classification algorithm of the improved ResNet 50 model described in step (2) is as follows: for the ResNet50 scene classification model, a transfer learning mechanism is introduced to train the model, and the model weights pre-trained on ImageNet are migrated to the model to accelerate the convergence of the network model; in the selection of the activation function in step (2), the Relu function is specifically changed to the LRelu function to prevent the gradient from disappearing.

[0015] Furthermore, the target detection algorithm of the improved YOLOv5 model described in step (3) is as follows: for the YOLOv5 target detection model, after obtaining the data, Mosaic data enhancement and real-time data loading are used to improve the bounding box prediction loss function, and the model composition of the GT target bounding box UID allocator, the GT target bounding box UID matcher, and the bounding box position and classification loss weight algorithm is used to improve the position accuracy and classification accuracy; in step (3), in terms of collecting combined target features, a convolutional attention mechanism is added, and for the prediction part, on the basis of the traditional sigmoid function σ(x), an h-swish function is introduced in the MobileNetv3 network, and the CIOU_Loss of the loss function originally used in the Bounding box is changed to GIOU_Loss; finally, a deepsort tracking module is added to the detection module of the YOLOv5 algorithm to prevent missed detection, thereby improving the robustness of the system in detecting foreign objects.

[0016] Furthermore, the track line detection method based on the traditional algorithm described in step (4) is: for straight rails, the edge features of the railway track are extracted using Hough transformation; for curved rails, the rails are first segmented to obtain segmented straight rails, and then the edge features of the railway track are extracted using Hough, and then they are merged to obtain the curved track edge extraction. Combining the two, a track line contour segmentation algorithm suitable for four different types of complex environments, namely straight lines, curves, multi-track straight lines, and multi-track curves, is proposed; and the rail contour extraction algorithm based on improved deep learning described in step (4) is: a deep learning algorithm based on DeepLabv3+ semantic segmentation network is used to extract the rail contour; finally, the data results obtained by the two models are processed by contour union, and their common parts are left as the final extracted track contour.

[0017] Furthermore, the foreign object alarm strategy algorithm described in step (5) is: using step (4) to obtain the corresponding coordinates of the track contour of a single image, using step (3) to obtain the coordinates of the foreign object in a single image, and making a weighted judgment on the foreign object intrusion situation of the entire image based on the proposed custom function and algorithm, and finally normalizing it to the alarm level of the entire image.

[0018] Compared with the existing track foreign body detection technology, the present invention has the following advantages and effects: the method described in step (1) of the present invention can semi-automatically obtain a visible light pedestrian annotation frame data set in the data preprocessing stage, and further manually improve the annotation accuracy during the later review; the method described in step (2) is aimed at outdoor scenes of railway tracks, where common lighting, weather, etc. have large differences. The present invention introduces a scene classification algorithm based on improved deep learning, and guides the subsequent human body detection model to complete high-precision foreign body recognition according to the scene classification results, thereby solving the problem that such models are difficult to apply to complex environments under different scenes; the target detection algorithm based on the improved YOLOv5 model described in step (3) improves the loss function after obtaining the data, improves the position accuracy and classification accuracy, and also adds a convolutional attention mechanism in collecting combined target features. In addition, a tracking module is added to prevent the system from missing detection, thereby ultimately solving the problem that the track foreign body detection system is not robust enough. , the traditional YOLOv5 algorithm framework has the problem of insufficient accuracy in detecting certain specific environmental conditions; the rail contour extraction algorithm based on traditional algorithms and improved deep learning in step (4) combines the advantages of traditional rail contour segmentation methods and rail contour segmentation methods based on deep learning, specifically using an improved Hough transform algorithm to detect the track line and segment the rail contour, and using a deep learning algorithm based on DeepLabv3+ semantic segmentation network to perform track line semantic segmentation and extract the rail contour, and then performing contour union processing on the data results obtained by the two, ultimately improving the accuracy of extracting the rail contour; the foreign object alarm strategy algorithm designed in step (5) can guide the analysis of the alarm level of the entire image, derive a linear alarm level according to a custom function, realize accurate judgment and processing of track foreign object intrusion, greatly reduce the occurrence of false alarms and excessive alarms, and reduce the possibility of accidents.

[0019] Another advantage of the present invention is that the hardware conditions of the system are based on vehicle-mounted cameras. Compared with the traditional track foreign object detection method of installing static cameras in sections along the track to monitor the track in real time 24 hours a day, it has stronger economic applicability and higher fault tolerance, greatly improving the efficiency of track foreign object intrusion detection and reducing the possibility of train accidents caused by foreign objects invading the track. Description of the drawings:

[0020] Figure 1 This is a general flow chart of the rail foreign body detection method using semi-automatic labeling and improved deep learning provided by an embodiment of the present invention.

[0021] Figure 2 The invention provides a thermal infrared-based human body segmentation algorithm, which maps the segmentation result to a visible light image, quickly obtains a visible light human body annotation box, and establishes a YOLOv5 training data set.

[0022] Figure 3 The present invention provides a scene classification algorithm based on an improved ResNet 50 model, which guides the execution of a human body recognition method of a human body recognition model in different scenes.

[0023] Figure 4 This is the YOLOv5 network model structure diagram.

[0024] Figure 5 This is an implementation diagram of a rail profile extraction algorithm based on a traditional algorithm and improved deep learning provided in an embodiment of the present invention. Specific implementation method:

[0025] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0026] The application principle of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0027] The overall process of the present invention is as follows Figure 1 As shown, a method for detecting foreign objects on a rail using semi-automatic labeling and improved deep learning in an embodiment of the present invention includes the following steps:

[0028] S1: By selecting a thermal infrared camera and a visible light camera with the same focal length, thermal infrared and visible light videos are recorded simultaneously in scenes where the scene temperature is below 25 degrees Celsius. On this basis, the OTSU algorithm is used to automatically segment the thermal infrared human image, taking advantage of the fact that the thermal infrared human body is easily segmented by the OTSU algorithm below 25 degrees Celsius. The segmentation results are then mapped to the visible light image, thereby directly obtaining a large number of visible light human body annotation results. Finally, a small amount of manual correction is performed on a small number of possible segmentation errors, and the visible light human body annotation box is semi-automatically obtained to establish a YOLOv5 training data set.

[0029] S2: The training data set is divided into various environments according to the environmental scenes of the pictures and labeled, including six categories: sunny, cloudy, rainy, foggy, snowy and night. Then, the proposed classification algorithm based on the improved deep residual network ResNet 50 is used to perform offline training on the labeled data set to obtain a ResNet 50 model that can recognize the image environment; the YOLOv5 training data set established in S1 is classified by using the ResNet 50 model to obtain a variety of training data sets based on different environmental scenes; the improved ResNet 50 classification algorithm is as follows: first, a transfer learning mechanism is introduced to train the model, and the model weights pre-trained on ImageNet are transferred to our model to accelerate the convergence of the network model; second, in the selection of activation function, the Relu function is changed to the LRelu function to prevent the gradient of the deep network from disappearing.

[0030] S3: Next, the improved YOLOv5 target detection algorithm is used to perform offline training on the various training data sets based on different environmental scenarios obtained in S2 separately, so as to obtain the improved YOLOv5 model based on different environmental scenarios; the steps of the improved YOLOv5 target detection algorithm are as follows: after obtaining the data, Mosaic data enhancement and real-time data loading are used to improve the bounding box prediction loss function, and the models of GT target bounding box UID allocator, GT target bounding box UID matcher, and bounding box position and classification loss weight algorithm are used to improve the position accuracy and classification accuracy; in terms of collecting combined target features, a convolutional attention mechanism is added; finally, in the prediction part, on the basis of the traditional sigmoid function σ(x), the h-swish function is introduced in MobileNetv3, and the CIOU_Loss of the loss function originally used in the Bounding box is changed to GIOU_Loss; finally, a deepsort tracking module is added to the detection module of the YOLOv5 algorithm to prevent missed detection, thereby improving the robustness of the system in detecting foreign objects.

[0031] S4: The model obtained through the above training can detect whether there are foreign objects intruding during the train driving scene, and a new rail contour extraction algorithm based on improved deep learning is proposed to segment the rail and extract the contour. The traditional track line detection method uses Hough transform to detect the pre-processed track line image, which can roughly locate the possible track line in the image, and then screen the track line position based on the characteristics of the track line itself, and finally segment and extract the rail contour; while the rail contour extraction method based on deep learning uses the DeepLabv3+ semantic segmentation network to perform track line semantic segmentation to extract the rail contour. We improve the traditional rail contour extraction method and the rail contour extraction method based on deep learning. The specific method is to take the contour union processing of the data results obtained by the two models, leaving their common parts as the final extracted rail contour, so as to be more accurately used for alarm analysis. After the rail contour is extracted, the results obtained by running the two models are passed into the alarm strategy model.

[0032] S5: Use the proposed alarm strategy model to conduct alarm analysis on the incoming results; the proposed alarm strategy model algorithm is: for the foreign object detection results and rail contour extraction results input by S3 and S4, the upper-level input features are weightedly fused, and finally normalized to the alarm level of a single image; the principle of the alarm strategy model is: feature extraction is performed on the input results of foreign object detection to obtain the image absolute coordinate vectors of the positions of n foreign objects.

[0033] The absolute coordinate vector of the foreign body position is shown in (1), where x i is the horizontal coordinate of the foreign body in the picture, y i is the vertical coordinate of the foreign body in the picture.

[0034] A i =(x i ,y i ),i∈{1,2,……,n} (1)

[0035] Based on this result, feature extraction is performed on the input result of semantic segmentation to obtain the absolute value of the pixel width of the railroad track in the i-th row of pixels, and its expression is shown in (2).

[0036]

[0037] Where W i1 , W i2 The semantic boundary of the railroad track is at the ordinate y i The horizontal axis value of .

[0038] At the same time, the distance between the foreign object and the semantic boundary of the rail is obtained, and its expression is shown in (3).

[0039]

[0040] The alarm judgment value expression is shown in (4).

[0041]

[0042] Alarm when P≥0.5.

[0043] S6: Based on the output of the S5 alarm strategy model, determine whether the train needs to sound an alarm.

[0044] like Figure 2 , the specific process of S1 is:

[0045] S11: Select thermal infrared cameras and visible light cameras with the same focal length and deploy them on the train.

[0046] S12: When the scene temperature is below 25 degrees Celsius, thermal infrared and visible light video are recorded simultaneously.

[0047] S13: The video stream recorded and collected by S12 is processed by the frame extraction method, with a time interval of 1s, that is, a picture is extracted from the video every 1s. In particular, for images with human body foreign bodies, dual threads are used to perform time sequence matching on the pictures collected by the two cameras while extracting frames.

[0048] S14: On this basis, taking advantage of the fact that the thermal infrared human body can be easily segmented by the OTSU algorithm below 25 degrees Celsius, the OTSU algorithm is used to automatically complete the segmentation of the thermal infrared human body image.

[0049] S15: Map the thermal infrared human image segmentation results into visible light images, so as to directly obtain a large number of visible light human body annotation box results.

[0050] S16: Perform a small amount of manual correction on a small number of possible segmentation errors and semi-automatically obtain the visible light human body annotation box.

[0051] S17: Based on the visible light human body annotation results obtained semi-automatically in S15, they are divided into training set, test set and validation set according to the ratio of 8:1:1, and finally the YOLOv5 training data set is established.

[0052] like Figure 3 , the specific process of S2 is:

[0053] S21: First, migrate the loaded pre-trained model to our scene classification model. Then, label the training dataset established in S17 and input it into the improved ResNet 50 network for training. Use the cross-validation method to train and evaluate the performance of the model, and finally obtain an optimal classifier that can recognize the environmental scene where the training dataset images are located.

[0054] S22: In terms of transfer learning, you need to obtain the network weights of the ResNet 50 network pre-trained in the ImageNet dataset, use the command wget to download the pre-trained model, and use torch.load_state_dict(torch.load(". / resnet50-pre.pth"), strict=False) to load the model pre-trained weights.

[0055] S23: In terms of activation function, in order to avoid the problem of gradient disappearance, the Relu function is changed to the LRelu function to ensure that the network can converge well at deeper layers. The LRelu function is shown in (5).

[0056]

[0057] S24: In terms of model training, batch_size=64, epoch=20, and the model is trained to obtain a classifier that can recognize the environmental scene where the training data set images are located.

[0058] S25: Based on this classifier, we classify the YOLOv5 training dataset established in S17 based on environmental scenarios, and finally obtain a variety of YOLOv5 training datasets based on different environmental scenarios.

[0059] like Figure 4 , the specific process of S3 is as follows:

[0060] S31: According to the multiple training data sets based on different environmental scenarios obtained in S25, for different environmental scenarios, different environmental scenario training data sets are used to train an improved YOLOv5 model based on this environmental condition, and a rail foreign body detection algorithm under different environmental scenario conditions based on the improved YOLOv5 network is obtained.

[0061] S32: In the data loader, YOLOv5 uses mosaic data enhancement at the input end. Set the parameter augment = True in create_dataloader, check whether Mosaic in hyp.scratch.yaml is set to 1.0, and retain the original information. After each detection input of a new picture, Mosaic data enhancement is performed by adjusting the image brightness and using the mosaic method to achieve the purpose of real-time data enhancement. The four pictures are spliced ​​by random scaling, random cropping, and random arrangement.

[0062] S33: In terms of collecting combined target features, a convolutional attention mechanism is added. The attention mechanism Modified SAM is applied in YOLOv4, but the relevant attention mechanism is not applied in YOLOv5. Therefore, a convolutional attention mechanism module CBAM is added to the feature extraction network, and the accumulation function corresponding to the standard convolution layer + CBAM module is added to commom.py. In the backbone network, CBAM is added after the first convolution, and the corresponding "Conv" is changed to "Conv_CBAM".

[0063] S34: Improve the bounding box prediction loss function, and use the model of GT target bounding box UID allocator, GT target bounding box UID matcher, bounding box position and classification loss weight algorithm to improve the position accuracy and classification accuracy. The CIOU_Loss loss function originally used in the Bounding box is changed to GIOU_Loss. The specific transformation is shown in (6) and (7).

[0064]

[0065]

[0066] S35: In terms of prediction, the traditional sigmoid function σ(x) consumes a lot of computing resources on the mobile terminal, while the Relu6 function is easy to quantize and deploy. Therefore, based on the traditional sigmoid function σ(x), the h-swish function is introduced to speed up the process while maintaining accuracy. The h-swish function is shown in (8).

[0067]

[0068] S36: Finally, the deepsort tracking module is added to the detection module code of the YOLOv5 algorithm to prevent missed detection, and the target's appearance information is added to the calculation of inter-frame matching, thereby improving the system's robustness in detecting foreign objects.

[0069] like Figure 5 , the specific process of S4 is as follows:

[0070] S41: Improve the traditional track segmentation method: For straight rails, use Hough transform to extract the edge features of the railroad track; for curved rails, first segment the rails to obtain segmented straight rails, then use Hough transform to extract the edge features of the railroad track, and then merge them to obtain the curved track edge extraction; finally, combine the two to propose a track line contour segmentation algorithm suitable for four different types of complex environments: straight line, curve, multi-track straight line, and multi-track curve.

[0071] S42: Since natural scenes contain a large number of irrelevant targets, most of which have straight line components, which will have a great impact on the recognition of the track area, it is necessary to first divide the ROI (Region of Interest) area in the image. Identifying the rails in the ROI area can greatly improve the recognition accuracy and eliminate the interference of irrelevant factors. We use the cv2.setMouseCallback function to obtain the coordinates of the pixel points by clicking on the pixel points in the image; the ROI area is a rectangle. When selecting, just select the pixel coordinates of the two endpoints of the diagonal of the rectangle, and the corresponding coordinate values ​​can be returned, and the ROI area is drawn by the coordinate values.

[0072] S43: Before performing Hough transform, a preprocessing process is often included to enhance the image edge. The preprocessing of this algorithm includes three stages: grayscale value conversion, histogram equalization, and Gaussian filtering. In the grayscale value conversion stage, we use the cv2.cvtColor function with the parameter cv2.COLOR_RGB2GRAY to convert the original three-channel RGB color image into a grayscale image.

[0073] S44: After the image is converted to a single-channel grayscale image, histogram equalization is performed on the image to enhance the contrast. Histogram equalization enhances the image contrast by stretching the pixel intensity distribution range. The equalized image is only approximately uniformly distributed, and the dynamic range expansion of the image is essentially to expand the quantization interval and reduce the quantization level. There are obvious boundaries between regions. We use the cv2.equalizeHist function to perform histogram equalization on the grayscale image. For natural scenes under normal lighting conditions, after histogram equalization, the image contrast tends to increase, which is conducive to the detection of target edges.

[0074] S45: Perform Gaussian filtering on the image. Gaussian filtering performs weighted averaging on the entire image. The value of each pixel is obtained by weighted averaging the value of itself and other pixels in the domain, eliminating Gaussian noise in the image. The specific operation of Gaussian filtering is: use a Gaussian kernel (also called convolution, template) to scan each pixel in the image, and replace the value of the central pixel of the template with the weighted average gray value of the pixels in the domain determined by the template. We use the cv2.GaussianBlur function to perform Gaussian filtering on the image and define the Gaussian kernel size by ourselves.

[0075] S46: Perform Hough transform on the preprocessed image and extract the rail contour. The principle of Hough transform is to transform the curve (including straight line) in the image space into the parameter space, and determine the description parameters of the curve by detecting the extreme points in the parameter space, so as to extract the regular curve in the image. The polar coordinate system is used for representation in Hough line detection, and the polar coordinate expression of the straight line is shown in (9).

[0076] ρ = xcosθ + ysinθ (9)

[0077] In expression (9), ρ is the distance from the origin to the line, θ is the angle between the line and the y-axis, and (x, y) is the coordinate of a point on the line.

[0078] Substitute the coordinates of each point in the coordinate system into the formula, calculate the size of ρ for each given θ, and count the ρ of the same size to determine the straight line.

[0079] S47: We use a deep learning algorithm based on improved semantic segmentation to extract the rail contour. The specific process is: use the on-board camera to obtain the rail images collected during the train running, and send them to the DeepLabv3+ semantic segmentation network trained in advance by the system to perform track line semantic segmentation.

[0080] S48: Perform perspective transformation on the binary image containing only the track line obtained by segmentation to obtain a bird's-eye view, screen the valid track line pixel points, use the least squares method to perform polynomial fitting on the valid track line points, and output the polynomial fitting coefficients of the left and right track lines. Based on the coefficients, we can determine the segmented rail contour.

[0081] S49: Finally, the data results obtained by the above two models are processed by contour union, and their common parts are left as the rail contour we segmented and extracted for alarm analysis and subsequent data processing stages. This method has stronger accuracy and anti-interference ability than traditional track segmentation methods and track segmentation methods based on deep learning. It can effectively solve the problems of poor environmental adaptability, weak robustness, and inaccurate extraction of curved track contours of traditional track contour segmentation algorithms.

[0082] The specific processes of S5 and S6 are as follows:

[0083] The foreign object detection results and the rail semantic segmentation results are passed into the alarm strategy model, and the alarm strategy model is used to output whether an alarm is needed. If the model determines that an alarm is needed, an alarm is issued in the background, and the foreign object is framed with a red frame in the foreground video frame, triggering operations such as train alarms and emergency braking. If the model determines that an alarm is not needed, the foreign object that needs attention is framed with other color frames in the foreground monitoring video stream.

Claims

1. A rail foreign body detection method based on semi-automatic labeling and improved deep learning, characterized in that The steps include: (1) A human body segmentation algorithm based on thermal infrared is proposed, the segmentation results are mapped to visible light images, the visible light human body annotation box is quickly obtained, and a YOLOv5 training dataset is established; (2) A scene classification algorithm based on the improved ResNet 50 model is proposed to guide the human recognition method of the human recognition model in different scenarios; (3) Using the target detection algorithm of the proposed improved YOLOv5 model, the classification model is trained offline to detect foreign objects on the rails and realize foreign object detection; (4) A rail contour extraction algorithm based on traditional algorithms and improved deep learning is proposed to achieve rail segmentation and contour extraction in a single image; (5) Propose a foreign object alarm strategy algorithm based on the rail segmentation results and the location of the foreign object, guide the analysis of the alarm level of the entire image, and realize the alarm level judgment; The step (1) of quickly obtaining the visible light human body annotation frame is obtained in the following manner: by selecting a thermal infrared camera and a visible light camera with the same focal length, and recording thermal infrared and visible light videos at the same time in a scene with a temperature below 25 degrees Celsius; on this basis, taking advantage of the fact that the thermal infrared human body is easily segmented by the OTSU algorithm below 25 degrees Celsius, the OTSU algorithm is used to automatically complete the segmentation of the thermal infrared image, and then the segmentation result is mapped to the visible light image, thereby directly obtaining a large number of visible light human body annotation frame results; finally, a small amount of manual correction is performed on a small number of possible segmentation errors, and the visible light human body annotation frame is semi-automatically obtained to establish a YOLOv5 training data set; step (3) The target detection algorithm of the improved YOLOv5 model is as follows: for the YOLOv5 target detection model, after obtaining the data, Mosaic data enhancement and real-time data loading are used to improve the bounding box prediction loss function, and the model structure of the GT target bounding box UID allocator, GT target bounding box UID matcher, and bounding box position and classification loss weight algorithm is used to improve the position accuracy and classification accuracy; in terms of collecting combined target features, a convolutional attention mechanism is added; finally, in the prediction part, on the basis of the traditional sigmoid function σ(x), the h-swish function is introduced in MobileNetv3 to convert the original The loss function CIOU_Loss used by box is changed to GIOU_Loss; finally, the deepsort tracking module is added to the detection of the YOLOv5 algorithm to prevent missed detection, thereby improving the robustness of the system in detecting foreign objects; the traditional algorithm in step (4) refers to performing Hough transform processing on the pre-processed image and extracting the rail contour; the improved deep learning rail contour extraction algorithm in step (4) refers to using the on-board camera to obtain the rail image collected during the train running, and sending it to the DeepLabv3+ semantic segmentation network trained in advance by the system for track line semantic segmentation; then, the data results obtained by the two models are processed by contour union, leaving their common parts as the final extracted rail contour.

2. The method for detecting foreign bodies on rails by semi-automatic labeling and improved deep learning according to claim 1, characterized in that: The scene classification algorithm of the improved ResNet 50 model described in step (2) is as follows: for the ResNet50 scene classification model, a transfer learning mechanism is introduced to train the model, and the model weights pre-trained on ImageNet are migrated to the model to accelerate the convergence of the network model; secondly, in the selection of activation function, the Relu function is changed to the LRelu function to prevent the gradient from disappearing.

3. The method for detecting foreign objects on rails by semi-automatic labeling and improved deep learning according to claim 1, characterized in that: The foreign body alarm strategy algorithm of step (5) is as follows: using step (4) to obtain the corresponding coordinates of the track contour of a single image, using step (3) to obtain the coordinates of the foreign body in a single image, making a weighted judgment on the foreign body intrusion situation of the entire image based on the proposed custom function and algorithm, and finally normalizing it to the alarm level of the entire image.

Images