Rail foreign matter identification method based on monocular thermal infrared camera

Abstract

The present application relates to a kind of monocular thermal infrared camera-based rail foreign object identification method.It includes: on the basis of traditional semi-automatic labeling algorithm, utilize a variety of different network models to carry out semi-automatic labeling, obtain with more accurate semi-automatic image contour information;In semantic segmentation network, thermal image feature map and the feature map based on thermal image gray processing are fused, and weight factor standardization module operation is added, obtain foreign object segmentation map;Propose a kind of improved cross-modal fusion backbone neural network, by transformer, the cross-modal feature points and modal information of segmentation map and thermal infrared map are fused, network parameters are compressed using distillation learning method, realize the light foreign object identification;Propose a kind of monitoring alarm device, for monitoring and alarming rail foreign object intrusion;The present application improves the efficiency and accuracy of rail foreign object identification under various environments, can effectively reduce the possibility of train accident caused by foreign object intrusion rail.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and pattern recognition, and intelligent transportation systems (ITS), specifically to a method for identifying foreign objects on railway tracks based on a monocular thermal infrared camera. Background Technology

[0002] Rail transit refers to a type of transportation vehicle or system that requires vehicles to run on specific tracks. In recent years, my country's rail transit construction and operation have flourished, making significant contributions to alleviating urban passenger congestion and promoting the development of rail-based urban economic belts. However, during normal operation of urban rail transit, incidents such as foreign objects intruding into the track boundaries and colliding with trains occur frequently. Due to the suddenness, uncertainty, and high risk of foreign object intrusion incidents, a collision between a train and a foreign object can lead to train service interruption, endangering operational safety, damaging vehicle equipment, and causing injuries or fatalities, seriously threatening the safe operation of trains. Therefore, timely detection of foreign object intrusion during train operation, providing early warnings and alarms before the train's arrival, and taking timely and effective braking measures to avoid collisions are prerequisites for ensuring the safe operation of rail transit trains and are currently a highly valued and concerned issue for my country's rail transit operation and management departments.

[0003] In 2021, the total number of railway passengers reached 2.357 billion, the total number of railway freight transported reached 3.812 billion tons, and the total converted turnover reached 3,913.494 billion ton-kilometers. Railways remain an important means of transportation for the people and goods in my country.

[0004] Traditional methods for identifying foreign objects require the installation of multiple cameras along the railway tracks. These cameras lack analytical capabilities, necessitate manual inspection, are costly, and carry a high risk of missing objects. Therefore, a new technical solution is needed.

[0005] In early railway foreign object intrusion detection algorithms, Zhang Xudong (Zhang Xudong. Design of Railway Foreign Object Intrusion Detection System Based on Intelligent Camera [D] Beijing Jiaotong University, 2017) proposed a railway foreign object intrusion detection system based on intelligent camera. This system designed a foreign object intrusion detection algorithm based on intelligent camera for railway lines. By distributing detection nodes along the railway line, it achieved the functions of distributed foreign object image acquisition, processing, analysis, automatic alarm, and foreign object image storage. Wang Qun (Wang Qun. Research on Track Intrusion Recognition Technology Based on Binocular Stereo Vision [D]. Beijing Jiaotong University, 2018) proposed applying binocular stereo vision technology to track foreign object intrusion detection, solving the problems of false detection of suspended objects in the air and inability to locate intruders by monocular camera video monitoring. Simultaneously, it used the 3D reconstruction results to estimate the volume of the intruder.

[0006] In the field of deep learning-based techniques, Wang Lei (Wang Lei. Research and System Design of High-Precision Pedestrian Detection Algorithm for Railway Traffic Safety [D]. Beijing Jiaotong University, 2019.) proposed a small-scale pedestrian detection algorithm. This algorithm optimizes the network structure based on the Faster R-CNN network by improving the weak semantic segmentation loss function, channel radial transformation, and feature fusion. Wang Ning (Wang Ning. Railway Intrusion Pedestrian Classification Algorithm Based on Feature Fusion [D]. Beijing Jiaotong University, 2019.) proposed a railway pedestrian classification algorithm based on feature fusion, namely, the fusion of convolutional neural network features and manual pedestrian features. He Wenyu et al. from Jiangxi University of Science and Technology (He Wenyu, Yang Jie, Zhang Tianlu. Track foreign object intrusion detection algorithm based on deep learning [J]. Computer Engineering and Design, 2020, 41(12):8.) proposed a method that combines traditional machine learning algorithms to perform basic image processing, delineate the intrusion area, and use the YOLO-V3 deep learning network model for multiple trainings. The trained model detects the delineated intrusion area and detects whether there are foreign objects in the area. This method makes it possible to detect foreign object intrusion in real time. However, this requires the installation of multiple camera devices along the line, which is costly and lacks mobility, and does not give full play to the real-time advantage of the YOLOv3 algorithm. Zhu Mengrui, Niu Hongxia, et al. from Lanzhou Jiaotong University (Zhu Mengrui, Niu Hongxia. An improved YOLOv3 model algorithm for railway foreign object intrusion identification [D]. Journal of Beijing Jiaotong University, 2019.) proposed an improved YOLOv3 model algorithm for foreign object intrusion identification. This algorithm includes using inverse perspective transformation to determine the track position and divide the dangerous area, using K-means++ clustering algorithm to select the prior box size suitable for the dataset, and using MobileNetv2 feature extraction network to reduce the number of model parameters. This algorithm can identify foreign objects in dangerous areas and significantly improve the speed of foreign object identification. However, it requires a large number of transfer learning samples, the training dataset is not easy to obtain, and the accuracy of identification cannot be guaranteed against interference from external factors.

[0007] In summary, although current deep learning-based methods for foreign object detection have shown initial success in other fields, there is still room for improvement in terms of detection speed, accuracy, and efficiency in the specific field of intelligent transportation systems for railways. Summary of the Invention

[0008] The purpose of this invention is to provide a method for identifying foreign objects on railway tracks based on a monocular thermal infrared camera. This method aims to address the problems of existing methods for identifying foreign objects on railway tracks, such as poor real-time performance, susceptibility to interference from factors like light, and low accuracy in identifying foreign objects. This will effectively improve the safety factor of train track operation.

[0009] A method for identifying foreign objects on railway tracks based on a monocular thermal infrared camera, comprising the following steps: Step 1: A semi-automatic image contour information annotation method is proposed. For the initial thermal image, after the foreign object is initially located, different semantic segmentation models are used to segment it according to different foreign object target categories. The original image set is automatically annotated according to the segmentation results. After manual review and proofreading, accurate semi-automatic annotated image contour information is obtained and used to train the improved semantic segmentation network proposed in this patent. Step 2: An improved semantic segmentation network is proposed to segment the original thermal image. First, the thermal image is processed by grayscale to obtain a coarse segmentation map. The obtained thermal image feature map and the coarse segmentation map based on the grayscale processing of the thermal image are fused. At the same time, bilateral filtering is used to denoise and normalize the image to obtain pixel-normalized temperature information. Then, the temperature information of the thermal image is added to the calculation weight of the improved semantic segmentation network to complete the image segmentation and obtain the foreign object segmentation image, which is used as the input of the improved cross-modal fusion backbone neural network. Step 3: Propose an improved cross-modal fusion backbone neural network to achieve lightweight foreign object recognition on railway tracks. The image obtained by segmentation based on the improved semantic segmentation network and the original thermal infrared image are input into the improved multimodal fusion backbone network. At the same time, cross-modal feature points and modal information are fused by transformer. The neural network is simplified by using an improved distillation learning method to achieve lightweight foreign object recognition.

[0010] Furthermore, the accurate semi-automatic labeled image contour information obtained in step (1) is obtained through the following steps: First, YOLOv5 is used for preliminary localization of foreign objects on the railway tracks. Images captured by a monocular thermal infrared camera are input into a pre-trained YOLOv5 model to obtain precise object locations. The image is then cropped based on the object's bounding box. Next, training and recognition are performed on different target labels. Based on these results, semi-automatic annotation is conducted to obtain the final annotation results. The labels refer to human bodies, railway tracks, stones, and others. Different models are used for training and recognition based on these labels: a vision-based Human Action Recognition (HAR) network for human body recognition, a convolutional neural network (Faster R-CNN) for railway track recognition, and a deep neural network (DNN) algorithm for stones and other foreign objects. Then, the obtained foreign object images are input again into the corresponding models for feature extraction and model optimization to improve accuracy. Multiple copies of the images to be labeled are created, and image segmentation is performed on each copy. Finally, a mask is used to apply the image to the model. The R-CNN network obtains a binary matrix of the object's contour. Different models, as described above, are used to test each copy to obtain multiple target labels. Finally, all target labels are merged into a complete label. After automatically obtaining images with each label, manual checks are performed to check for omissions or labeling errors, and these are corrected. After completion, the labeled image is saved as an XML file to obtain accurate semi-automatic labeled image contour information.

[0011] Furthermore, the segmentation of the foreign object image using an improved semantic segmentation network described in step (2) is obtained through the following steps: First, the thermal image is processed into grayscale to effectively capture contextual information based on the grayscale image and balance the utilization of local and global features. Then, it is converted into a feature map format. A probabilistic graphical model is then used to process the image, resulting in a rough segmentation plot based on grayscale processing (RSPbGP). Simultaneously, a VGG-16 deep convolutional neural network is employed to fuse the thermal image feature map and the feature map based on grayscale processing, yielding a rough segmentation plot based on the VGG-16 network. The improved semantic segmentation network (RSPbCNN) is used to process RSPbGP and RSPbCNN by introducing boundary information through a cross-layer method. The features of RSPbGP and RSPbCNN are then fused to obtain a coarse segmentation map. Simultaneously, bilateral filtering is used to denoise and normalize the thermal image, and the Canny operator is used to extract sub-pixel edges to obtain pixel-normalized temperature information. The temperature information from the thermal image is then added to the computational weights of the improved semantic segmentation network. The kernel size is adjusted based on the temperature information to gradually extract global features and contextual information of the image, thereby increasing computational speed and reducing memory usage. Subsequently, the weight factors obtained from edge detection are introduced into the improved semantic segmentation network model. The feature map and the weights of each pixel are fed into the improved ReNet network to gradually extract global features and contextual information of the image. Finally, an upsampling layer composed of deconvolutional networks is used to gradually restore the resolution of the feature map and output the segmentation result, obtaining the segmentation map Segment_Thermal, which is the foreign object segmentation image.

[0012] Furthermore, the improved cross-modal fusion-based backbone neural network described in step (3) is as follows: A dual-stream parallel feature extraction network and a feature fusion module based on an asymmetric attention mechanism are presented. The feature extraction network adopts the UNet network. The feature fusion module uses a transformer to fuse cross-modal feature points and modal information, including calculating segmentation map values, calculating thermal infrared channel map values, calculating the cross-attention between the segmentation map (Segment_Thermal) and the thermal infrared channel map (Thermal), fusing the cross-attention, inputting it into the transformer's encoder to obtain a one-dimensional vector, concatenating them into a tensor of the same size, and inputting it into the feature pyramid of the YOLOv5 network. Furthermore, an improved distillation learning method is used to compress network parameters and simplify the neural network. Simultaneously, the improved cross-modal fusion backbone neural network is used as a student model, and YOLOv5 is used as a teacher model. Foreign object detection tasks are performed using the student model and the teacher model respectively, calculating the loss function, simplifying the student model, and finally obtaining the improved cross-modal fusion backbone neural network.

[0013] Furthermore, according to steps (1), (2), and (3), a monitoring and alarm device is proposed, comprising: a monocular thermal infrared camera, a processor, a memory, a display, and an alarm; the monocular thermal infrared camera is fixed by a base inside the driver's cab or outside the railway vehicle, and is used to acquire thermal infrared video streams; the processor is used to process the video streams and execute the foreign object detection model obtained in step (3); the memory is used to store all computer programs, video stream processing results, and logs; the display is used to display the foreign object identification results and potential risk levels on the railway tracks; and the alarm is used to emit alarm sounds to deter potential live foreign objects from intruding into the railway tracks and to remind the railway train driver.

[0014] Compared with existing railway foreign object identification technologies, the present invention has the following advantages and effects: The method described in step (1) of the present invention can semi-automatically annotate image contour information in the data preprocessing stage, and further improve the image contour annotation accuracy during the later manual review and proofreading, ultimately greatly reducing the manual annotation cost, improving annotation efficiency and accuracy, and enhancing training quality; The improved semantic segmentation network described in step (2) can combine the segmentation advantages of thermal image feature maps and feature maps based on thermal image grayscale processing and fuse them, and then add the temperature information carried by the thermal image to the calculation weights of the improved semantic segmentation network to complete image segmentation and obtain foreign object segmentation images, ultimately improving the segmentation accuracy of this type of model. This provides an innovative method to solve the problem of low segmentation accuracy in this type of model; the backbone neural network based on the improved cross-modal fusion described in step (3) can complete lightweight foreign object recognition. This patent proposes an innovative method to fuse cross-modal feature points and modal information of segmentation map and thermal infrared map through transformer, and uses the improved distillation learning method to compress network parameters and simplify the neural network. Finally, it solves the problems of slow recognition speed, low recognition accuracy, poor real-time performance, low recognition accuracy in complex environments, and weak robustness of traditional railway foreign object recognition algorithms. Compared with the existing technology, the method of this patent makes the accuracy of railway foreign object recognition by monocular thermal infrared camera more reliable.

[0015] Another advantage of this invention is that the system's hardware is based on vehicle-mounted cameras. Compared with the traditional method of detecting foreign objects on the track by installing static cameras in sections along the track to monitor the track 24 hours a day, this method is more economical and has a higher fault tolerance rate. It greatly improves the efficiency of identifying foreign object intrusions on the track and reduces the possibility of train accidents caused by foreign object intrusions on the track. Attached Figure Description

[0016] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of a method for identifying foreign objects on railway tracks based on a monocular thermal infrared camera, provided by an embodiment of the present invention. Figure 2 This is a flowchart of a semi-automatic image contour information annotation method provided in an embodiment of the present invention; Figure 3 This is a flowchart of the improved semantic segmentation network provided in the embodiments of the present invention; Figure 4 This is a flowchart of the improved cross-modal fusion backbone neural network provided in the embodiments of the present invention; Detailed Implementation

[0018] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following detailed description, in conjunction with the accompanying drawings and preferred embodiments, describes a method for identifying foreign objects on railway tracks based on a monocular thermal infrared camera, including its specific implementation, structure, features, and effects.

[0019] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0020] The following describes in detail, with reference to the accompanying drawings, a specific scheme of the present invention for identifying foreign objects on railway tracks based on a monocular thermal infrared camera.

[0021] See Figure 1 This invention provides a method for identifying foreign objects on railway tracks based on a monocular thermal infrared camera, the method comprising: S1. A proposed semi-automatic image contour information annotation method is used to reduce the cost of manual annotation. For the initial thermal image, YOLOv5 is used for preliminary localization of foreign objects. Then, different models are used for semantic segmentation for different target categories. The original image set is automatically annotated based on the segmentation results. After manual review and proofreading, accurate semi-automatic annotated image contour information is obtained. The obtained image contour information will be used to train the improved semantic segmentation network proposed in this patent. S2. Using the proposed improved semantic segmentation network, the original thermal image is semantically segmented more efficiently, resulting in a semantic segmentation map obtained from the improved semantic segmentation network. This map serves as the input to the improved cross-modal fusion backbone neural network. The thermal image is processed to obtain a coarse segmentation map. Simultaneously, bilateral filtering is applied to the image for denoising and pixel normalization to obtain pixel temperature normalization information. The feature map of the thermal image and the feature map based on the grayscale processing of the thermal image are fused. The temperature information from the thermal image is added to the computational weights of the improved semantic segmentation network, completing the segmentation of the original image and obtaining a foreign object segmentation map. This map serves as the input to the improved cross-modal fusion backbone neural network. S3. Real-time foreign object detection on railway tracks is achieved using the proposed backbone neural network based on improved cross-modal fusion, realizing lightweight foreign object detection. The segmentation map obtained based on the improved semantic segmentation network and the original thermal infrared image are input into the improved multimodal fusion backbone network. Cross-modal feature points and modal information are fused through a transformer, and the neural network is simplified using an improved distillation learning method to achieve lightweight foreign object recognition. In practical applications of this embodiment, the method further includes the following steps before S1: S0. Under different brightness scenarios, select thermal infrared cameras with the same focal length to acquire corresponding train track images and videos in the field, take a portion of them for manual annotation, and use them as part of the final dataset.

[0022] In the first step of YOLOv5 training, a training dataset in COCO format is used to train a pre-built instance segmentation model for recognizing pixel information in the orbital region, and the trained instance segmentation model is obtained.

[0023] Simultaneously, the COCO dataset and manually collected images were pre-selected into a single dataset, each containing only labels for human bodies, railway tracks, stones, and other types of objects. These were then used to train the following three network models: human body recognition labels were used to train a vision-based Human Action Recognition (HAR) network; railway track recognition images were used to train a convolutional neural network (Faster R-CNN); and images labeled with stones and other foreign objects were used to train a deep neural network (DNN) algorithm. Finally, the parameters of the corresponding trained models were saved.

[0024] In the specific implementation of this embodiment, combined with Figure 2 S1 specifically includes: S11. The image to be labeled is copied into multiple copies. The pre-trained YOLOv5 model is used to perform preliminary localization of foreign objects in all copies, and this is used to obtain instance segmentation images, which are then used as inputs for different network models.

[0025] S12. Obtain the parameters of the trained network models, including three main types: Human Action Recognition (HAR), Faster R-CNN, and Deep Neural Network (DNN). Input all the obtained image segmentation results into the network models to obtain the location information of the corresponding labels obtained by each network. Filter out the desired labels from each copy. For example, human image information is filtered out from the HAR copy, railway track image information is filtered out from the Faster R-CNN copy, and the last model needs to exclude images of stones and other foreign objects from the copied images. Finally, merge all images with target labels to obtain a single image with complete labels.

[0026] S13. Manually review and proofread the image with all annotations, and supplement and modify any missing, incorrect, or blurry annotations to obtain the final annotated image.

[0027] In the specific implementation of this embodiment, combined with Figure 3 S2 specifically includes: S21. In order to more effectively capture contextual information based on grayscale images and balance the utilization rate of local and global features, this embodiment uses a weighted average method to perform grayscale processing on the captured thermal images. After grayscale processing, the resulting grayscale images are converted into feature map format. Subsequently, a probabilistic graphical model (PGM) is used to process the image to more effectively capture contextual information based on grayscale images and balance the utilization rate of local and global features. After completing the above process, this embodiment obtains a rough segmentation plot based on grayscale processing (RSPbGP).

[0028] S22. Simultaneously with S21, this embodiment performs denoising and pixel normalization operations on the image using bilateral filtering, and extracts pixel edges from the thermal image using the Canny operator, with bilateral filtering used for denoising. Subsequently, pixel normalization is performed to extract pixel temperature information.

[0029] The normalization operation is characterized by the pre-established normalization formula as follows: (1) Where i,j represent the position of the calculated pixel, k,l represent the positions of the surrounding pixels, g(i,j) is the pixel value obtained after convolution, and f(k,l) are the surrounding pixel values. Here, w(i,j,k,l) ​​is the most affected. The pixel temperature information is obtained through this normalization operation.

[0030] S23. Based on S22, the denoised thermal image is fed into a deep convolutional neural network. The deep convolutional neural network adopts the VGG-16 network and adds an attention mechanism in the last convolutional layer to obtain a rough segmentation plot based on deep convolutional neural network (RSPbCNN).

[0031] S24. Process RSPbGP and RSPbDL. After introducing boundary information using a cross-layer method, fuse the RSPbGP features and RSPbDL features. In this embodiment, the fusion method is the average value method, that is, the pixel information is added together and the average value is taken to obtain the fused coarse segmentation map.

[0032] S25. The temperature information from the thermal image, i.e., the single-pixel normalized weight factor obtained in S22, is added to the calculation weights of the improved semantic segmentation network to complete the segmentation of the original image, resulting in a foreign object segmentation map. This embodiment modifies the convolution kernel size based on temperature information to progressively extract global features and contextual information of the image, while simultaneously increasing computational speed and reducing memory usage. The improved semantic segmentation network model incorporates the weight factors obtained from edge detection in S22, sending the feature map and pixel weights into the decoder. The decoder is an improved ReNet network, progressively extracting global features and contextual information of the image. The improved ReNet network algorithm adds pixel weight factors to the area of ​​action of the convolution process, giving the convolution kernel a variable shape. Finally, an upsampling layer composed of a deconvolution network is used to progressively restore the resolution of the feature map, outputting the segmentation result, resulting in the segmentation map Segment_Thermal.

[0033] In the specific implementation of this embodiment, combined with Figure 4S3 specifically includes: S31. Building upon S25, to fuse the features of the segmentation image and the original thermal infrared channel image to achieve the goal of identifying foreign objects on railway tracks even at night, this example uses an asymmetric cross-attention method to fuse the two modalities of images. Specifically, after downsampling the images of both modalities using UNet, the segmentation image calculates the query value, and the original thermal infrared channel image calculates the index key value and content value. These three values ​​are then input into the Multi Head Attention module to calculate cross-attention, and then into the MultiLayer Perceptron module to obtain a one-dimensional vector. Finally, this vector is reshaped into a three-channel tensor of the same size as the original feature map, and this process is repeated three times to obtain three pairs of tensors of different sizes. These three pairs of tensors each represent semantic information from shallow to deep, containing information from both the segmentation image and the original thermal infrared channel image, and also serving as input to the subsequent student model network.

[0034] The formulas for calculating the query, index key, and content value are equations (2) to (4):

[0035]

[0036]

[0037] in, and These are the segmentation diagram and the original thermal infrared channel diagram, respectively. , and The transformation matrix; , and These are the query, index key, and content value for the corresponding image.

[0038] The formula for calculating asymmetric cross attention is Equation (5):

[0039] in, It represents the depth of the query tensor.

[0040] S32. To obtain a lightweight, easily deployable network model without sacrificing accuracy, this example employs Knowledge Distillation (KD). Specifically, a pre-trained YOLOv5 model is used as the teacher model, the tensor obtained from the multimodal fusion is used as input, and the output of the last fully connected layer of the teacher model is used as the soft label. This allows the student model to learn the teacher model's output in the foreign object detection task. Simultaneously, the ground truth labels of the original images are used as hard labels, and the lightweight student model is trained using a combination of these two types of labels.

[0041] Among them, the loss function guiding the knowledge distillation model adopts the cross-entropy loss function, and the calculation formula is Equation (6)~(8):

[0042]

[0043]

[0044] in, and This is a scaling factor, set externally; in this example, we choose 0.8 and 0.2. and Let T and T be the probabilities of the teacher model and student model respectively when predicting that the model belongs to the i-th class at temperature T. The higher T is, the smoother the output probability distribution becomes, the greater the entropy of the distribution, and the information carried by the negative label will be relatively amplified. The model training will pay more attention to the negative label. The softmax function is used for calculation. and These are the logarithms of the teacher model and the student model, respectively. For the true label value in the i-th class, the value is [value]. Positive labels are assigned a value of 1, and negative labels are assigned a value of 0.

[0045] S33. To ensure the generalization performance of the network, it is necessary to screen multiple lightweight networks that have undergone knowledge distillation. This example uses ten-fold cross-validation to screen networks with high recall and precision. Specifically, the Receiver Operating Characteristic Curve (ROC) is plotted and the Area Under Curve (AUC) is calculated. The network with the largest AUC is selected for deployment, resulting in a neural network model capable of quickly identifying foreign objects on railway tracks offline.

[0046] The neural network model obtained in step S3 of this embodiment is deployed in a monitoring and alarm device. The monitoring and alarm device includes a monocular thermal infrared camera, a processor, a memory, a display, and an alarm. The monocular thermal infrared camera is fixed to the cab or outside a railway vehicle by a base and is used to acquire thermal infrared video streams. Optionally, an external protective cover is used to protect the monocular thermal infrared camera. The processor is used to process the video stream and execute the foreign object detection model obtained in step S3. The memory is used to store all computer programs, video stream processing results, and logs, and can be any type of volatile or non-volatile storage device or a combination thereof. The device can be implemented using a combination of various methods, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), or read-only memory (ROM). The display is used to show the results of foreign object identification on the rails and the potential risk level. Optionally, the identification results can be a foreign object detection frame, foreign object type, or their composition, while the risk level is set by hyperparameters. The alarm is used to emit an alarm sound to deter potential live foreign objects from intruding into the rails and to alert the train driver. It can be implemented by any controllable sound-emitting device, light-emitting device, or a combination thereof.

[0047] This embodiment presents a semi-automatic image contour annotation method. It semi-automatically annotates image contour information during the data preprocessing stage and further improves the accuracy of image contour annotation during subsequent manual review and proofreading. Therefore, it can reduce manual annotation costs, improve annotation efficiency and accuracy, and enhance training quality. The proposed improved semantic segmentation network in this embodiment combines the segmentation advantages of thermal image feature maps and feature maps based on thermal image grayscale processing, and fuses them. The temperature information from the thermal image is then added to the computational weights of the improved semantic segmentation network to complete image segmentation and obtain the foreign object segmentation image. This ultimately improves the segmentation accuracy of such models and provides an innovative method to solve the problem of low segmentation accuracy in these models. The proposed improved cross-modal fusion backbone neural network in this embodiment proposes an innovative method to fuse cross-modal feature points and modal information from the segmentation image and thermal infrared image using a transformer. An improved distillation learning method is used to compress network parameters and simplify the neural network. Compared with existing technologies, this solves the problems of slow recognition speed, low recognition accuracy, poor real-time performance, low recognition accuracy in complex environments, and weak robustness of traditional railway foreign object recognition algorithms. The combination of all methods in the embodiments of this application can effectively improve the sensitivity and accuracy of foreign object identification and detection on railway tracks based on a monocular thermal infrared camera, and improve the efficiency of object detection.

[0048] Accordingly, this application also provides a monitoring and alarm device, which is used to emit alarm sounds to deter potential live foreign objects from intruding into the railway tracks and to remind railway train drivers.

[0049] It should be noted that this invention is described with reference to flowchart illustrations and / or block diagrams of methods and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0050] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0051] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

Claims

1. A method for identifying foreign objects on railway tracks based on a monocular thermal infrared camera, characterized in that... Includes the following steps: Step 1: For the initial thermal images, use YOLOv5 to perform preliminary localization of foreign objects. Then, use different models for semantic segmentation for different target categories. The original image set is automatically labeled based on the segmentation results. After manual review and proofreading, accurate semi-automatic labeled image contour information is obtained. The obtained image contour information will be used to train the improved semantic segmentation network. Step 2: Use the improved semantic segmentation network to perform semantic segmentation on the original thermal image, process the thermal image into grayscale to obtain a coarse segmentation map, and simultaneously perform denoising and pixel normalization operations on the image to obtain pixel temperature normalization information. Fuse the feature map of the thermal image and the feature map based on the grayscale processing of the thermal image, and add the temperature information of the thermal image to the calculation weights of the improved semantic segmentation network to complete the segmentation of the original image and obtain the foreign object segmentation map, which is used as the input of the improved cross-modal fusion backbone neural network. Step 3: Input the segmentation map obtained from the improved semantic segmentation network and the original thermal infrared image into the improved cross-modal fusion backbone neural network. The cross-modal feature points and modal information are fused through the transformer, and the neural network is simplified using the improved distillation learning method to achieve lightweight foreign object recognition. The improved cross-modal fusion backbone neural network described in step three is characterized by comprising two parts: a dual-stream parallel feature extraction network and a feature fusion module based on an asymmetric attention mechanism; wherein, the feature extraction network adopts the UNet network; the feature fusion module uses a transformer to fuse cross-modal feature points and modal information, and includes the following steps: (1) Calculate the segmentation graph Segment_Thermal( The query value of ) is calculated using the formula shown in (2); , (2) Calculate the thermal infrared channel map. The index Key and content Value of ) are calculated using the formulas shown in (3) and (4); , , (3) Calculate the cross attention between the segmentation map Segment_Thermal and the thermal infrared channel map Thermal. The calculation formula is shown in (5). , in, and These are the segmentation diagram and the original thermal infrared channel diagram, respectively. , and The transformation matrix; , and These are the query, index key, and content value for the corresponding image; It is the depth of the query tensor; the cross attention is fused and input into the transformer's encoder to obtain a one-dimensional vector, which is then concatenated into a tensor of the same size, i.e., input into the feature pyramid of the YOLOv5 network.

2. The method for identifying foreign objects on railway tracks based on a monocular thermal infrared camera according to claim 1, characterized in that, Initial localization of foreign objects is achieved using YOLOv5, followed by semantic segmentation using different models for different target categories. This includes: inputting images captured by a monocular thermal infrared camera into a pre-trained YOLOv5 model to obtain precise foreign object locations, and cropping the image based on the foreign object recognition bounding box to obtain the cropped image; training and recognition based on different target labels, followed by further semi-automatic labeling based on these results to obtain the final labeled results; where the labels refer to human body, railway track, and stone, and different models are trained based on these labels for recognition.

3. The method for identifying foreign objects on railway tracks based on a monocular thermal infrared camera according to claim 1, characterized in that, The obtained foreign object images are input into different improved models for recognition, resulting in images with target label location information. The specific steps include: inputting the obtained foreign object images into the corresponding models again for feature extraction and model optimization to make each model more accurate; then copying the images to be labeled to form multiple copies; performing image segmentation on each image copy; using the Mask R-CNN network to obtain the binary matrix of the foreign object contour; finally, testing each copy using the different models described above to obtain multiple target labels; finally, merging all target labels into the final complete label; and finally, after automatically obtaining images with each label, manually checking for omissions or labeling errors and correcting them; and finally saving the labeled images as XML files.

4. The method for identifying foreign objects on railway tracks based on a monocular thermal infrared camera according to claim 1, characterized in that, The steps for obtaining a coarse segmentation map through grayscale processing include: performing grayscale processing on the thermal image and converting it into a feature map format; then using a post-probabilistic graphical model to process the image more effectively, capturing contextual information based on the grayscale image, and balancing the utilization of local and global features to obtain a coarse segmentation map based on grayscale processing.

5. The method for identifying foreign objects on railway tracks based on a monocular thermal infrared camera according to claim 1, characterized in that, The method for obtaining pixel temperature normalization information by performing denoising and normalization operations on the image using bilateral filtering is characterized by: simultaneously performing sub-pixel edge extraction on the thermal image using the Canny operator in step one, and performing denoising operations using bilateral filtering; the normalization operation is characterized by its normalization formula as shown in (1). （1） Where i,j represent the position of the calculated pixel, k,l represent the positions of the surrounding pixels, g(i,j) is the pixel value obtained after convolution, and f(k,l) are the surrounding pixel values. Here, w(i,j,k,l) ​​is the most affected. The pixel temperature information is obtained through this normalization operation.

6. The method for identifying foreign objects on railway tracks based on a monocular thermal infrared camera according to claim 1, characterized in that, The fusion of thermal image feature maps and feature maps based on thermal image grayscale processing includes: feeding the thermal image into a deep convolutional neural network (DCNN) using a VGG-16 network, with an attention mechanism added to the last convolutional layer to obtain a coarse segmentation map based on the DCNN; processing the coarse segmentation map based on grayscale processing and the coarse segmentation map based on the deep neural network; introducing boundary information using a cross-layer method; and fusing the features of the coarse segmentation map based on grayscale processing and the coarse segmentation map based on the deep neural network to obtain a fused coarse segmentation map.

7. The method for identifying foreign objects on railway tracks based on a monocular thermal infrared camera according to claim 1, characterized in that, The process of incorporating temperature information from thermal images into an improved semantic segmentation network for weight calculation includes: adjusting the convolution kernel size based on temperature information to progressively extract global features and contextual information of the image, thereby increasing computational speed and reducing memory usage. The improved semantic segmentation network model introduces weight factors obtained from edge detection in step two, and feeds the feature map and pixel weights into a decoder. The decoder is an improved ReNet network, which progressively extracts global features and contextual information of the image. The improved ReNet network algorithm adds pixel weight factors to the convolutional processing area, giving the convolution kernel a variable shape. Finally, an upsampling layer composed of a deconvolutional network is used to progressively restore the resolution of the feature map, outputting the segmentation result, resulting in the segmentation map Segment_Thermal.

8. The method for identifying foreign objects on railway tracks based on a monocular thermal infrared camera according to claim 1, characterized in that, The simplification of the neural network using the improved distillation learning method includes: to compress network parameters, using the improved cross-modal fusion backbone neural network described in step three as the student model and YOLOv5 as the teacher model, performing foreign object detection tasks using the student model and the teacher model respectively, calculating the loss function, and simplifying the student model. The loss function includes the following: (1) Calculate the Teacher model at a temperature equal to T Under the condition of softmax output in the first i Values ​​on the class, where T The higher the value, the smoother the output probability distribution becomes, the greater the entropy of its distribution, and the more the information carried by the negative label will be amplified. The model training will pay more attention to the negative label. The calculation formula is shown in (6). , (2) Calculate the softmax output of the Student model in the i-th class under the condition that the temperature is equal to T. The calculation formula is shown in (7). , (3) Calculate the Loss function of the knowledge distillation model. The calculation formula is shown in (8). , Among them, among them, and These are the proportionality coefficients, set to 0.8 and 0.2 respectively; and Let T and T be the probabilities of the teacher model and student model respectively when predicting that the model belongs to the i-th class at temperature T. The higher T is, the smoother the output probability distribution becomes, the greater the entropy of the distribution, and the information carried by the negative label will be relatively amplified. The model training will pay more attention to the negative label. The softmax function is used for calculation. and These are the logarithms of the teacher model and the student model, respectively. For the true label value in the i-th class, the value is [value]. Positive labels are assigned a value of 1, and negative labels are assigned a value of 0.

Images