Multi-source remote sensing online detection system and method for foreign matter invasion along railway

Abstract

The invention discloses a multi-source remote sensing online detection system and method for foreign matter invasion along a railway, and belongs to the technical field of railway safety detection.Potential danger levels are predicted based on foreign matter features and train statuses, a danger level prediction module comprises a track prediction unit, a collision time calculation unit and a danger level division unit, the trajectory prediction unit predicts the motion trajectory of the target in the next three seconds by adopting Kalman filtering based on the target speed and acceleration output by the millimeter wave radar in combination with the target position detected by the LiDAR, and the collision time calculation unit obtains the real-time speed and position of the train, calculates the minimum collision time in combination with the target prediction trajectory and the railway limit, and calculates the collision time of the target in the next three seconds. And a danger level is divided by combining an intrusion state and a target type. Potential risks of foreign matters and trains can be accurately quantified through the danger degree prediction module, a core basis for graded response is provided for the subsequent alarm and control output module, and finally a safety protection closed loop of the mining area railway in the foggy environment is achieved.

Description

Technical Field

[0001] This invention belongs to the field of railway safety inspection technology, specifically relating to a multi-source remote sensing online detection system and method for foreign object intrusion along railway lines. Background Technology

[0002] Detection of foreign objects such as people, vehicles, and obstacles along railway lines is a crucial aspect of safe railway transportation in mining areas. The Huaibei mining area, located in the Huai River basin, has a relatively low-lying terrain, making it prone to moisture retention. Furthermore, mining activities generate large amounts of dust and particulate matter, leading to frequent foggy weather and low visibility in the Huaibei mining area. During foggy weather, the visibility of vehicles and pedestrians along the railway line is reduced, making it difficult for train drivers to assess information along the railway line, thus increasing the probability of accidents.

[0003] With the rapid development of visual target detection algorithms such as YOLO, railway encroachment target detection based on visual cameras has become an effective means. However, visual cameras suffer from low visibility, color distortion, and short detection range in foggy weather, failing to meet the requirements for long-distance real-time detection. In contrast, LiDAR and millimeter-wave radar are less affected by fog and have strong penetration capabilities. Addressing the real-time accuracy requirements for train target detection, LiDAR can provide high-precision 3D point cloud data. Combined with current deep learning algorithms such as PointNet and PointRCNN, it can meet the real-time accuracy requirements for target detection. In foggy environments, LiDAR signals are also affected by scattering and attenuation from water droplets, leading to incomplete point cloud data and increased noise. Existing methods can improve data quality through point cloud denoising methods such as statistical filtering and deep learning, and simulate fog data of different concentrations to enhance the generalization ability of algorithm models under foggy conditions. However, problems such as short detection range and high computational complexity still exist. Millimeter-wave radar, on the other hand, performs more stably in foggy environments. Utilizing high-frequency electromagnetic waves, millimeter-wave radar can penetrate fog, rain, and other adverse weather conditions, providing stable distance and speed information. However, millimeter-wave radar has low resolution, cannot provide detailed target shape and edge information, has low detection accuracy, and cannot quantify the potential risks between foreign objects and trains. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-source remote sensing online detection system and method for foreign object intrusion along railway lines, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multi-source remote sensing online detection system for foreign object intrusion along railway lines, comprising:

[0006] A multi-source sensor module is used to collect multi-modal data along the railway line. The multi-source sensor module includes LiDAR, millimeter-wave radar, visible light camera and infrared sensor. All sensors are integrated and installed on a standardized bracket at the front of the train.

[0007] The data preprocessing module is used to synchronize, filter, enhance, and label the collected multimodal data, and output a time- and space-aligned multimodal dataset.

[0008] The rail identification and clearance calibration module is used to dynamically identify the rail position and construct a dynamic clearance area according to the line type;

[0009] The multimodal fusion detection module is used to extract and fuse features from multiple sources to classify and locate foreign objects.

[0010] The hazard prediction module predicts the potential hazard level based on the characteristics of the foreign object and the train status. The hazard prediction module includes a trajectory prediction unit, a collision time calculation unit, and a hazard level classification unit. The trajectory prediction unit predicts the target's trajectory within the next 3 seconds based on the target speed and acceleration output by millimeter-wave radar and the target position detected by LiDAR, using Kalman filtering. The collision time calculation unit obtains the real-time speed and position of the train and calculates the minimum collision time between the target and the train by combining the predicted target trajectory with the railway clearance. The hazard level classification unit classifies the hazard level according to the intrusion status, minimum collision time, and target type.

[0011] The alarm and control output module is used to output response commands according to the hazard level and record detection data.

[0012] Preferably, the LiDAR uses a 128-line lidar with a detection range of ≥200 meters and a point cloud density of ≥200 points / m².

[0013] The millimeter-wave radar uses a 77GHz high-frequency millimeter-wave radar with a detection range of ≥300 meters and a speed measurement range of 0~100km / h.

[0014] The visible light camera is a high-definition industrial camera, equipped with a large aperture lens with an aperture of ≥F1.4;

[0015] The infrared sensor is an uncooled infrared thermal imager with a resolution of 640×512 and a detection distance of ≥150 meters.

[0016] Preferably, the data preprocessing module includes:

[0017] The time and space synchronization unit achieves time synchronization of each sensor through GPS timing, with a synchronization accuracy of ≤1ms. Based on preset sensor extrinsic parameters, it uniformly transforms the coordinates of millimeter-wave radar point cloud, infrared image, and visible light image to the LiDAR coordinate system.

[0018] The filtering and denoising unit uses a combination algorithm of statistical filtering and radius filtering for LiDAR point clouds, Kalman filtering for millimeter-wave radar data, a dehazing algorithm of dark channel prior and guided filtering for visible light images, and histogram equalization for infrared images.

[0019] The data augmentation and annotation unit performs data augmentation on the preprocessed point cloud and image, and uses a manual-assisted semi-automatic annotation tool to annotate the targets in the dataset.

[0020] Preferably, the rail identification and clearance calibration module includes:

[0021] The rail identification unit uses the intensity and shape features of LiDAR point clouds, employs the RANSAC algorithm to fit the straight line of the rail, and combines the track gauge standard to determine the center coordinates and extension direction of the two rails.

[0022] The dynamic clearance construction unit uses the center of the rail as a reference to construct a dynamic clearance region and maps the clearance region to the coordinate system of each sensor.

[0023] Preferably, the multimodal fusion detection module includes:

[0024] The multi-scale feature extraction unit performs multi-scale feature extraction on LiDAR point clouds, millimeter-wave radar point clouds, visible light images, and infrared images based on the feature pyramid network.

[0025] The cross-modal attention fusion unit introduces self-attention and cross-attention mechanisms to dynamically adjust the weights of features from each modality.

[0026] The SSD detection head unit employs an improved single-shot multi-frame detector, setting anchor boxes at different scale layers of the fused feature map to output target category, 3D target bounding box, and intrusion status.

[0027] Preferably, the alarm and control output module includes:

[0028] The graded response unit outputs corresponding alarm and control commands according to different hazard levels;

[0029] The data recording and uploading unit records the detection results and raw data from multiple sensors in real time, and uploads them to the mine railway monitoring center via a 4G / 5G module.

[0030] Preferably, the multi-source sensor module, data preprocessing module, rail identification and clearance calibration module, multi-modal fusion detection module, hazard prediction module, and alarm and control output module all communicate with the industrial control computer.

[0031] Preferably, a multi-source remote sensing online detection method for foreign object intrusion along railway lines is provided, and the method is as follows:

[0032] S1. First, multi-source sensor modules are integrated into a standardized bracket at the front of the train. External parameter calibration is completed using a checkerboard calibration board and a laser target. During normal data acquisition, data is collected synchronously at a set frequency. In foggy weather, the acquisition frequency of LiDAR and millimeter-wave radar is increased, the parameters of the camera and infrared sensor are adjusted, and the visibility value corresponding to each frame of data is recorded. Then, the multi-source data is interpolated and matched to achieve time alignment based on the GPS timestamp. The external parameter matrix is ​​called to map the data of each sensor to the LiDAR coordinate system to achieve spatial alignment, forming a spatiotemporally aligned data frame.

[0033] S2. Filter, denoise, enhance, and label the synchronized multi-source data;

[0034] S3. Extract a subset of points with an intensity value > 80 from the denoised LiDAR point cloud. Use the RANSAC algorithm to fit the straight line of the rail. Combine the 1435mm±5mm track gauge standard to determine the center line of the two rails. Verify and correct the position by comparing the train's GPS position with the CAD map of the line to ensure that the rail position accuracy is ≤ 0.05 meters. Then, based on the GPS positioning, determine whether the line is a straight segment or a curved segment. For straight segments, construct the clearance based on the midpoint of the rail center line. For curved segments, adjust the width of the outer clearance according to the radius R. Map the clearance to the coordinate system of each sensor.

[0035] S4. The LiDAR point cloud is converted into a BEV feature map, the millimeter-wave radar data is converted into a range and velocity feature map, and the visible light and infrared images are extracted into resolution feature maps through the ResNet50 network. Then, based on FPN, three-scale feature layers of small, medium and large scales are constructed and fused to form multi-scale feature sets of each sensor. A self-attention mechanism is applied to the multi-scale feature set of a single sensor, the pixel cosine similarity is calculated to generate a weight matrix and normalized to enhance the feature of the target region. Then, an improved SSD detection head is used, and anchor boxes of corresponding sizes are set in different scale layers of the fused feature map. The target category and 3D target box information are output through three parallel convolutional layers, and the intrusion status is judged by combining dynamic boundary.

[0036] S5. Static foreign objects use real-time 3D coordinates as a fixed trajectory. Dynamic foreign objects are predicted second-by-second for the next 3 seconds based on millimeter-wave radar velocity and acceleration data and LiDAR position using Kalman filtering. Then, the predicted trajectory of the foreign object is mapped to the same coordinate system as the train's trajectory, and the intersection of the trajectories is determined. When there is an intersection, the distance D between the train and the first intersection position and the train speed V are used as the basis for the determination. 列 and the velocity component V of the foreign object along the direction of train travel 物 According to the formula TTC=D / |V 列 -V 物The minimum collision time is calculated, the travel distance is corrected when the train brakes, and finally, the danger level is divided into four levels: high, medium, low and no risk, based on the intrusion status, minimum collision time and foreign object type. If the millimeter-wave radar speed detection error is >10% in foggy weather, the lower limit of TTC is reduced by 20%. When a static foreign object has intruded into the limit and TTC ≤8 seconds, the risk level is upgraded from medium risk to high risk.

[0037] S6: When there is a high risk, an audible and visual alarm will be triggered and a deceleration or stop command will be output. When there is a medium risk, an alarm will be triggered and target information will be pushed. When there is a low risk, a prompt will only be displayed on the vehicle screen. When there is no risk, there will be no response.

[0038] Preferably, in S4, the method further includes: applying cross-attention to LiDAR and millimeter-wave radar, visible light camera and infrared sensor respectively, adjusting the weights of millimeter-wave radar and infrared sensor to 0.6 to 0.7 in foggy weather, adjusting the weights of LiDAR and visible light camera to 0.3 to 0.4, and then refining the features through three 3×3 two-dimensional convolutions.

[0039] Preferably, in S5, the high-risk determination condition is that the boundary has been violated and TTC ≤ 5 seconds, or the boundary has not been violated but the distance from the boundary is ≤ 2 meters and TTC ≤ 8 seconds; the medium-risk condition is that the boundary has been violated and 5 seconds < TTC ≤ 10 seconds, or the boundary has not been violated but the distance from the boundary is ≤ 2 meters and 8 seconds < TTC ≤ 15 seconds; the low-risk condition is that the boundary has been violated and TTC > 10 seconds, or the boundary has not been violated but the distance from the boundary is ≤ 2 meters and TTC > 15 seconds; and the no-risk condition is that the boundary has not been violated and the distance from the boundary is > 2 meters or TTC is infinite.

[0040] Compared with the prior art, the beneficial effects of the present invention are:

[0041] 1. This invention can accurately quantify the potential risks of foreign objects and trains through a hazard prediction module, providing a core basis for graded response of subsequent alarm and control output modules, and ultimately realizing a closed-loop safety protection system for mining railways in foggy conditions.

[0042] 2. This invention utilizes multi-source sensor fusion technology, combining the advantages of LiDAR, millimeter-wave radar, visible light cameras, and infrared sensors, to solve the problem of insufficient detection capability of a single sensor in foggy conditions. It enables long-distance, multi-dimensional detection of foreign objects along railway lines in mining areas under foggy conditions, significantly improving the detection coverage and reliability.

[0043] 3. This invention, through a cross-modal attention fusion mechanism, can dynamically adjust the weights of features of each modality, fully utilize the complementarity between different sensor data, effectively handle the situation of foreign objects occluding each other and blending with the background, and significantly improve the accuracy and precision of foreign object detection.

[0044] 4. This invention, through dynamic rail identification and clearance zone construction technology, can adapt to changes in scenarios such as track settlement and curves in mining areas, solving the problem that traditional fixed clearance cannot accurately match the actual line conditions. It provides a more realistic benchmark for judging foreign object intrusion, reducing the occurrence of false alarms and missed alarms. Attached Figure Description

[0045] Figure 1 This is a system configuration diagram of the present invention;

[0046] Figure 2 This is a flowchart of the present invention;

[0047] Figure 3 This is the SSD detection head unit of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Please see Figures 1-3 This invention provides a multi-source remote sensing online detection system for foreign object intrusion along railway lines. The multi-source sensor module is used to collect multimodal data along the railway line. The multi-source sensor module includes a LiDAR, millimeter-wave radar, visible light camera, and infrared sensor. All sensors are integrated and installed on a standardized bracket at the front of the train. The LiDAR uses a 128-line lidar with a detection range ≥200 meters and a point cloud density ≥200 points / m². The millimeter-wave radar uses a 77GHz high-frequency millimeter-wave radar with a detection range ≥300 meters and a speed range of 0–100 km / h. The visible light camera uses a high-definition industrial camera equipped with a large aperture lens with an aperture ≥F1.4. The infrared sensor uses an uncooled infrared thermal imager with a resolution of 640×512 and a detection range ≥150 meters. The multi-source sensor module, data preprocessing module, rail identification and clearance calibration module, multimodal fusion detection module, hazard prediction module, and alarm and control output module all communicate with an industrial control computer.

[0050] Furthermore, the multi-source sensor module is used to collect multimodal data along the railway line. By leveraging the complementary advantages of multiple sensors, it addresses the issue of insufficient performance of a single sensor in foggy weather. All sensors are integrated and installed on standardized brackets at the front of the train and on poles around the railway. The bracket angle is adjustable to ensure that the field of view covers the railway clearance and a range of 30 meters on both sides. Specific parameters are as follows:

[0051] The LiDAR employs a 128-line lidar with a detection range ≥200 meters, a point cloud density ≥200 points / m², a horizontal field of view ≥120°, and a vertical field of view ≥25°. It is used to acquire high-precision 3D spatial coordinates and shape features of targets, providing core data for rail identification and target positioning. The millimeter-wave radar uses a 77GHz high-frequency millimeter-wave radar with a detection range ≥300 meters, a speed measurement range of 0–100 km / h, a horizontal field of view ≥90°, and a speed measurement error ≤0.2 m / s. It is used to penetrate water vapor in foggy weather to obtain target distance and speed information, compensating for the LiDAR signal attenuation problem. The visible light camera uses a 4K resolution high-definition industrial camera equipped with a large aperture lens ≥F1.4 and a frame rate ≥25fps. It is used to acquire target color and texture features to assist in distinguishing target categories. The infrared sensor uses an uncooled infrared thermal imager with a resolution of 640×512, a detection range ≥150 meters, and a temperature sensitivity ≤50 mK. It is used to identify targets with temperature differences from the background in foggy weather, compensating for the shortcomings of the visible light camera.

[0052] The data preprocessing module is used to synchronize, filter, enhance, and label the acquired multimodal data, outputting a time- and space-aligned multimodal dataset. The module includes: a time- and space synchronization unit, which synchronizes the time of each sensor via GPS timing with an accuracy ≤1ms, and transforms the coordinates of millimeter-wave radar point clouds, infrared images, and visible light images to the LiDAR coordinate system based on preset sensor extrinsic parameters; a filtering and denoising unit, which uses a combination of statistical filtering and radius filtering for LiDAR point clouds, Kalman filtering for millimeter-wave radar data, a dark channel prior and guided filtering dehazing algorithm for visible light images, and histogram equalization for infrared images; and a data enhancement and labeling unit, which enhances the preprocessed point clouds and images and uses a manually assisted semi-automatic labeling tool to label the targets in the dataset.

[0053] Furthermore, in the time and space synchronization unit, the principle is to utilize the high-precision time reference of GPS timing to ensure strict synchronization of each sensor in the time dimension, providing a unified time reference for subsequent multimodal data fusion. Coordinate transformation based on preset sensor extrinsic parameters is necessary because different sensors have different installation positions and angles. The extrinsic parameters describe the position and attitude relationship between each sensor's coordinate system and the LiDAR coordinate system. Through the coordinate transformation matrix, the coordinates of millimeter-wave radar point clouds, infrared images, and visible light images can be accurately mapped to the LiDAR coordinate system, achieving spatial alignment.

[0054] In the filtering and denoising unit, for LiDAR point clouds, statistical filtering identifies and removes outliers that deviate from the normal distribution by analyzing the statistical characteristics of the point cloud data, while radius filtering removes isolated noise points based on the spatial density of the point cloud. The combination of the two effectively purifies the LiDAR point cloud. For millimeter-wave radar data, Kalman filtering is used, leveraging the recursive estimation characteristics of Kalman filtering to predict and update target distance, velocity, and other data measured by millimeter-wave radar, reducing fluctuations caused by measurement noise. The dark channel prior dehazing algorithm for visible light images estimates the fog concentration and transmission rate based on the dark channel characteristics of foggy images, and then optimizes the transmission rate using guided filtering, preserving image edge details to achieve dehazing. Histogram equalization of infrared images adjusts the grayscale distribution of the image, increasing the grayscale difference between the target and the background, enhancing the contrast of the infrared image, and making the target easier to identify.

[0055] In the data augmentation and annotation unit, data augmentation works by performing a series of transformations on preprocessed data, such as rotation, scaling, flipping, and brightness adjustment, to simulate different acquisition scenarios and target shapes, increasing the diversity of the dataset and thus improving the generalization ability of subsequent models. The human-assisted semi-automatic annotation tool, based on manual annotation, combines algorithms to automatically generate some annotation suggestions, reducing the workload of manual annotation, improving annotation efficiency, and ensuring annotation accuracy, providing high-quality labeled datasets for model training.

[0056] The rail identification and clearance calibration module dynamically identifies rail positions and constructs dynamic clearance regions based on track type. The rail identification unit uses the intensity and shape features of LiDAR point clouds, employs the RANSAC algorithm to fit the rail straight lines, and determines the center coordinates and extension direction of the two rails in conjunction with track gauge standards. The dynamic clearance construction unit constructs dynamic clearance regions based on the rail center and maps these regions to the coordinate systems of each sensor.

[0057] Furthermore, in the rail identification unit, the differences in reflection intensity of different objects by LiDAR point clouds and the shape characteristics of the rails themselves are used to identify the rails. The process is as follows: First, points with high intensity features are extracted from the LiDAR point cloud. Then, based on the RANSAC algorithm, several points are randomly selected to fit a straight line. Through iterative calculation, the straight line model that best represents the direction of the rail is found, and noise points that do not conform to the model are eliminated. Finally, combined with the standard value of railway gauge, the center coordinates of the two rails and their extension directions are determined, thereby accurately locating the rail position.

[0058] In the dynamic clearance construction unit, the identified rail center is used as a reference. Based on the actual type of railway line, such as straight or curved sections, a dynamic clearance area that meets safety requirements is constructed. The process is as follows: First, the location information of the rail center is determined. Then, according to the different line types and corresponding clearance standards, a dynamic clearance area is constructed. Finally, through coordinate transformation, the constructed clearance area is mapped onto the coordinate systems of each sensor, such as millimeter-wave radar, visible light camera, and infrared sensor. This ensures that each sensor, when collecting data and detecting targets, can determine whether a target has encroached on the clearance area based on a unified clearance area that conforms to the actual line conditions.

[0059] The multimodal fusion detection module extracts and fuses features from multiple data sources to classify and locate foreign targets. This module includes: a multi-scale feature extraction unit that extracts features from LiDAR point clouds, millimeter-wave radar point clouds, visible light images, and infrared images at multiple scales based on a feature pyramid network; a cross-modal attention fusion unit that introduces self-attention and cross-attention mechanisms to dynamically adjust the feature weights of each modality; and an SSD detection head unit that uses an improved single-shot multi-frame detector to set anchor boxes at different scale layers of the fused feature map, outputting the target category, 3D target bounding box, and intrusion status.

[0060] Furthermore, the principle of the multi-scale feature extraction unit is that foreign objects of different sizes appear at different scales in images or point clouds. The feature pyramid network can extract features of the same modality data at multiple scales through bottom-up downsampling and top-down upsampling, combined with lateral connections, thereby adapting to the detection of foreign objects of different sizes. The process is as follows: LiDAR point clouds and millimeter-wave radar point clouds are converted into two-dimensional feature representations, which are then input into the feature pyramid network along with visible light images and infrared images. The network first performs convolution and downsampling on the input data from bottom to top, generating feature maps at different levels. Lower-level feature maps retain more details, while higher-level feature maps contain more semantic information. Then, the higher-level feature maps are upsampled from top to bottom and fused with the corresponding lower-level feature maps through lateral connections, ultimately generating a multi-scale feature map set for each modality of data.

[0061] The principle of the cross-modal attention fusion unit is that self-attention can uncover the correlation between features at different locations within a single modality, highlighting important regions, while cross-attention can establish correspondences between features of different modalities, achieving complementary advantages between modalities, thereby dynamically allocating the weights of features of each modality and improving the fusion effect. The process is as follows: First, for the multi-scale feature maps of each modality obtained by the multi-scale feature extraction unit, the self-attention mechanism is applied to calculate the attention weight of each feature location with other locations. After normalizing the weights, they are multiplied with the original feature map to enhance the target-related features within the single modality. Next, a cross-attention mechanism is introduced. For LiDAR and millimeter-wave radar features, the attention weights between them are calculated; similarly, for visible light and infrared image features, the attention weights between them are calculated. Then, based on these weights, the contribution of different modal features is dynamically adjusted. For example, in foggy weather, millimeter-wave radar and infrared images are less affected by fog, so their weights are increased. Finally, the modal features after weight adjustment are fused to obtain a cross-modal fused feature map.

[0062] The improved single-shot multi-frame detector (SSD) can simultaneously detect targets of different sizes by setting anchor boxes of different sizes and proportions on fused feature maps at different scales. Combining classification and regression branches, it achieves target category determination, 3D bounding box localization, and intrusion status determination. The process is as follows: The fused feature map obtained through cross-modal attention fusion is input into the improved SSD detector head. The detector head sets anchor boxes of corresponding sizes at different scale layers of the fused feature map; small-scale layers have smaller anchor boxes, and large-scale layers have larger anchor boxes. Then, the classification branch classifies each anchor box to determine its target type. The regression branch predicts the offset of the anchor box to the actual 3D target bounding box, obtaining the target's 3D position and size. Finally, combined with the boundary area generated by the rail recognition and boundary calibration module, it determines whether the target intrudes, and outputs the target category, 3D target bounding box, and intrusion status.

[0063] The hazard prediction module predicts the potential hazard level based on foreign object characteristics and train status. It includes a trajectory prediction unit, a collision time calculation unit, and a hazard level classification unit. The trajectory prediction unit uses the target velocity and acceleration output from millimeter-wave radar, combined with the target position detected by LiDAR, and employs Kalman filtering to predict the target's trajectory within the next 3 seconds. The collision time calculation unit acquires the train's real-time speed and position, and calculates the minimum collision time between the target and the train by combining the predicted target trajectory with the railway clearance. The hazard level classification unit classifies the hazard level based on the intrusion status, minimum collision time, and target type.

[0064] Furthermore, the hazard level prediction module uses a three-level logic—target trajectory prediction, collision risk calculation, and multi-dimensional level classification—combining foreign object characteristics and train status to predict potential hazard levels. The core process and technical details are as follows:

[0065] Before prediction, two key types of data need to be acquired to ensure the accuracy and real-time nature of the input information: core input foreign object characteristics and train status data. The basic attributes of the foreign object characteristics data include the object category, such as personnel, vehicles, static obstacles, 3D size, and real-time location. Dynamic attributes include speed, acceleration, degree of occlusion, and whether it has entered the dynamic clearance area constructed by the rail recognition and clearance calibration module. Train status data comes from the locomotive control system and GPS. Real-time operating data includes train speed, acceleration, and current location. Inherent parameters include train length, braking distance, and direction of travel.

[0066] Kalman Filter-Based Foreign Object Trajectory Prediction: A differentiated trajectory prediction strategy is adopted to address the differences in dynamic and static characteristics of foreign objects, resolving prediction biases caused by positional fluctuations and occlusion in foggy conditions. Static Foreign Object Handling: If the millimeter-wave radar detects a foreign object with a velocity of 0, such as a falling rock or a fallen tree, the real-time 3D coordinates output by the multi-modal fusion detection module are directly used as the fixed trajectory, requiring no additional prediction; only the relative distance between the object and the train needs to be updated in real time. Dynamic Foreign Object Handling: For moving foreign objects such as personnel and vehicles, a motion model is constructed based on the velocity and acceleration time-series data output by millimeter-wave radar and combined with the real-time position detected by LiDAR. The construction process is as follows: the position and velocity of the foreign object at time t are used as the initial state vector for filtering. The theoretical position and velocity at time t+1 are predicted according to Newton's laws of motion. At the same time, process noise is introduced to improve the robustness of the prediction, such as speed fluctuations caused by sudden changes in direction of personnel or vehicle braking. Then, the predicted value is corrected by combining the measured position of the multimodal fusion detection module at time t+1. Finally, the second-by-second predicted trajectory of the foreign object within the next 3 seconds is output, covering the critical time window of train braking response.

[0067] Collision risk calculation based on relative motion: This method calculates the minimum collision time by analyzing the relative motion relationship between the train and the foreign object, quantifying the time risk of both. The core calculation logic is as follows: First, the predicted trajectory of the foreign object every second is mapped to the same coordinate system as the train's trajectory. Then, it is determined whether there is a spatial intersection. If the trajectories do not intersect, it is directly determined that there is no possibility of collision, and the TTC is set to infinity. If the trajectories do intersect, the first intersection position is extracted, which is the position where the foreign object first enters the train's path. Then, based on the first intersection position and the real-time motion state of both, the minimum collision time TTC is calculated as TTC = D / |V. 列 -V 物 | where D is the straight-line distance between the foreign train and the initial intersection point, V 列 V is the train speed. 物This represents the velocity component of the foreign object along the direction of train travel; positive for the same direction and negative for the opposite direction. If the train has already applied the brakes, the braking deceleration must be included in the calculation to correct the actual distance traveled by the train during the TTC period, avoiding an overestimation of TTC due to failure to consider braking delay.

[0068] The multi-dimensional hazard classification combines three core dimensions—intrusion limit status, TTC (Total Traffic Concern), and foreign object category—to construct four hazard levels, avoiding "over-response" or "under-response" caused by a single-dimensional judgment. The specific classification criteria are as follows:

[0069]

[0070] To address the issue of decreased sensor data accuracy in foggy weather, two key optimizations have been added to the module:

[0071] TTC fault tolerance correction: If the millimeter-wave radar deviates in detecting the speed of foreign objects due to fog noise, the lower limit of TTC will be reduced by 20% to avoid risky misjudgments caused by underestimation of speed.

[0072] Static foreign object risk escalates: In foggy weather, the distance at which drivers can detect static obstacles is shortened. If a static foreign object has encroached on the limit and TTC ≤ 8 seconds, the risk level is directly upgraded from medium to high, allowing more braking reaction time.

[0073] The alarm and control output module is used to output response commands according to the hazard level and record detection data. The module includes: a graded response unit, which outputs corresponding alarm and control commands based on different hazard levels; and a data recording and uploading unit, which records detection results and raw data from multiple sensors in real time and uploads them to the mine railway monitoring center via a 4G / 5G module.

[0074] Furthermore, the graded response unit matches corresponding alarm and control strategies based on the different hazard levels output by the hazard prediction module to achieve accurate and non-excessive safety responses. The process is as follows: Upon receiving the hazard level from the hazard prediction module, if a high-risk condition is determined, an audible and visual alarm is immediately triggered, and an emergency deceleration or stop control command is simultaneously output to the train control system. If the risk is medium, an audible and visual alarm is triggered, and detailed information such as the location and type of the foreign object is pushed to the display screen in the train driver's cab to alert the driver. For low-risk conditions, only relevant information about the foreign object is displayed on the screen, without triggering an audible and visual alarm. If there is no risk, no alarm or control operation is performed.

[0075] The data recording and uploading unit utilizes storage and communication technologies to retain key data during the detection process in real time and transmit it to the monitoring center for subsequent analysis, backtracking, and system optimization. The process is as follows: It receives the detection results from the multimodal fusion detection module and raw data from multiple sensor sources in real time, organizes this data by timestamp, and stores it in the module's built-in high-speed storage medium. Simultaneously, via the 4G / 5G communication module, the data is packaged and uploaded to the server of the mine railway monitoring center at preset time intervals or upon the occurrence of an alarm event, for monitoring personnel to view and use for subsequent data analysis.

[0076] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-source remote sensing online detection system for foreign matter intrusion along a railway line, characterized in that, The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion.

2. The multi-source remote sensing on-line detection system for foreign matters intruding into the limit of a railway line according to claim 1, characterized in that, The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion.

3. The multi-source remote sensing on-line detection system for foreign matters intruding into the limit of a railway line according to claim 1, characterized in that, The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion.

4. The multi-source remote sensing on-line detection system for foreign matters intruding into the limit of a railway line according to claim 1, characterized in that, The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion.

5. The multi-source remote sensing on-line detection system and method for foreign matters intruding into the limit of a railway line according to claim 1, characterized in that, The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object detection system based on multi-source sensor fusion. The application relates to a railway foreign object The multi-scale feature extraction unit performs multi-scale feature extraction on the LiDAR point cloud, the millimeter wave radar point cloud, the visible light image and the infrared image respectively based on a feature pyramid network; The cross-modal attention fusion unit introduces a self-attention and cross-attention mechanism to dynamically adjust the feature weights of each modality. The SSD detection head unit adopts an improved single multi-frame detector, sets anchor boxes in different scale layers of the fused feature map, and outputs target categories, 3D target frames and intrusion states.

6. The multi-source remote sensing on-line detection system for foreign matters intruding into the limit of a railway line according to claim 1, characterized in that, The alarm and control output module includes: The hierarchical response unit outputs corresponding alarm and control instructions according to different danger levels; The data recording and uploading unit records the detection results and the original data of the multi-source sensors in real time, and uploads them to the mine railway monitoring center through a 4G / 5G module.

7. The multi-source remote sensing on-line detection system for foreign matters intruding into the limit of a railway line according to claim 1, characterized in that, The multi-source sensor module, the data preprocessing module, the rail identification and gauge calibration module, the multi-modal fusion detection module, the danger level prediction module and the alarm and control output module are all in communication with the industrial computer.

8. A multi-source remote sensing online detection method for foreign matter intrusion along a railway line, applicable to the multi-source remote sensing online detection system for foreign matter intrusion along a railway line according to any one of claims 1-7, characterized in that, The online detection method of foreign matter intrusion along the railway line by multi-source remote sensing is as follows: S1, first, the multi-source sensor module is integrated in the standardized bracket at the front end of the train, the external parameter calibration is completed by using the chessboard calibration plate and the laser target, the data is collected synchronously at the set frequency during normal collection, the collection frequency of LiDAR and millimeter wave radar is increased in foggy weather, the parameters of the camera and the infrared sensor are adjusted, and the visibility value corresponding to each frame of data is recorded, then the multi-source data is matched and interpolated based on the GPS timestamp to realize time alignment, the external parameter matrix is called to map the data of each sensor to the LiDAR coordinate system to realize spatial alignment, and a time and space aligned data frame is formed; S2, filtering, denoising, enhancing and labeling are performed on the synchronized multi-source data; S3, a point cloud subset with intensity value > 80 is extracted from the denoised LiDAR point cloud, the RANSAC algorithm is used to fit the rail straight line, the center lines of the two rails are determined according to the 1435mm±5mm gauge standard, the position accuracy of the rails is ensured to be ≤0.05 meters by verifying and correcting the train GPS position and the CAD map of the line, then the line is judged to be a straight line segment or a curve segment based on the GPS positioning, the limit is constructed based on the midpoint of the rail center line for the straight line segment, the width of the outer limit is adjusted according to the radius R for the curve segment, and the limit is mapped to the coordinate system of each sensor; S4, the LiDAR point cloud is converted into a BEV feature map, the millimeter wave radar data is converted into a distance and speed feature map, the visible light and infrared images are extracted into resolution feature maps through a ResNet50 network, then three scale feature layers of small, medium and large are constructed based on FPN and fused to form multi-scale feature sets of each sensor, the self-attention mechanism is applied to the multi-scale feature set of a single sensor, a weight matrix is generated by calculating the pixel cosine similarity and is normalized, the target region features are enhanced, then an improved SSD detection head is used to set anchor boxes of corresponding sizes in different scale layers of the fused feature map, three parallel convolution layers are used to output target categories and 3D target frame information, and the intrusion state is judged in combination with the dynamic limit. S5, static foreign objects take real-time 3D coordinates as fixed trajectories, dynamic foreign objects predict future 3-second trajectories per second based on millimeter wave radar speed and acceleration data and LiDAR position, then map the foreign object prediction trajectory and the train travel trajectory to the same coordinate system, judge the trajectory intersection, when there is intersection, based on the distance D between the train and the first intersection position, the train speed V 列 and the speed component V 物 of the foreign object along the train travel direction, calculate the minimum collision time according to the formula TTC = D / |V 列 -V 物 , correct the travel distance when the train brakes, finally combine the intrusion state, the minimum collision time and the foreign object category to divide the risk level into four grades: high, medium, low and no risk, if the millimeter wave radar speed detection error is >10% in fog, the lower limit of TTC is compressed by 20%, and if the static foreign object has intruded and TTC≤8 seconds, the risk is upgraded from medium to high. S6, trigger sound and light alarm and output deceleration or stop command when high risk, trigger alarm and push target information when medium risk, only prompt on vehicle display screen when low risk, and no response when no risk.

9. The method according to claim 8, wherein, In S4, further comprising: applying cross attention to LiDAR and millimeter wave radar, visible light camera and infrared sensor respectively, adjusting the weight of millimeter wave radar and infrared sensor to 0.6-0.7 in foggy weather, adjusting the weight of LiDAR and visible light camera to 0.3-0.4, and then purifying the features through 3 times of 3*3 two-dimensional convolution.

10. The method according to claim 8, characterized in that, In S5, the high risk judgment condition is that the limit has been invaded and TTC≤5 seconds, or the limit has not been invaded but the distance to the limit is ≤2 meters and TTC≤8 seconds, the medium risk is that the limit has been invaded and 5 seconds

Images