A high-speed train foreign matter intruding recognition method based on multi-modal data fusion

Abstract

The application provides a high-speed train foreign matter intrusion identification method based on multi-modal data fusion, comprising: synchronously acquiring multiple heterogeneous data through a multi-source data acquisition framework, and optimizing the data; adopting an adaptive alignment technology to ensure the consistency of the multiple heterogeneous data in time and space dimensions through rigid alignment and flexible alignment; fusing the multiple heterogeneous data through a dynamic weighting mechanism, adjusting the contribution proportion of each data source according to real-time environmental conditions, and calibrating the data features in combination with an attention mechanism; based on the comprehensive feature representation after fusion, utilizing an identification process to realize the classification, positioning and risk level judgment of potential threat targets, evaluating the targets in combination with path constraint information, and outputting a graded early warning result; and through multi-task joint optimization and resource adaptation technology, efficient reasoning is realized in a resource-limited computing environment, and the early warning result is fed back to a control system.

Description

Technical Field

[0001] This invention relates to the field of early warning and identification technology, and in particular to a method for identifying foreign object intrusions into high-speed trains based on multimodal data fusion. Background Technology

[0002] High-speed train operation safety is a core issue in the railway transportation sector, its importance self-evident, directly impacting passenger safety and property security as well as transportation efficiency. As train speeds continue to increase, so do the potential risks in the operating environment. Especially under complex conditions, timely detection and identification of foreign objects on the tracks has become a crucial link in ensuring train safety. Research in this field is not only about technological innovation but also a vital support for public safety and social stability.

[0003] However, current technologies often fall short when dealing with complex environments. Many methods experience a significant decrease in perception capabilities in special scenarios such as nighttime, heavy fog, or high-speed movement, resulting in insufficient accuracy in identifying foreign objects. A deeper problem lies in the fact that the integration of information from different sources often lacks an effective coordination mechanism, preventing it from leveraging its combined advantages and thus affecting the reliability of the overall judgment. This limitation makes trains susceptible to safety hazards under extreme conditions.

[0004] Further analysis reveals that the core of this problem lies in the importance of dynamically adjusting different sensing methods according to environmental changes. Simply put, in different scenarios, certain sensing methods may be more suitable for capturing key information, while others may be interfered with or perform poorly. For example, in a nighttime environment, ordinary images may struggle to clearly show small objects on a track, while thermal imaging technology can detect anomalies through temperature differences; however, if the ambient temperature is close to the object's temperature, the effectiveness of thermal imaging will be significantly reduced. In such cases, if the priority of various sensing methods cannot be flexibly adjusted according to the actual situation, it is difficult to ensure the accuracy of the recognition results. This lack of dynamic adjustment directly leads to insufficient adaptability in foreign object recognition in changing environments.

[0005] Therefore, how to flexibly adjust the emphasis of different sensing methods according to specific scenario conditions in a complex and ever-changing operating environment to ensure the accuracy and stability of foreign object identification has become a key issue that urgently needs to be addressed in the field of high-speed train safety operation. Summary of the Invention

[0006] This invention provides a method for identifying foreign object intrusions on high-speed trains based on multimodal data fusion, comprising the following steps: S1. Construct an on-board multimodal sensing array to simultaneously acquire visible light images, infrared thermal imaging, lidar point clouds, and track boundary mask data. Perform differentiated preprocessing on each modal data to obtain preprocessed multimodal data. S2. A deep calibration mechanism combining time-stamp-based rigid alignment and target feature-based flexible alignment is used to perform spatiotemporal alignment on the preprocessed multimodal data; a multi-branch lightweight feature extraction architecture is constructed to extract features from each spatiotemporally aligned modal data to obtain preliminary features for each modality; S3. Construct a dynamic gating fusion unit, calculate the dynamic weight coefficients of each modality with real-time environmental features as input, calibrate the preliminary features of each modality by combining channel attention mechanism and spatial attention mechanism, and fuse the calibrated modal features by an adaptive stitching layer according to the dynamic weight coefficients to obtain a deep fusion feature map. S4. Based on the deep fusion feature map, a lightweight detection head structure is used to identify foreign objects in the track and surrounding area, generating target category and pixel-level location information; the target location information and track boundary mask are intersected in real time to determine whether the target has intruded into the track safety limit and the degree of intrusion; the target category, three-dimensional volume, movement speed and intrusion depth are combined to output the graded warning result through the graded warning decision model.

[0007] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: This invention discloses a method for foreign object intrusion identification on high-speed trains based on multimodal data fusion. It addresses the unique challenges of poor robustness of single-modal perception, insufficient multimodal fusion, and the inability to meet the real-time requirements of onboard edge computing in the high-speed train operating environment, proposing a complete solution. The problem manifests in complex operating conditions, such as nighttime, heavy fog, or high-speed motion, where a single data source struggles to accurately identify foreign objects, while multi-source data fusion faces challenges in spatiotemporal alignment and computational efficiency. This invention constructs a multimodal perception array, combining visible light images, infrared thermal imaging, LiDAR point clouds, and track boundary masks to achieve differentiated preprocessing and spatiotemporal alignment of data sources. A dynamic weighted deep feature fusion mechanism adaptively adjusts the weights of each modality according to environmental conditions to ensure identification accuracy. Furthermore, through lightweight model optimization and real-time inference deployment, processing latency is controlled to the millisecond level, meeting onboard computing requirements. Ultimately, this invention significantly improves the robustness and real-time performance of foreign object identification on high-speed trains in complex environments, providing a reliable guarantee for safe train operation. Attached Figure Description

[0008] Figure 1 This is a flowchart of a method for identifying foreign object intrusion on high-speed trains based on multimodal data fusion, according to the present invention.

[0009] Figure 2 This is a flowchart of step S5 in an embodiment of the present invention.

[0010] Figure 3This is a flowchart of step S5 in an embodiment of the present invention.

[0011] Figure 4 This is a logic diagram for foreign object intrusion identification and graded early warning in an embodiment of the present invention. Detailed Implementation

[0012] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] like Figures 1 to 3 This embodiment of a method for identifying foreign object intrusion on high-speed trains based on multimodal data fusion may specifically include: This invention provides a method for identifying foreign object intrusions on high-speed trains based on multimodal data fusion, aiming to solve the problems of poor robustness of single-modal perception, insufficient multimodal fusion, and difficulty in meeting the real-time requirements of onboard edge computing in existing technologies. The technical solution of this invention will be described in detail below with reference to specific embodiments to make the objectives, technical solutions, and advantages of this invention clearer.

[0014] Step S1: Construct an on-board multimodal sensing array, simultaneously acquire visible light images, infrared thermal imaging, lidar point clouds, and track boundary mask data, perform differentiated preprocessing on each modal data, and obtain preprocessed multimodal data.

[0015] In this embodiment, to meet the safety monitoring needs of high-speed train operating environments, an onboard multimodal sensing array is constructed to simultaneously acquire various heterogeneous data, including visible light images, infrared thermal imaging, lidar point clouds, and track boundary masks. The selection of these data sources is based on the characteristics of high-speed train operating scenarios. For example, visible light images provide rich texture details, infrared thermal imaging is suitable for capturing target thermal features at night or in low visibility conditions, lidar point clouds provide accurate three-dimensional spatial information, and track boundary masks are used to define the boundaries of safe areas. This multimodal combination ensures reliable sensing data acquisition under various complex operating conditions. The data acquisition equipment is typically installed at the front or side of the train to ensure a comprehensive view covering the track and surrounding area.

[0016] After data acquisition, differentiated preprocessing is performed based on the characteristics of each data source to improve data quality and lay the foundation for subsequent processing. For visible light images, Gaussian filtering is first used to remove noise interference, followed by adaptive histogram equalization to enhance image contrast, making the edge features of tracks and potential foreign objects more prominent. To address motion blur caused by high-speed train operation, the pixel displacement within the exposure period is estimated based on the real-time speed information reported by the train control system. This determines the direction and length parameters of the motion blur kernel. Deconvolution is then used to restore the blurred image and recover the texture details of the target. For example, when a train is traveling at 300 km / h, visible light images may become blurred due to rapid movement; the above method can effectively restore the clear outlines of track edges and small foreign objects.

[0017] For infrared thermal imaging data, considering its imaging advantages in adverse weather conditions such as heavy fog or heavy rain, an adaptive temperature threshold segmentation technique is employed to distinguish foreground targets from the background. The specific segmentation process involves dynamically adjusting the threshold range based on the ambient temperature distribution to extract target areas with significant thermal characteristics. Subsequently, morphological dilation operations are used to further enhance the target contour, reducing incomplete imaging caused by weather interference. For example, at night or in foggy weather, infrared data can capture the thermal signals of people near the track; the above processing can clearly delineate the target shape, avoiding missed detections due to low visibility.

[0018] For lidar point cloud data, due to its large data volume and significant noise content, voxel mesh downsampling technology is first used to reduce data redundancy and decrease the computational burden. Then, statistical filtering methods are employed to remove ground noise and isolated noise points, ensuring the purity of the point cloud data. Next, density-based clustering is used to separate the target from the background, obtaining the 3D spatial pose information of the object. For example, when rocks fall near a track, clustering can distinguish the rock point cloud from the ground point cloud, accurately determining the spatial location and size of the rock.

[0019] For track boundary mask data, a bidirectional mapping technique between the world coordinate system and the image coordinate system is used to map it to the actual track environment, marking high-risk areas such as curve sections and switch areas, thus improving the accuracy of subsequent boundary violation determination. Simultaneously, to ensure consistency of multi-source data in the temporal dimension, high-precision timestamp synchronization technology is employed to align various types of data at the acquisition end, avoiding data misalignment issues caused by acquisition delays. These preprocessing steps lay a solid foundation for subsequent multimodal data alignment and fusion.

[0020] In one embodiment, the preprocessing of visible light images can be adaptively adjusted according to train speed and lighting conditions. For example, in a tunnel exit scene with high-speed train operation and low lighting, the image may be simultaneously affected by motion blur and low contrast. To address this, firstly, random noise is removed using Gaussian filtering, and then an adaptive histogram equalization method is used to dynamically adjust the contrast based on the gray-level distribution of local image regions, making track edges and foreign object features more prominent. Next, to address the motion blur problem, blur kernel parameters are estimated based on the real-time train speed, and image sharpness is restored through deconvolution. It should be noted that this method can maintain image quality across different speed ranges; for example, it can effectively correct blur at speeds of 200 km / h and 350 km / h, ensuring the accuracy of subsequent feature extraction.

[0021] The key to infrared data preprocessing lies in the accuracy and anti-interference capability of target segmentation. One possible approach is to dynamically adjust the adaptive temperature threshold based on ambient temperature and weather conditions. For example, in low-temperature winter environments, the thermal characteristics of people near the track are more pronounced, and the threshold can be appropriately lowered to capture weak thermal signals; while in high-temperature summer environments, there is more background thermal noise, and the threshold needs to be appropriately increased to reduce false positives. After segmentation, morphological dilation is used to expand the target region, filling in contour breaks caused by imaging interference. This processing method is particularly effective in heavy rain or fog, ensuring the integrity of the target contour and providing reliable input for subsequent feature extraction.

[0022] In one embodiment, the preprocessing of the LiDAR point cloud can be optimized based on the complexity of the track scene. For example, in scenarios with abundant vegetation or debris along the track, the point cloud data may contain a large number of non-target points. By downsampling using voxel grids, the point cloud data is divided into fixed-size grid cells, retaining only representative points within each grid, thereby significantly reducing the data volume. Subsequently, statistical filtering methods are used to remove isolated noise points based on the local density distribution of the point cloud, such as removing points with excessively large average distances from their neighborhoods. Next, a density-based clustering algorithm is used to divide the point cloud into multiple clusters, each cluster representing a potential target, such as a rockfall on the track or an intruding vehicle. In this way, the target can be accurately separated from the background, providing high-quality data for subsequent spatial feature extraction.

[0023] The processing of track boundary masks aims to provide a spatial reference for subsequent intrusion determination. In one possible implementation, the mask data is mapped to the actual track environment through a two-way mapping between the world coordinate system and the image coordinate system. For example, on straight track sections, the mask directly marks the safety boundaries on both sides of the track; while on curved sections or switch areas, the mask is dynamically adjusted according to track geometry parameters to mark narrower safety areas or special danger areas. This mapping process, combined with real-time train position information, ensures a high degree of consistency between the mask and the actual scene. Simultaneously, high-precision timestamp synchronization technology aligns the mask data with other modal data in time, avoiding determination errors caused by data misalignment.

[0024] Step S2: A deep calibration mechanism combining time-stamp-based rigid alignment and target feature-based flexible alignment is used to perform spatiotemporal alignment on the preprocessed multimodal data; a multi-branch lightweight feature extraction architecture is constructed to extract features from each spatiotemporally aligned modal data to obtain preliminary features for each modality.

[0025] After preprocessing the multimodal data, spatiotemporal alignment of various heterogeneous data is required to ensure the accuracy of subsequent fusion processes. A deep calibration mechanism combining rigid alignment based on timestamps and flexible alignment based on target features is employed. First, rigid spatiotemporal benchmark alignment between sensors is achieved through high-precision timestamp synchronization technology, ensuring basic consistency in acquisition time for various data types. Subsequently, flexible compensation is performed through target-level feature association technology, such as establishing correspondences between different modalities based on the target's geometry or thermal features, further improving alignment accuracy. This method is particularly effective in high-speed dynamic scenarios, significantly reducing acquisition delay errors and sensor installation deviations caused by high-speed train movement.

[0026] Building upon spatiotemporal alignment, a multi-branch lightweight feature extraction architecture is constructed to extract preliminary features for different modalities. For visible light images and infrared thermal imaging data, a lightweight deep network structure is employed to efficiently extract fine-grained semantic features, such as texture information of track edges or thermal distribution features of targets. For LiDAR point cloud data, a specially designed point cloud processing network learns the global geometric structural features of targets, such as the 3D shape and spatial location of foreign objects. Simultaneously, for track boundary mask data, a fully connected network is used to explicitly model spatial boundary constraint features, transforming the geometric information of the safe area into a computable feature vector. This multi-branch architecture provides accurate and complementary feature inputs for subsequent multimodal fusion detection and localization.

[0027] In this embodiment of the application, the environmental feature vector is assumed to be... ,in Represents the normalized value of light intensity. This represents the weather label code value (e.g., fog is coded as 1, rain is coded as 2, and sunny is coded as 0). For the environmental feature dimension, the gated weighted network uses a lightweight convolutional structure and calculates the weight coefficients of each modality through forward propagation. Its output formula is as follows:

[0028] in, These correspond to the weighting coefficients for the visible light, infrared, and lidar modes, respectively. The Sigmoid activation function is used to normalize the weight values ​​to the [0,1] interval. Represents lightweight convolution operations. As a bias vector, it can quickly respond to changes in environmental parameters and automatically increase the weight gain of infrared and lidar modes under harsh conditions (such as nighttime or heavy fog), weakening the influence of visible light modes that are severely affected by environmental interference, and ensuring the stability of fusion.

[0029] Simultaneously, a dual mechanism of channel attention and spatial attention is introduced, assuming that the feature maps of each modality are respectively... (Visible light) (Infrared) (LiDAR), the channel attention mechanism generates channel weights through a squeeze-excitation operation. The formula is:

[0030] Spatial attention mechanisms generate spatial weights through global pooling and convolution operations. The formula is:

[0031] The modal features are weighted and calibrated using dual attention weights, i.e. ( This represents element-wise product, effectively removing background noise and redundant information, and focusing on enhancing the feature representation of weak and intrusive foreign objects. Finally, an adaptive splicing layer adaptively adjusts the fusion ratio of each modality feature. The fusion formula is:

[0032] Generate a deep fusion feature map that incorporates texture, temperature, spatial, and boundary information. This provides high-quality feature support for subsequent foreign object detection tasks.

[0033] In one embodiment, the rigid alignment process is achieved through high-precision timestamp synchronization technology. For example, during train operation, each sensor records the data acquisition time with millisecond-level precision, ensuring the consistency of visible light images, infrared thermal images, and lidar point clouds in the temporal dimension. The specific alignment process involves sorting and matching data frames based on timestamps, for example, grouping data frames with a time difference of less than 1 millisecond into the same group. Subsequently, the lidar point cloud is projected onto the image pixel coordinate system using the geometric parameters of the sensor installation location, eliminating spatial deviations caused by installation eccentricity. This method effectively suppresses acquisition delay errors in high-speed operating scenarios, providing a reliable benchmark for subsequent flexible alignment.

[0034] Flexible alignment aims to further improve alignment accuracy in complex dynamic scenes. One possible implementation involves target-level feature association. For example, edge contour features of the target are extracted from visible light images, thermal distribution features from infrared data, and 3D geometric features from LiDAR point clouds. Then, feature matching techniques are used to establish target correspondences across different modalities. The specific matching process can be based on the shape similarity or spatial consistency of the targets; for example, targets with similar shapes and close spatial locations are classified as the same object. This approach compensates for subtle deviations not addressed in rigid alignment, ensuring high consistency of multimodal data at both the pixel and spatial levels.

[0035] In one embodiment, the multi-branch feature extraction architecture is optimized for the characteristics of different modalities. For visible light images, a lightweight convolutional network is used to extract high-resolution texture features, such as the color features of cracks or foreign objects on the track surface. For infrared thermal imaging data, a similarly lightweight network is used, but the focus is on extracting the thermal distribution features of the target, such as the thermal signals of personnel or machinery. For LiDAR point cloud data, a point cloud processing network is used to extract three-dimensional geometric features, such as the size and spatial distribution of foreign objects. For track boundary mask data, a fully connected network is used to transform the geometric boundaries of the safe area into feature vectors, for example, encoding the safe distances on both sides of the track into computable constraints. Through the above multi-branch processing, it is ensured that the various modal features are both independent and complementary, laying the foundation for subsequent fusion.

[0036] Step S3: Construct a dynamic gating fusion unit, calculate the dynamic weight coefficients of each modality using real-time environmental features as input, calibrate the preliminary features of each modality by combining channel attention mechanism and spatial attention mechanism, and fuse the calibrated modal features through an adaptive stitching layer according to the dynamic weight coefficients to obtain a deep fusion feature map.

[0037] After completing the spatiotemporal alignment and preliminary feature extraction of multimodal data, deep fusion of various features is required to improve the accuracy and robustness of foreign object recognition. A dynamic gating fusion unit is designed, which takes environmental features as input and calculates the dynamic weight coefficients of each modality in real time through a lightweight network. Environmental features include information such as light intensity and weather conditions. For example, in nighttime or foggy weather, the weights of infrared thermal imaging and LiDAR point clouds are automatically increased, reducing the impact of heavily interfered visible light images and ensuring the stability of the fusion process.

[0038] Simultaneously, a dual mechanism of channel attention and spatial attention is introduced to finely calibrate the features of each modality. The channel attention mechanism generates channel weights by analyzing the global distribution of the feature map, focusing on strengthening feature channels that contribute to foreign object identification, such as thermal feature channels in infrared data. The spatial attention mechanism generates spatial weights through global pooling operations, highlighting the spatial location of target regions in the feature map, such as the spatial distribution of foreign objects near the track. Through this dual attention mechanism, background noise and redundant information are effectively eliminated, enhancing the feature representation capability for small or flexible foreign objects.

[0039] Finally, an adaptive stitching layer is used to fuse the features of each modality according to dynamic weights, generating a deep fusion feature map that incorporates texture, temperature, spatial, and boundary information. This fusion feature map can combine the advantages of multimodal data. For example, in nighttime scenes, it combines infrared thermal features and LiDAR spatial features to ensure accurate identification of people near tracks; in strong light and backlight scenes, it combines visible light texture features and LiDAR geometric features to avoid missed detections due to lighting interference. Through the above fusion process, high-quality feature support is provided for subsequent foreign object detection tasks.

[0040] In one embodiment, the dynamic gating fusion unit takes environmental features as input and calculates the weight coefficients of each modality in real time. For example, light intensity and weather conditions are encoded into feature vectors, where light intensity is measured by sensors and normalized, and weather conditions are converted into numerical values ​​using a preset encoding table, such as 0 for sunny days, 1 for foggy days, and 2 for rainy days. Subsequently, a lightweight convolutional network processes the feature vectors and outputs the weight values ​​of each modality. For example, in a nighttime foggy scene where the light intensity is below 0.2 and the weather code is 1, the infrared modality weight is increased to 0.5, the lidar weight is increased to 0.4, and the visible light weight is decreased to 0.1. This dynamic adjustment ensures that the fusion process adapts to different operating conditions and improves recognition robustness.

[0041] In one possible implementation, channel attention and spatial attention mechanisms are combined to perform weighted calibration of features across different modalities. For example, the channel attention mechanism calculates the statistical information of each channel in the feature map using global average pooling, and then generates channel weights through a fully connected network to strengthen feature channels that contribute to foreign object recognition, such as automatically enhancing the response intensity of thermally relevant channels in infrared data in nighttime scenes. The spatial attention mechanism generates a spatial weight map through global pooling and convolution operations, highlighting the spatial response of targets within the orbital region and suppressing interference from distant background regions. All these weights are dynamically calculated by the network based on the input feature map, without the need for manually preset fixed thresholds. This dual calibration significantly reduces background noise interference and enhances the feature representation of small foreign objects.

[0042] In one embodiment, the adaptive stitching layer fuses the features of each modality according to dynamic weights. For example, in nighttime scenes, infrared modal features have a higher weight, and their thermal distribution information is retained first during fusion; LiDAR modal features have a lower weight, retaining their spatial geometric information; visible light modal features have the lowest weight and are only used as auxiliary information to supplement texture details. The specific fusion process involves weighted summation of the feature maps of each modality to generate the final deep fused feature map. This feature map can integrate the advantages of multiple modalities under different operating conditions. For example, in heavy rain, the fused feature map can combine infrared thermal features and LiDAR spatial features to ensure accurate capture of flexible floating objects near the track.

[0043] This embodiment describes in detail the first half of the steps in the high-speed train foreign object intrusion identification method based on multimodal data fusion, such as... Figure 4 As shown, this includes multimodal data source selection and preprocessing, spatiotemporal alignment and initial feature extraction, and dynamic weighted deep feature fusion. Subsequent steps will be elaborated in the next description, focusing on foreign object intrusion identification, risk level determination, and model optimization and deployment.

[0044] Step S4: Based on the deep fusion feature map, a lightweight detection head structure is used to identify foreign objects in the track and surrounding area, generating target category and pixel-level location information; the target location information and the track boundary mask are intersected in real time to determine whether the target has intruded into the track safety limit and the degree of intrusion; the target category, three-dimensional volume, movement speed and intrusion depth are combined to output the graded warning result through the graded warning decision model.

[0045] After deep fusion and feature extraction of multimodal data, foreign object identification and intrusion determination are performed based on the fused feature map to achieve accurate classification and location of foreign objects along the track. A lightweight detection head structure is used to process the fused feature map, identifying various foreign objects in and around the track, such as fallen rocks, intruders, and construction machinery, and generating pixel-level location information of the targets. The detection process not only focuses on the category of the target but also includes the target's confidence score to ensure the reliability of the identification results.

[0046] After target detection, the generated target bounding box is intersected with a pre-built track boundary mask in real time to quickly determine whether the target has intruded into the track safety clearance and the degree of intrusion. The track boundary mask, as a geometric constraint on the safety area, precisely defines the safety boundaries on both sides and above the track. For example, the safety boundary width is 2.5 meters for straight sections and 2.0 meters for curved sections based on curvature. The intersection operation determines the degree of intrusion by calculating the area ratio of the overlapping region between the target bounding box and the mask. For example, an overlap ratio exceeding 0.8 is considered severe intrusion, between 0.2 and 0.8 is considered slight intrusion, and below 0.2 is considered no intrusion. Specific thresholds can be configured according to the safety protection level of different track segments.

[0047] Based on this, a hierarchical early warning decision-making model is constructed by integrating multi-dimensional indicators such as target category attributes, 3D volume estimation, real-time movement speed, and intrusion depth. The early warning levels are divided into three categories: emergency warning, routine warning, and safety alert. Emergency warnings correspond to foreign objects that may immediately threaten train safety, such as large falling rocks in the center of the track; routine warnings correspond to foreign objects that pose a potential risk but do not immediately affect safety, such as small gravel at the track edge; and safety alerts correspond to low-risk targets, such as personnel activity far from the track. The final output includes the target's category, confidence level, pixel coordinates, and spatial location information, providing a reliable basis for subsequent decision-making and handling.

[0048] In one embodiment, the lightweight detection head structure optimizes the fused feature map to achieve accurate classification and localization of over a hundred types of foreign objects. For example, foreign objects that may appear along the track include rocks larger than 0.3 meters in diameter, intruders taller than 1.5 meters, and construction machinery wider than 1.0 meter. The detection head extracts feature information of targets of different sizes by performing multi-scale analysis on the fused feature map, such as the local texture features of small gravel and the overall geometric features of large machinery. Subsequently, through classification and regression operations, it generates category labels and pixel-level bounding boxes for the targets, ensuring accurate differentiation between targets and noise even in complex backgrounds.

[0049] In one possible implementation, the intersection calculation of the track boundary mask and the target bounding box is dynamically adjusted based on the track type and train speed. For example, the safety boundary parameters are dynamically adjusted according to the track type and train speed: on high-speed straight track sections, the safety boundary is defined as 2.5 meters on each side of the track, with the mask area covering a height range of 3.0 meters above the track; on curved sections, the lateral offset is calculated based on the track curvature radius, and the safety boundary is adjusted accordingly to 2.0 meters outside the track centerline; in low-speed turnout areas, the safety boundary narrows to 1.8 meters on both sides, and the height range is adjusted to 2.5 meters. These parameters are issued in real time by the train operation control system to ensure that the mask is highly matched to the current track scene. The intersection calculation determines the degree of intrusion by calculating the overlap ratio between the target bounding box and the mask area; for example, an overlap ratio of 0.8 indicates severe intrusion, while a ratio of 0.2 indicates slight intrusion. This determination method can adapt to different track scenes and ensure the accuracy of the determination.

[0050] In one embodiment, the tiered early warning decision model comprehensively assesses foreign object risk using multi-dimensional indicators. For example, if a 0.5 cubic meter rock is detected in the center of the track, and its location completely encroaches on the safety boundary while the train is approaching at a speed of 300 km / h, it is determined to be an emergency warning, requiring immediate braking response. If a small gravel with a volume of 0.05 cubic meters is detected at the edge of the track, and its location only slightly encroaches on the boundary while the train is traveling at a speed of 200 km / h, it is determined to be a routine warning, requiring recording and notification to inspection personnel. If a person is detected far from the track, and their location does not encroach on the boundary, it is determined to be a safety alert, requiring only continuous monitoring. This tiered approach ensures that the warning response matches the level of risk.

[0051] In one possible implementation, the output information of the identification results includes the target category, confidence score, pixel coordinates, and spatial location information. For example, if a rock is detected in the center of the track, the output information would be the category label "rockfall," a confidence score of 0.95, pixel coordinates within the image center region, and a three-dimensional spatial location of 50 meters in front of the train. This information is transmitted to subsequent systems in structured data form, ensuring that the driver or automatic control system can quickly obtain key information. Simultaneously, the output information can be combined with a visual interface to mark the target location and warning level on the cockpit display screen, facilitating manual confirmation.

[0052] Step S5 involves constructing a weighted joint training mechanism that includes target classification, intrusion determination, and modal fusion constraints. During training, the weights of each task are adaptively adjusted to enhance the perception of weak intrusive objects. To address the class imbalance problem, data augmentation and weight adjustment are used to balance the training effects of various targets. Knowledge distillation technology is employed, using the trained complex model as the teacher model to guide the lightweight student model in learning key features. Channel pruning technology is used to remove feature channels that contribute little to recognition. Combined with quantization technology, the model parameters are converted into low-precision integers. Inference acceleration tools are used to parallelize the computational operations to ensure that the model meets the latency requirements for real-time inference on the vehicle edge computing unit.

[0053] To ensure detection accuracy while meeting the real-time requirements of high-speed trains, the entire recognition model underwent multi-task joint training and lightweight optimization. A weighted joint training mechanism was constructed, incorporating target classification, intrusion determination, and modality fusion constraints. By adaptively adjusting the weights of each task, the model's ability to perceive small, intrusive objects was significantly enhanced. For example, during training, the weight ratio of the classification task was increased for difficult-to-identify targets such as small gravel and flexible floating objects, ensuring the model's sensitivity to these targets. Furthermore, to address class imbalance issues, such as a higher number of rockfall samples and a lower number of human intrusion samples, data augmentation and weight adjustments were used to balance the training effects across different target types.

[0054] In terms of model optimization, knowledge distillation is introduced, using a trained complex model as a teacher model to guide a lightweight student model in learning key features. The teacher model contains more parameters and deeper feature extraction capabilities, while the student model learns the output distribution of the teacher model, retaining core recognition capabilities while reducing computational complexity. Furthermore, channel pruning is used to remove feature channels that contribute less to recognition, for example, reducing the original number of feature channels from 256 to 128, significantly reducing computational load. Combined with quantization techniques, model parameters are converted from high-precision floating-point numbers to low-precision integers, for example, from 32-bit floating-point numbers to 8-bit integers, further reducing storage space and inference time.

[0055] In terms of deployment optimization, specialized inference acceleration tools are used to parallelize the model's computational operations. For example, batch optimization of convolutional operations on feature maps improves computational efficiency. Through multiple rounds of iterative optimization, the inference latency of the model on the onboard edge computing unit is ensured to be controlled within 100 milliseconds, meeting the real-time protection requirements under high-speed train operation conditions. This lightweight design enables the model to run efficiently on computationally limited onboard equipment while maintaining high accuracy in foreign object recognition.

[0056] In one embodiment, the multi-task joint training mechanism optimizes model performance through adaptive weight adjustment. For example, the training strategy employs a phased dynamic weight adjustment approach. In the early stages of training, the focus is on improving the model's basic ability to distinguish foreign object categories, with relatively high weights for target classification loss and relatively low weights for intrusion detection and modal fusion constraint loss. In the middle stages of training, as classification ability stabilizes, the weight of the intrusion detection loss is gradually increased to strengthen the model's judgment of target location and intrusion degree. In the later stages of training, modal fusion constraint loss is further introduced to ensure the complementarity of multimodal features. The sum of the three task loss weights remains constant across all stages, with the specific value adaptively adjusted based on the intrusion detection recall rate on the validation set.

[0057] In one possible implementation, knowledge distillation guides student model learning through a teacher model. For example, the teacher model is a deep network with a large number of parameters, capable of extracting complex multimodal features, such as subtle textures in visible light images and precise geometric information from LiDAR point clouds. The student model, on the other hand, is a lightweight network with fewer parameters that learns key features, such as the ability to focus on edge and thermal features of small pebbles, by mimicking the output distribution of the teacher model. Through this guidance approach, the student model reduces computational complexity while preserving as much of the teacher model's recognition accuracy as possible.

[0058] In one embodiment, channel pruning technology analyzes the importance of feature channels and removes redundant channels. For example, in the fused feature map, some channels contribute little to foreign object recognition, such as channels that only reflect background noise. Channel pruning can remove redundant channels with a contribution below a preset threshold in the fused feature map, significantly reducing the number of model parameters. Furthermore, INT8 quantization technology is used to convert the weights of the pruned model from 32-bit floating-point numbers to 8-bit integers, theoretically reducing the model's storage footprint to about a quarter of its original size, while also reducing memory bandwidth requirements during inference. The model optimized in these two steps significantly reduces computation and storage usage while maintaining controllable loss in target detection accuracy. This optimization method significantly reduces computational and storage burdens while maintaining model accuracy.

[0059] In one possible implementation, inference acceleration tools improve inference speed by parallelizing model computation operations. For example, batch processing of feature map convolution operations increases the number of feature maps computed at once from one to four, reducing computational latency. Simultaneously, the model's memory access patterns are optimized, such as storing data in contiguous blocks, reducing fragmented memory read times. Through these optimizations, the inference latency on the onboard edge device is reduced from 200 milliseconds to 90 milliseconds, ensuring millisecond-level response times are met for high-speed trains.

[0060] Step S6: The optimized lightweight model is deployed on the onboard edge computing unit to receive data streams from the multimodal sensing array in real time for foreign object identification and intrusion determination. The identification results are output in structured data form and fed back to the audible and visual alarm module in the train driver's cab in real time. The audible and visual alarm module triggers corresponding audible and visual feedback according to the warning level. The warning information is linked to the train operation control system to execute corresponding emergency strategies according to the warning level, including triggering automatic braking procedures, recording events and notifying the ground control center or updating monitoring logs. Log records are generated for each warning event and uploaded to the cloud system for analysis on a regular basis to identify high-risk areas and strengthen protective measures accordingly.

[0061] The trained and optimized lightweight model is deployed on the onboard edge computing unit to receive data streams from the multimodal sensing array in real time, performing millisecond-level foreign object identification and intrusion determination. The identification results are output in structured data form, including target category, location information, and warning level, and are fed back to the audible and visual alarm module in the train's driver's cab in real time. For example, when an emergency warning target is detected in the center of the track, a high-frequency sound and flashing red lights will be emitted in the driver's cab to remind the driver to take emergency measures.

[0062] Simultaneously, the early warning information is linked to the train operation control system, which executes corresponding emergency strategies based on the warning level. For example, an emergency warning triggers an automatic braking procedure to reduce the train speed to a safe range; a routine warning records the event information and notifies the ground control center to arrange inspections; and a safety alert only updates the monitoring log for subsequent analysis. In this way, proactive safety protection is achieved throughout the entire process, from foreign object detection and risk assessment to closed-loop handling, ensuring the safety and stability of the train in complex operating environments.

[0063] In one embodiment, the real-time inference process achieves millisecond-level response by optimizing data stream processing. For example, the onboard edge computing unit receives multimodal data streams at a frequency of 30 frames per second, with a frame interval of approximately 33 milliseconds. To meet real-time processing requirements, a pipelined parallel architecture is adopted, scheduling the computational tasks of feature extraction, feature fusion, and target detection in parallel, keeping the end-to-end inference latency per frame below 100 milliseconds. After lightweight optimization, the inference latency is significantly reduced compared to the large model scheme before optimization, meeting the protection requirements of millisecond-level response for high-speed trains.

[0064] In one possible implementation, the audible and visual alarm module is designed with different feedback mechanisms based on the warning level. For example, an emergency warning triggers an intermittent siren at a rhythm of 5 times per second, accompanied by a rapidly flashing red indicator light at a rhythm of 3 times per second, ensuring that the driver can quickly perceive the risk even in noisy environments; a regular warning triggers an intermittent siren at a rhythm of 2 times per second, accompanied by a slowly flashing yellow indicator light; a safety alert triggers only a single siren sound at a rhythm of 1 time per second, accompanied by a constantly lit green indicator light. Through this tiered feedback, the driver can quickly assess the level of risk and take appropriate measures.

[0065] In one embodiment, the emergency response strategy is implemented in conjunction with the train operation control system based on the warning level. For example, when an emergency warning target is detected, such as a large rock in the center of the track, the system automatically triggers a braking procedure, reducing the train speed from 300 km / h to 100 km / h, and simultaneously notifies the ground control center to dispatch an emergency team. When a routine warning target is detected, such as small gravel at the edge of the track, the system records the event details and uploads them to a cloud database, arranging subsequent inspections and handling. When a safety warning target is detected, such as unrelated personnel far from the track, the system only updates the monitoring log for later analysis. This closed-loop response mechanism ensures that risks are responded to and handled promptly.

[0066] In one possible implementation, the closed-loop handling process generates detailed log records for subsequent analysis and optimization. For example, each warning event generates a log record containing information such as the event time, train location, target category, warning level, and handling measures. The log record's field structure includes the event timestamp, real-time train mileage coordinates, identified target category and confidence level, warning level, and triggered emergency response measures. This data is stored in a structured format, facilitating subsequent statistical analysis of high-risk sections by the cloud system and the development of targeted protection plans. This log data is periodically uploaded to the cloud system, and data analysis identifies high-risk areas, such as sections of track frequently experiencing rockfalls. Targeted protective measures can then be strengthened to improve the overall safety level.

[0067] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the concept of this application. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for identifying foreign object intrusion on high-speed trains based on multimodal data fusion, characterized in that, Includes the following steps: S1. Construct an on-board multimodal sensing array to simultaneously acquire visible light images, infrared thermal imaging, lidar point clouds, and track boundary mask data. Perform differentiated preprocessing on each modal data to obtain preprocessed multimodal data. S2. A depth calibration mechanism combining rigid alignment based on timestamps and flexible alignment based on target features is used to perform spatiotemporal alignment on the preprocessed multimodal data. A multi-branch lightweight feature extraction architecture is constructed to extract features from each modal data after spatiotemporal alignment, thereby obtaining preliminary features for each modality. S3. Construct a dynamic gating fusion unit, calculate the dynamic weight coefficients of each modality with real-time environmental features as input, calibrate the preliminary features of each modality by combining channel attention mechanism and spatial attention mechanism, and fuse the calibrated modal features by an adaptive stitching layer according to the dynamic weight coefficients to obtain a deep fusion feature map. S4. Based on the deep fusion feature map, a lightweight detection head structure is used to identify foreign objects in the track and surrounding area, and to generate target category and pixel-level position information. The target location information is intersected with the track boundary mask in real time to determine whether the target has intruded into the track safety clearance and the degree of intrusion. Based on the comprehensive target category, three-dimensional volume, movement speed, and intrusion depth, a graded early warning decision model is used to output graded early warning results.

2. The method according to claim 1, characterized in that, The method further includes step S5, which specifically includes: A weighted joint training mechanism is constructed that includes target classification, intrusion determination and modality fusion constraints. During the training process, the weights of each task are adaptively adjusted to enhance the perception ability of weak intrusive foreign objects. To address the class imbalance problem, data augmentation and weight adjustment are used to balance the training effects of various objectives. The knowledge distillation technique is used to guide the lightweight student model to learn key features by using a trained complex model as a teacher model. By using channel pruning techniques to remove feature channels that contribute little to recognition, combining quantization techniques to convert model parameters into low-precision integers, and using inference acceleration tools to parallelize computational operations, the model can meet the latency requirements for real-time inference on the vehicle edge computing unit.

3. The method according to claim 2, characterized in that, The method further includes step S6, which specifically includes: The optimized lightweight model is deployed on the vehicle edge computing unit to receive data streams from the multimodal sensing array in real time for foreign object identification and intrusion determination. The identification results are output in the form of structured data and fed back to the audio-visual alarm module in the train cab in real time. The audio-visual alarm module triggers the corresponding audio-visual feedback according to the warning level. The early warning information is linked to the train operation control system, which executes corresponding emergency strategies based on the early warning level, including triggering automatic braking procedures, recording events and notifying the ground control center or updating the monitoring log. Logs are generated for each warning event and uploaded to the cloud system regularly for analysis to identify high-risk areas and strengthen protective measures accordingly.

4. The method according to claim 1, characterized in that, In step S1, the differential preprocessing of each modality data to obtain preprocessed multimodal data specifically includes: The visible light image is sequentially processed by noise removal, adaptive contrast enhancement and motion blur correction. The motion blur correction is based on the blur kernel parameters estimated by the real-time speed of the train and the image sharpness is restored by deconvolution operation. An adaptive temperature threshold segmentation technique was used to extract foreground targets from infrared thermal imaging data, and the target contours were enhanced by morphological dilation. The lidar point cloud data is sequentially subjected to voxel mesh downsampling, statistical filtering for noise reduction, and density-based clustering to separate the target from the background and obtain the three-dimensional spatial pose information of the foreign object. By mapping the track boundary mask data bidirectionally between the world coordinate system and the image coordinate system, high-risk areas such as curve sections and turnout areas are marked. High-precision timestamp synchronization technology is used to align the time of each modality data.

5. The method according to claim 1, characterized in that, In step S2, the specific processes of rigid alignment and flexible alignment include: The rigid alignment uses high-precision timestamp synchronization technology to sort and match the data frames of each sensor, grouping data frames with time differences within a preset range into the same group, and projecting the lidar point cloud onto the image pixel coordinate system using the geometric parameters of the sensor installation position to eliminate spatial deviation caused by installation eccentricity. The flexible alignment is achieved through target-level feature association, which extracts target edge contour features from visible light images, target thermal distribution features from infrared data, and target three-dimensional geometric features from lidar point clouds. By using feature matching technology, a target correspondence is established between different modalities to compensate for the subtle deviations not eliminated in rigid alignment, thereby achieving consistency of multimodal data at the pixel and spatial levels.

6. The method according to claim 1, characterized in that, In step S2, the feature extraction of each modality data by the multi-branch lightweight feature extraction architecture specifically includes: Lightweight convolutional networks are used to extract fine-grained semantic features from visible light images and infrared thermal imaging data, respectively. The visible light branch focuses on extracting track edge texture features, while the infrared branch focuses on extracting target thermal distribution features. The global geometric structure features of the target, including the three-dimensional shape and spatial location of the foreign object, are extracted from the lidar point cloud data through a point cloud processing network. The geometric boundary information of the safe area is transformed into feature vectors by using a fully connected network for the track boundary mask data, thus explicitly modeling spatial boundary constraints.

7. The method according to claim 1, characterized in that, In step S3, the process of the dynamic gating fusion unit calculating the dynamic weight coefficients of each modality specifically includes: Light intensity and weather conditions are encoded into environmental feature vectors, which are then processed by a lightweight network to output the weight coefficients corresponding to each mode. In low light or low visibility weather conditions, the weighting coefficients of infrared thermal imaging and lidar point clouds are automatically increased, while the weighting coefficients of visible light images that are severely interfered with are reduced. The channel attention mechanism analyzes the global distribution of each modality feature map through global average pooling to generate channel weights to strengthen feature channels that contribute to foreign object recognition. The spatial attention mechanism generates spatial weights through global pooling operations, highlighting the spatial location of the target region in the feature map and suppressing the response of the background region. The adaptive stitching layer performs a weighted summation of the modal features after dual attention calibration according to the dynamic weight coefficients to generate a deep fusion feature map that combines texture, temperature, spatial and boundary information.

8. The method according to claim 1, characterized in that, In step S4, the construction and determination process of the hierarchical early warning decision model specifically includes: A multi-dimensional risk assessment index is constructed by combining the target's category attributes, three-dimensional volume estimation, real-time movement speed, and intrusion depth. The warning levels are divided into three levels: emergency warning, regular warning and safety alert. Emergency warning corresponds to the scenario where the target has completely intruded into the safety clearance and its size or speed meets the high-risk threshold. Regular warning corresponds to the scenario where the target has slightly intruded or has potential risks. Safety alert corresponds to the low-risk scenario where the target has not intruded into the safety clearance. The intersection calculation of the target bounding box and the track boundary mask dynamically adjusts the safety boundary parameters based on the track type and train speed, and outputs the target category, confidence level, pixel coordinates and spatial location information in the form of structured data.

9. The method according to claim 2, characterized in that, In step S5, the specific process of knowledge distillation includes: The teacher model is a deep network with a large number of parameters, which has the ability to extract complex features of multiple modes, including fine texture features in visible light images and precise geometric features in lidar point clouds. The student model is a lightweight network that learns from the output distribution of training samples by mimicking the teacher model, retaining the ability to perceive key features of weak targets, including responses to edge features of small gravel and thermal features of targets. The channel pruning technique analyzes the contribution of each channel in the fused feature map to the foreign object recognition task, removes channels with a contribution below a preset threshold, and reduces the number of model parameters. The quantization technique converts the weight parameters of the pruned model from high-precision floating-point numbers to low-precision integers. The inference acceleration tool performs batch parallel processing of convolution operations and optimizes the model memory access mode, keeping the inference latency within the range that meets the real-time protection requirements of vehicles.

10. The method according to claim 1, characterized in that, In step S4, the process of the lightweight detection head structure identifying foreign objects specifically includes: Multi-scale analysis is performed on the deep fusion feature map to extract the local texture features of small targets and the overall geometric features of large targets, respectively. The target category label and confidence score are generated through classification operations, and the pixel-level bounding box of the target is generated through regression operations. The recognition results are filtered based on the confidence score, and the detection results that meet the preset confidence conditions are retained to ensure the reliability of the recognition results in complex backgrounds. The selected target bounding box is intersected with the track boundary mask in real time, and the degree of intrusion of the target is determined by calculating the area ratio of the overlapping area.

Images