Rail scene thunder-vision fusion anti-invasion monitoring method and system based on sparse feature fusion

Abstract

The invention belongs to the technical field of track detection, and more specifically relates to a track scene thunder-vision fusion anti-intrusion monitoring method and system based on sparse feature fusion. The method comprises the steps that a laser radar obtains three-dimensional point cloud data, a high-definition camera obtains visible light image data, and data preprocessing is carried out; performing external parameter automatic calibration optimization and time alignment compensation on the laser radar and the high-definition camera to obtain time-space aligned multi-modal data; fusing the camera semantic features and the laser radar geometric features based on multi-modal data of space-time alignment to obtain fused features; and after multi-modal feature fusion is completed, a target heat map is generated through a sparse detection head, and graded early warning is realized in combination with a dynamic safety distance model. The method solves the problems that an existing recognition algorithm is high in false alarm rate, and the false alarm rate is higher under low-light conditions such as rainy days or dusk; the data dependence is strong, and the ability to detect unlabeled novel foreign matters is almost zero.

Description

Technical Field

[0001] This invention belongs to the technical field of track detection, and more specifically, relates to a track scene radar-visual fusion anti-intrusion monitoring method and system based on sparse feature fusion. Background Technology

[0002] With the rapid development of rail transit, track safety monitoring technology has become a core component of ensuring train operation safety. As a key part of the railway safety assurance system, track foreign object intrusion detection aims to identify obstacles on the track in real time and accurately, preventing damage to train operation and track facilities. Manual monitoring is limited by human eye fatigue and reaction speed, resulting in a high false negative rate and failing to meet the needs of complex scenarios. Therefore, in recent years, deep learning-based object detection algorithms have been widely applied in track environments.

[0003] Chinese patent document CN110781896A discloses a method, cleaning method, system, and resource allocation method for identifying and cleaning railway track waste. The method includes acquiring a railway track video stream; decomposing the video stream frame by frame and preprocessing each frame to filter image noise, the preprocessing including Cohen-Sutherland cropping; inputting the preprocessed image into a trained waste identification model; extracting image sample features; performing multi-scale prediction and bounding box prediction to identify waste; classifying the identified waste through multi-label classification; and removing the bounding boxes of the same waste detected multiple times through non-maximum suppression.

[0004] Existing recognition algorithms suffer from high false alarm rates, which are even higher under low-light conditions such as rain or dusk. They are also highly data-dependent, with virtually no ability to detect novel, unlabeled foreign objects. Detection in low-light conditions, such as at night or in tunnels, presents significant challenges, as camera image quality deteriorates drastically, target-background contrast decreases, and the feature extraction capabilities of traditional algorithms are significantly weakened. LiDAR, on the other hand, possesses high-precision 3D point cloud acquisition capabilities, overcoming lighting limitations to effectively detect low-reflectivity objects and providing long-range, high-resolution environmental perception. However, single-liDAR systems suffer from sparsity issues, multi-source noise interference, unquantified uncertainties, insufficient real-time performance, and response delay risks, and cannot acquire the semantic features of the target. Summary of the Invention

[0005] The present invention aims to overcome at least one of the defects of the prior art and provide a method and system for anti-intrusion monitoring of track scenes based on sparse feature fusion, so as to solve problems such as the contradiction between sparsity and computational efficiency, multi-source noise interference and uncertainty unquantified, and insufficient real-time performance and response delay risk.

[0006] The detailed technical solution of this invention is as follows: A radar-visual fusion intrusion prevention and monitoring method for orbital scenes based on sparse feature fusion, the method comprising: S1. The lidar acquires 3D point cloud data and the high-definition camera acquires visible light image data. Dynamic voxelization and contrast enhancement processing are performed on the 3D point cloud and visible light image respectively. S2. Perform automatic extrinsic parameter calibration and optimization and time alignment compensation on the lidar and high-definition camera to achieve millimeter-level spatial registration and sub-millisecond-level time synchronization between the lidar and the high-definition camera, and obtain spatiotemporally aligned multimodal data; S3. Based on spatiotemporal alignment, multimodal data is used to fuse camera semantic features with LiDAR geometric features to obtain fused features; S4. After completing the multimodal feature fusion, a target heatmap is generated through a sparse detection head, and a hierarchical early warning is achieved by combining it with a dynamic safety distance model.

[0007] Furthermore, the dynamic voxelization and contrast enhancement processing for the 3D point cloud and visible light images respectively specifically includes: dynamic voxelization processing of the 3D point cloud data acquired by the lidar: S11. Voxelized 3D point cloud data, using SAFDNet's adaptive voxel discarding strategy, dynamically adjusting voxel sizes based on distance to implement a hierarchical voxel strategy: For close distances d < 30m: voxel size 0.1m, capturing fine details at close range; for medium distances 30m ≤ d < 60m: voxel size 0.2m, balancing accuracy and efficiency; for long distances d ≥ 60m: voxel size 0.3m, reducing redundant calculations at long distances. The voxel size is dynamically adjusted based on distance, and the calculation method is as follows: (1); In formula (1), The distance from the point cloud in the scene to the LiDAR. Voxel size.

[0008] Then calculate the centroid coordinates of the point cloud within the voxel. Average reflection intensity and point cloud density Concatenate the data to generate voxel-level feature vectors. , C is the feature dimension, N is the number of voxels, and R is the set of real numbers; where the centroid coordinates and point cloud density are geometric features, and the average reflection intensity is the intensity feature.

[0009] The voxel-level eigenvector is denoted as ,in This refers to voxel features, which integrate the spatial location, density, and reflection intensity information of point clouds within a voxel. It can comprehensively characterize the features of a voxel, where C is the feature dimension and N is the number of voxels.

[0010] S12, noise reduction treatment; To suppress noise interference, a noise-resistant submanifold convolution NRConv is introduced to incorporate the voxel features of the lidar. voxel coordinates projected onto the image plane The input to the NRConv module suppresses noise and preserves effective features through 3D-2D joint convolution, yielding effective point cloud voxel features: (1); In formula (1), The point cloud voxel features after noise reduction processing are the effective point cloud voxel features.

[0011] Furthermore, feature enhancement is performed on the visible light image data, specifically as follows: S13. Contrast Enhancement: To improve feature quality under low-light conditions, adaptive histogram equalization and Retinex enhancement are performed sequentially on the image features to obtain the enhanced image. That is, valid image data: Adaptive histogram equalization enhances local details by limiting contrast: Configuration parameters: Contrast threshold =3.0, image tile grid size =16×16; The calculation formula is as follows: (2); In formula (2), The input is a visible light image. The image after adaptive histogram equalization; Retinex enhancement, combined with a multi-scale Retinex algorithm to enhance color realism, is configured with Gaussian kernel standard deviation as the parameter. =15; The final enhanced image Effective image: (3); In formula (3), is the Gaussian kernel standard deviation.

[0012] Furthermore, S2 specifically includes: S21. Spatial Registration: Addressing the inherent differences between the coordinate systems of the LiDAR and the camera, a three-step progressive calibration process is employed. Combined with high-quality images enhanced for contrast, SuperGlue is used for automatic extrinsic parameter calibration and optimization of the LiDAR and high-definition camera, achieving millimeter-level spatial registration. Specifically, this includes: S211, Coarse LiDAR Pose Correction: The input for coarse LiDAR pose correction is the 3D point cloud data collected by the LiDAR, which includes 3D coordinates, reflection intensity, and timestamp information. Due to sensor movement or dynamic changes in the scene, these point clouds have inter-frame misalignment and motion distortion.

[0013] The system inputs 3D point cloud data, dynamically integrates multiple frames of point clouds using the continuous-time ICP algorithm, and then constructs a dynamic point cloud model using a linear iVox structure. The LiDAR pose is optimized by minimizing the distance between the scan points and model points, outputting the optimized LiDAR pose transformation matrix and a dynamic point cloud model that filters out dynamic noise points while preserving static scene features. Finally, the LiDAR pose is optimized, eliminating sensor installation and motion errors, and providing an accurate point cloud reference for subsequent feature matching and fine registration. The output pose transformation matrix directly affects the accuracy of subsequent extrinsic parameter calibration, ensuring more accurate spatial coordinate transformation between the lidar and the camera, thereby guaranteeing the accuracy of spatiotemporal synchronization and laying a reliable foundation for subsequent steps such as sparse feature fusion.

[0014] S212, Feature Matching and Initial Estimation; First, the centroid coordinates and average reflection intensity of the point cloud within the effective point cloud voxel feature voxels are extracted. The centroid coordinates represent the spatial position, and the average reflection intensity represents the echo characteristics. The projection model is selected according to the field of view (FoV) of the lidar: pinhole projection is used when FoV is less than 150°, and panoramic projection is used when FoV is ≥ 150°. The three-dimensional centroid coordinates are mapped to a two-dimensional grayscale image according to the reflection intensity. At the same time, a point index map is generated to quickly query the three-dimensional coordinates corresponding to each pixel. The centroid coordinates are projected onto the image plane, and the average reflection intensity of the voxels is used as the pixel value to generate a two-dimensional intensity image.

[0015] After generating a two-dimensional intensity image of the LiDAR, the SuperPoint algorithm is used to detect key points in the LiDAR intensity image and the camera image respectively, and a local descriptor is generated for each key point. These descriptors include gradient and texture feature information around the key point, ensuring that the descriptors of the same physical point in the cross-modal image have high similarity.

[0016] Subsequently, a graph neural network (GNN) is used to match key point descriptors between the two modalities. The GNN treats key points as nodes in a graph and the relationships between key points as edges. Through a multi-layer message passing mechanism, the associated features between nodes are learned, and then the matching probability of the LiDAR and camera key point descriptors is calculated. A matching threshold of 0.05 is set, and only point pairs with a matching probability higher than this threshold are retained to filter out low-reliability matches and ensure the spatial consistency of corresponding point pairs.

[0017] Furthermore, an initial rotation matrix estimation is performed between the lidar and camera coordinate systems. The initial estimation object is the rotation matrix from the lidar coordinate system to the camera coordinate system. This method describes the attitude transformation relationship between two sensors in space. Its core objective is to quickly solve for the initial rotation relationship between the two coordinate systems using cross-modal correspondences provided by feature matching pairs. This provides an iterative starting point for subsequent fine registration and avoids the optimization process getting trapped in local optima. Based on keypoint matching pairs, the RANSAC algorithm is used for initial estimation, estimating the rotation matrix from the LiDAR coordinate system to the camera coordinate system. Using the Cauchy kernel function Construct an error function to quantify the deviation between the coordinates of the LiDAR points projected onto the camera coordinate system and the actual camera point coordinates: (4); In formula (4), For the Cauchy kernel function, It is the rotation matrix from the lidar coordinate system to the camera coordinate system, which is continuously optimized during calculation to find the optimal rotation relationship. This is the optimized rotation matrix, obtained by minimizing the error in subsequent formulas. Indicates the first The coordinates of a lidar point in its own coordinate system Indicates the first The coordinates of each LiDAR point in the camera coordinate system are M, where M is the number of corresponding point pairs involved in the calculation. Robust estimation is performed using the Cauchy kernel function to find the rotation matrix that minimizes the error. That is, the optimal rotation matrix.

[0018] S213, Fine-grained registration, i.e., NID optimization; First, based on the optimal rotation matrix, the effective point cloud voxel features are... The corresponding point cloud coordinates are projected onto the camera image plane. The nearest point corresponding to each pixel is retained through a depth buffer test, and occluded invalid points are filtered out to output a set of visible point clouds. including three-dimensional coordinates and reflection intensity.

[0019] Then, with visible point clouds and image enhancement Calculate the normalized information distance using the input: (5); In formula (5), For entropy, For mutual information; This is the lidar point cloud after visible point removal, retaining only the visible points from the camera's view, including 3D coordinates. Based on the reflection intensity, occluded invalid points are filtered out, focusing on the effective area that matches the image features.

[0020] Finally, the Nelder-Mead optimization algorithm is employed, with the NID value as the optimization objective, to iteratively adjust the rotation matrix. Translation vector The initial extrinsic parameter matrix is ​​used until the NID value converges to its minimum, and the optimal extrinsic parameter matrix for achieving millimeter-level spatial registration is output. The extrinsic parameter matrix includes the rotation matrix. Translation vector .

[0021] S22, Time Alignment Compensation; The automatic extrinsic calibration based on SuperGlue solves the problem of spatial coordinate system alignment between the LiDAR and the camera (outputting millimeter-level extrinsic matrix), while the temporal alignment compensation compensates for the temporal sampling difference between the two. Together, they form a complete closed loop of "spatiotemporal synchronization". Spatial registration accuracy is the basis of temporal alignment. If the spatial coordinate system is not aligned, even if the temporal synchronization is accurate, cross-modal features will still fail to fuse due to spatial deviation.

[0022] The automatic extrinsic calibration based on SuperGlue achieves spatial alignment through "dynamic point cloud preprocessing - feature matching - fine registration". Temporal alignment compensation is based on the spatial alignment result and achieves temporal alignment through "hardware-triggered coarse synchronization - Kalman filter fine compensation". Finally, it outputs spatiotemporally aligned multimodal data, providing a spatiotemporally consistent data source for subsequent feature fusion.

[0023] The time alignment achieved through "hardware-triggered coarse synchronization - Kalman filter fine compensation" specifically includes: S221, Hardware-triggered coarse synchronization: First, a preliminary unification of the time base is achieved through hardware-triggered coarse synchronization. Specifically, the LiDAR and the camera are connected using a hardware synchronization trigger signal line. When the sensor collects data, a synchronization trigger signal is sent, which initially reduces the original timestamp error between the two from the 500μs level to the 1ms level, laying the foundation for subsequent fine compensation.

[0024] S222, Kalman filter fine compensation: After hardware coarse compensation, Kalman filtering fine compensation is performed. A first-order Kalman filter model is used to iteratively smooth the residual time error. The core operation is to iteratively smooth the residual time error based on the Kalman filter model, so that the timestamp error of the multi-sensor is stabilized at <1ms, ensuring that the LiDAR point cloud and the camera image are strictly synchronized in the time dimension.

[0025] The formula for predicting timestamps using a first-order Kalman filter is as follows: (6); In formula (6), This represents the synchronized timestamp predicted at time k, which is a weighted fusion result of the current measurement and historical estimates. This is the sensor's raw timestamp at time k, directly acquired from the hardware clock of the LiDAR or camera. This is the synchronized timestamp predicted at time k-1, which is the time estimate of the previous time. =0.95 is the filter coefficient, used to balance current measurements with historical estimates. This is the frame interval time.

[0026] Use the current sensor's original timestamp Synchronization timestamp predicted from the previous moment By performing weighted fusion and iterative adjustments, the timestamp error of the multiple sensors was eventually stabilized at <1ms.

[0027] Furthermore, through spatiotemporal synchronization registration and synchronization trigger compensation, the LiDAR and camera data have achieved sub-millisecond time synchronization and millimeter-level spatial registration, forming spatiotemporally aligned multimodal data. Based on this, the viewpoint-aware query generation stage begins, where a lightweight detection model establishes a direct correlation between the 2D semantic features of the camera image and the 3D geometric features of the LiDAR. Specifically, S3 includes: S31, View Perception Query Generation, i.e., Lightweight View Detection; The view perception query generation adopts lightweight view detection, takes spatiotemporally aligned multimodal data with time alignment compensation output as input, and obtains the three-dimensional coordinates of the object in the lidar coordinate system and the corresponding camera semantic features through 2D detection, monocular 3D detection and three-dimensional coordinate transformation, providing an accurate cross-modal data foundation for sparse feature fusion.

[0028] 2D detection aims to quickly locate foreign objects in the image plane using a lightweight FCOS detector; monocular 3D detection infers the three-dimensional position of the foreign object in space based on the 2D results; 3D coordinate transformation uses an external parameter matrix to unify the monocular 3D coordinates to the lidar coordinate system, ensuring consistency of the spatial reference of multimodal data, thereby realizing the initial mapping from image semantics to spatial geometry, and preparing for subsequent high-precision fusion and alarm decision-making.

[0029] The viewpoint-aware query generation specifically includes: S311, 2D inspection; Using a lightweight FCOS detector, we rapidly detect 2D bounding boxes of foreign objects based on spatiotemporally aligned multimodal data, achieving preliminary localization of foreign objects on the image plane. The output includes 2D detection results containing category and confidence level. Furthermore, based on the 2D bounding boxes, we output intermediate layer semantic features of the corresponding image locations. These are the semantic features of the camera, which provide the semantic feature foundation for subsequent 3D localization.

[0030] S312, Monocular 3D Inspection; Based on the 2D detection results, combined with camera intrinsic parameters and depth estimation models, the three-dimensional spatial location information of the foreign object is deduced from the monocular image, and the three-dimensional outline of the foreign object in space is initially constructed. The 2D information of the foreign object is extended to 3D space to obtain the 3D information of the foreign object.

[0031] S313, Three-dimensional coordinate transformation; Based on the optimal extrinsic parameter matrix obtained from calibration, the 3D information of foreign objects obtained from monocular 3D detection is transformed from the camera coordinate system to the lidar coordinate system, realizing a unified expression of multimodal data under the same spatial reference. The 2D coordinates are converted into 3D spatial points, i.e., spatial mapping relationships, through the camera's intrinsic and extrinsic parameter matrices, and the intrinsic parameter inverse matrix is ​​used to achieve this. Image coordinates The depth d is converted to normalized coordinates in the camera coordinate system, and then converted to the inverse extrinsic matrix. After removing the influence of camera extrinsic parameters, the three-dimensional coordinates of the target in the camera coordinate system are finally obtained. The formula is as follows: (7); In formula (7), The target's three-dimensional coordinates in the lidar coordinate system. The inverse matrix of the camera's extrinsic parameters. The inverse matrix of the camera's intrinsic parameters. , where are the image coordinates of the center point of the 2D detection box, and d is the target depth value predicted by monocular 3D detection, output by monocular 3D detection.

[0032] This is the inverse of the camera extrinsic matrix, used to transform points from the world coordinate system to the camera coordinate system; This is the inverse of the camera intrinsic parameter matrix, used to convert image plane coordinates into normalized coordinates in the camera coordinate system; , where are the image coordinates of the center point of the 2D detection box, and d is the target depth value predicted by monocular 3D detection, output from the monocular 3D detection step.

[0033] S32, Dynamic Feature Diffusion; After completing the lightweight view detection, the dynamic feature diffusion step is entered: taking the three-dimensional coordinates of the foreign object in the lidar coordinate system and the corresponding camera semantic features as input, the diffusion radius and sparse convolution operation are dynamically adjusted to obtain the diffused dense point cloud features, which solves the feature discontinuity problem caused by the sparse lidar point cloud and provides a more complete cross-modal feature expression for subsequent uncertainty perception fusion.

[0034] Furthermore, the dynamic feature diffusion specifically includes: S321. Calculation of diffusion range; The diffusion radius is dynamically adjusted based on the target size and distance, using the 3D coordinates of the foreign object in the lidar coordinate system generated by the view-perception query and the corresponding camera semantic features as input. The calculation formula is as follows: (7); In formula (7),, The feature diffusion radius of the i-th target determines the neighborhood range of the 3D sparse convolution. The average size of the target is used to ensure that the diffusion range matches the physical size of the target, with an exponential decay coefficient. This automatically shrinks the diffusion range of distant targets, reducing redundant calculations. d represents the straight-line distance between the target and the lidar, derived from the three-dimensional coordinate transformation result of the step.

[0035] S322, 3D sparse convolution: The sparse point cloud features are diffused using a 3×3×3 sparse convolution, as follows: (8); In formula (8), The characteristics of the dense point cloud after diffusion. The input is sparse point cloud features, which are effective point cloud voxel features. It is obtained through the spatial mapping relationship of S313. Indicates the dynamic diffusion radius, ⊕ indicates feature splicing. The operation is a 3×3×3 sparse convolution. While preserving the original sparsity, it diffuses the features of the center point to the neighboring voxels, thus solving the problem of feature loss caused by the sparsity of distant point clouds.

[0036] S33, Modal uncertainty quantification; Based on the diffused dense point cloud features and camera semantic features, the fusion weights are dynamically allocated by quantifying the localization uncertainty of the two types of sensors, and finally the fused features are output. ; Steps S32 and S33 are closely linked through a feature transfer and fusion strategy: dynamic feature diffusion, through adaptive diffusion radius calculation and 3D sparse convolution calculation, diffuses the sparse point cloud features of the LiDAR to neighboring voxels, generating dense point cloud features containing spatial context information. And retain the semantic association with the three-dimensional coordinates corresponding to the dynamic feature diffusion output.

[0037] Furthermore, to address multi-sensor noise interference, a confidence-driven dynamic fusion strategy is designed, wherein the modal uncertainty quantification specifically includes: S331, Uncertainty Modeling; The object of lidar modal modeling is the diffused lidar point cloud features output by the dynamic feature diffusion step. The system assesses localization reliability by measuring the spatial distance between point cloud geometric features and the target's true location. The input is the dense point cloud features output from the step-by-step dynamic feature diffusion; the true bounding box coordinates B are derived from labeled data; and the output is the LiDAR modal confidence weights. The value ranges from [0,1], and the larger the value, the more reliable the positioning.

[0038] The lidar modes are as follows: (9); In formula (9), For the confidence weight of lidar, Features of lidar This represents the Euclidean distance of the lidar point cloud on the BEV plane. Let B be the regression function for camera features, and B be the coordinates of the ground truth bounding box. Camera modal modeling is as follows: (10); In formula (10), This represents the confidence weight of the camera mode, with a value range of [0,1]. A larger value indicates that the camera positioning is more reliable. This is a regression function for camera features, which converts camera semantic features into predicted bounding box coordinates. The camera semantic features derived from the step-lightweight viewpoint detection include information such as the 2D detection bounding box, category, and confidence score of the foreign object. B represents the coordinates of the ground truth bounding box, derived from the labeled data or the baseline coordinates of the preceding detection results. This represents the Euclidean distance of the LiDAR point cloud on the BEV plane. It is the Euclidean distance on the BEV plane based on the predicted bounding box and the ground truth bounding box. To quantify the uncertainty of positioning, the closer the distance, the higher the confidence level.

[0039] S332, Weighted Fusion: The core objective of weighted fusion is to dynamically assign weights based on sensor confidence levels, fusing camera semantic features and LiDAR geometric features, with camera semantic features as the input. and lidar features Improve detection robustness; (11); In formula (11), This feature is a fusion feature that integrates semantic and geometric features.

[0040] This invention dynamically allocates weights based on the current confidence level of the sensor. In low-reflectivity object detection scenarios, the camera's contribution weight can be increased to 65%, significantly enhancing detection robustness.

[0041] Furthermore, S4 specifically involves: fusing features Target heatmaps are generated using sparse detectors, and graded early warnings are achieved by combining them with a dynamic safety distance model. S41. Central heat map generation: Target heat map is generated through sparse detection head, and graded early warning is achieved by combining dynamic safety distance model; To address the challenges of perceiving sparse features, this invention designs a center heatmap generation algorithm based on a dynamic Gaussian kernel. The input consists of fused feature voxels output by step modal uncertainty quantization and target physical size parameters obtained through step-lightweight perspective detection-dynamic feature diffusion. The output is a two-dimensional center heatmap. Its pixel value range is [0,1], which represents the probability density of the corresponding position being the geometric center of the target.

[0042] First, an algorithm for generating center heatmaps based on dynamic Gaussian kernels is designed, and the Gaussian kernel parameters are dynamically adjusted: The input consists of the target width W and length L obtained from monocular 3D detection and feature diffusion; the output is the Gaussian kernel standard deviation. Among them, W and L depend on the spatiotemporally synchronized extrinsic parameter matrix and the S313-S332 cross-modal feature fusion results.

[0043] Calculate the Gaussian kernel standard deviation based on the target physical dimensions: (12); In formula (12), and The design sets the target width and length; this design matches the Gaussian kernel coverage to the actual size of the target, avoiding the blurring or oversharpening problems caused by traditional fixed kernel parameters.

[0044] Then, a center heatmap generation algorithm based on dynamic Gaussian kernels is used to generate the heatmap, resulting in the center heatmap. Abstract fusion features The probability distribution is converted into a center heatmap, which is a two-dimensional heatmap with pixel values ​​ranging from [0,1]. Each pixel value represents the probability that the corresponding location is the center of the target.

[0045] S42, Tiered early warning strategy; Based on the dynamic relationship between the actual distance between the target and the track and the train's operating speed, combined with the probability distribution of the target's location in the central heat map, a graded early warning system is implemented through a preset safe distance threshold.

[0046] Dynamic safety distance: (13); In formula (13), The safety protection threshold is dynamically adjustable based on the train's real-time speed at different speeds. The train speed V is input in real time from the track signal system. The model comprehensively considers the train braking distance and emergency avoidance space to ensure that the safety protection threshold is dynamically adjustable at different speeds. The train speed V is input in real time from the track signal system, forming a cross-system data linkage with the acquired data and data processing.

[0047] Establish a three-level early warning threshold: Yellow alert: Target distance < yellow alert threshold and speed > 50km / h → trigger audible and visual alerts to remind monitoring personnel to pay attention; Orange alert: Target distance < orange alert threshold and speed > 30km / h → activate braking system pre-charge air to shorten subsequent braking response time; Red alert: If the target distance is less than the red alert threshold, emergency braking will be triggered immediately to force the train to stop and avoid a collision. The three-level warning thresholds are set based on multiples of the safe distance, namely, yellow warning threshold = 4 × safe distance, orange warning threshold = 2 × safe distance, and red warning threshold = safe distance.

[0048] Preferably, the target distance is determined by extracting the target position from the high-response area generated by the central heat map, combining the three-dimensional coordinates in the lidar coordinate system output by the lightweight perspective detection, and then calculating it after spatial mapping to the orbital plane.

[0049] Specifically, millimeter-level spatiotemporal synchronization accuracy is achieved through SuperGlue-based automatic extrinsic calibration technology, ensuring accurate spatial mapping between 3D coordinates and heatmap positions, thereby guaranteeing the reliability of target distance calculation. Preferably, the high-response region is... Areas >0.7.

[0050] The generated central heatmap provides a probability distribution of target locations for multi-threshold decision-making, replacing the traditional "single-point detection" mode. It improves the robustness of detection in complex scenes through probability density weighting. Multi-threshold decision-making then transforms the spatial information in the heatmap into executable safety instructions, which is the final link in the "feature fusion-localization-decision" chain.

[0051] In another aspect of the present invention, a system for a radar-visual fusion intrusion prevention and control method for orbital scenes based on sparse feature fusion is provided, the system comprising a hardware module and a software module: The hardware modules include: a heterogeneous sensor combination of LiDAR and high-definition camera, and an edge computing unit. LiDAR: Deployed on both sides of the track to acquire high-precision 3D point cloud data, with a maximum effective detection range of 200 meters; HD camera: Equipped with a high dynamic range imaging module, a resolution of 2560×1440, and covering a 120° field of view; Edge computing unit: Integrates a high-performance GPU to support real-time processing and fusion computing of sensor data; The software modules include five core modules: data acquisition, data preprocessing, spatiotemporal synchronization and calibration, sparse feature fusion, and intrusion detection and early warning. The system comprises the following modules: a data acquisition module, which acquires relevant data collected by the hardware module; a data preprocessing module, which receives data from the data acquisition module and performs preprocessing; a spatiotemporal synchronization and calibration module, which ensures that the timestamp error of multiple sensors is <1ms through hardware triggering and Kalman filtering compensation; a sparse feature fusion module, which performs hierarchical extraction of LiDAR voxel features and camera image features respectively, and combines sparse convolution and uncertainty perception mechanisms to achieve cross-modal feature complementarity; and an intrusion detection and early warning module, which generates a central heat map based on a sparse detection head and outputs protection commands in conjunction with a hierarchical early warning strategy.

[0052] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) The present invention provides a method and system for preventing intrusion into orbital scenes based on sparse feature fusion and radar-visual fusion. By using a hierarchical dynamic voxelization strategy and SAFDNet's adaptive voxel discarding algorithm, the computational cost of long-distance monitoring is reduced by more than 60% compared with traditional dense feature methods. On a domestic edge computing platform, the inference time of the present invention meets the real-time requirements of orbital safety protection.

[0053] (2) The present invention provides a method and system for intrusion prevention monitoring of orbital scenes based on sparse feature fusion and laser-visual fusion. The uncertainty-aware fusion strategy dynamically adjusts the fusion weights by quantifying the positioning uncertainty of the camera and laser radar, which greatly improves the noise resistance. In low-reflectivity object detection scenarios, the camera weight can be increased to 65%, which improves the detection rate by 2.8% and reduces the false alarm rate by 15% compared with traditional methods. In addition, the noise-resistant submanifold convolution NRConv effectively suppresses point cloud noise and retains 92% of the effective features, which is a significant improvement compared with traditional methods.

[0054] (3) The present invention provides a method and system for anti-intrusion monitoring of track scenes based on sparse feature fusion, which achieves millimeter-level spatial registration and sub-millisecond-level time synchronization, i.e., spatiotemporal synchronization accuracy optimization, based on SuperGlue's automatic extrinsic parameter calibration and NID optimization algorithm. In multi-sensor asynchronous acquisition scenarios, the spatiotemporal alignment accuracy of the present invention is 3 times higher than that of the traditional ICP algorithm, ensuring the accuracy of the fused features. Attached Figure Description

[0055] Figure 1 This is a flowchart of the intrusion prevention and control monitoring method for track scenes based on sparse feature fusion, as described in this invention.

[0056] Figure 2 This is a schematic diagram of the point cloud and foreign objects acquired by the lidar in Embodiment 1 of the present invention.

[0057] Figure 3 This is a schematic diagram of the image and foreign object acquired by the camera in Embodiment 1 of the present invention. Detailed Implementation

[0058] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0059] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0060] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0061] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0062] Example 1 Reference Figure 1 In this example, a rail scene radar-vision fusion anti-invasion monitoring method based on sparse feature fusion is provided. The method includes: S1. Data acquisition and preprocessing: The lidar is used to acquire high-precision lidar data, i.e., three-dimensional point cloud data, in the rail scene; the high-definition camera is used to configure a high dynamic range (HDR) imaging module to acquire high-definition visible light image data in the rail scene, and perform data preprocessing, i.e., perform dynamic voxelization processing and contrast enhancement processing on the three-dimensional point cloud and visible light image respectively.

[0063] Preferably, performing dynamic voxelization processing and contrast enhancement processing on the three-dimensional point cloud and visible light image respectively specifically includes: performing dynamic voxelization processing on the three-dimensional point cloud data acquired by the lidar. The three-dimensional point cloud data graph acquired by the lidar is as shown in Figure 2 as follows: S11. Voxelize the three-dimensional point cloud data. Use the adaptive voxel discard strategy of SAFDNet. To balance long-distance sparsity and computational efficiency, adjust the voxel size dynamically according to the distance for a hierarchical voxel strategy: Short distance (d < 30m): Voxel size 0.1m, finely capture close-range details; Medium distance (30m ≤ d < 60m): Voxel size 0.2m, balance accuracy and efficiency; Long distance (d ≥ 60m): Voxel size 0.3m, reduce redundant long-distance calculations.

[0064] Generate voxels using the adaptive voxel discard StVD strategy of SAFDNet, dynamically adjust the voxel size according to the distance, and the calculation method is as follows: (1); In formula (1), is the distance from the point cloud in the scene to the lidar, is the voxel size.

[0065] Then calculate the centroid coordinates of the point cloud within the voxel, the average reflection intensity and the point cloud density , splice them to generate a voxel-level feature vector , , C is the feature dimension, N is the number of voxels, and R is the set of real numbers; among them, the centroid coordinates and point cloud density are geometric features, and the average reflection intensity is an intensity feature.

[0066] This voxel-level feature vector is denoted as where This refers to voxel features, which integrate the spatial location, density, and reflection intensity information of point clouds within a voxel, and can comprehensively characterize the features of a voxel. Here, C is the feature dimension and N is the number of voxels. The SAFDNet adaptive voxel discarding StVD strategy is adopted to filter low-density voxels at a distance and reduce redundant calculations. Preferably, low-density voxels are discarded when the number of points within the voxel is less than 3.

[0067] S12, noise reduction treatment; To suppress noise interference, a noise-resistant submanifold convolution NRConv is introduced to incorporate the voxel features of the lidar. voxel coordinates projected onto the image plane The input to the NRConv module suppresses noise and preserves effective features through 3D-2D joint convolution, yielding effective point cloud voxel features: (1); In formula (1), The resulting point cloud voxel features, or effective point cloud voxel features, are noise-resistant. Voxel-level feature vectors discard redundant spatial information from the point cloud while retaining key features such as centroid, density, and reflection intensity, reducing the data volume by approximately 70% and facilitating real-time processing by edge computing units. In this step, directly outputting the point cloud would not allow for efficient fusion with image features. Therefore, the point cloud needs to be converted into feature vectors with dimensions consistent with image features, enabling cross-modal association through a unified feature space.

[0068] Preferably, feature enhancement is performed on the visible light image data, wherein the visible light image acquired by the camera is as follows: Figure 3 As shown, specifically: S13. Contrast Enhancement: To improve feature quality under low-light conditions, adaptive histogram equalization and Retinex enhancement are performed sequentially on the image features to obtain the enhanced image. That is, valid image data: Adaptive Histogram Equalization: Enhances local details by limiting contrast through adaptive histogram equalization. Configuration parameters: Contrast threshold =3.0, image tile grid size =16×16.

[0069] The calculation formula is as follows: (2); In formula (2), For the input image, This is a visible light image after adaptive histogram equalization.

[0070] Retinex Enhancement: Combines multi-scale Retinex algorithms to enhance color realism; configuration parameters are Gaussian kernel standard deviation. =15; The final enhanced visible light image Effective image: (3); In formula (3), is the Gaussian kernel standard deviation.

[0071] S2. Parameter calibration and spatiotemporal synchronization: Spatiotemporal consistency of multiple sensors in orbital scenarios is a key guarantee for fusion accuracy. This system achieves millimeter-level spatial registration and sub-millisecond-level time synchronization between the LiDAR and the high-definition camera through dual modules of automatic extrinsic parameter calibration optimization and time alignment compensation, obtaining spatiotemporally aligned multimodal data.

[0072] Furthermore, S2 specifically includes: S21. Spatial Registration: Addressing the inherent differences between the LiDAR and camera coordinate systems, a three-step progressive calibration process is employed. Combined with high-quality images enhanced for contrast, SuperGlue is used for automatic extrinsic parameter calibration and optimization of the LiDAR and high-definition camera, achieving millimeter-level spatial registration. This includes: S211, Coarse LiDAR Pose Correction: The input for coarse LiDAR pose correction is 3D point cloud data collected by the LiDAR, which includes information such as 3D coordinates, reflection intensity, and timestamps. Due to sensor movement or dynamic changes in the scene, these point clouds may have inter-frame misalignment and motion distortion, such as point cloud shift caused by train vibration.

[0073] The process involves inputting 3D point cloud data, dynamically integrating multiple frames of point clouds using a continuous-time ICP algorithm, and then constructing a dynamic point cloud model using a linear iVox structure. The LiDAR pose is optimized by minimizing the distance between the scanned points and the model points. The output includes an optimized LiDAR pose transformation matrix, a dynamic point cloud model that filters out dynamic noise points while preserving static scene features such as stable point cloud clusters representing track and roadbed features, and finally, optimization of the LiDAR pose to eliminate sensor installation and motion errors. This provides an accurate point cloud reference for subsequent feature matching and fine registration. The resulting pose transformation matrix directly affects the accuracy of subsequent extrinsic parameter calibration, ensuring more precise spatial coordinate transformation between the LiDAR and camera, thereby guaranteeing accurate spatiotemporal synchronization and laying a reliable foundation for subsequent steps such as sparse feature fusion.

[0074] Preferably, the lidar pose is optimized by minimizing the distance between the scanning point and the model point, resulting in an optimized lidar pose transformation matrix that eliminates sensor installation and motion errors. The formula is as follows: (14); In formula (14), T represents the pose transformation matrix of the lidar, which is used to transform the point cloud from one coordinate system to another. The pose of the lidar is adjusted by optimizing this matrix. These are points in a 3D point cloud, i.e., scan points. , Represents the entire dynamic point cloud set. It is a projection function that projects 3D points, i.e., point cloud points, onto a specific model space. These are points in a dynamic point cloud model.

[0075] S212, Feature Matching and Initial Estimation: This step achieves initial alignment between the LiDAR and camera coordinate systems by performing cross-modal feature association between the LiDAR point cloud data and camera images. The specific process is as follows: First, the centroid coordinates and average reflection intensity of the point cloud within the effective point cloud voxel feature voxels are extracted. The centroid coordinates represent the spatial position, and the average reflection intensity represents the echo characteristics. The projection model is selected according to the field of view (FoV) of the lidar: pinhole projection is used when FoV is less than 150°, and panoramic projection is used when FoV is ≥ 150°. The three-dimensional centroid coordinates are mapped to a two-dimensional grayscale image according to the reflection intensity—the pixel value corresponds to the reflection intensity of the point cloud. At the same time, a point index map is generated to quickly query the three-dimensional coordinates corresponding to each pixel. Then, when generating the intensity image, the centroid coordinates are projected onto the image plane, and the average reflection intensity of the voxels is used as the pixel value to generate a two-dimensional intensity image; After generating a 2D intensity image of the LiDAR, the SuperPoint algorithm is used to detect key points in both the LiDAR intensity image and the camera image, and a local descriptor is generated for each key point. The SuperPoint algorithm automatically identifies locations with significant geometric features through a deep learning model, such as track edge inflection points and corner points of foreign object contours. Figure 2 , Figure 3 As shown in the red box, the descriptor includes gradient and texture feature information around the keypoint, ensuring high similarity of descriptors for the same physical point in cross-modal images. The above process converts the 3D point cloud features of the LiDAR into a 2D visualized image, providing a cross-modal input basis for subsequent SuperGlue feature matching. The quality of the generated intensity image directly affects the keypoint detection accuracy, thus determining the reliability of the initial alignment between the LiDAR and camera coordinate systems. That is, the centroid coordinates in the voxel features. It is the aggregated result of the spatial positions of all point clouds within a voxel, and can be regarded as the "representative point" of that voxel. When generating the intensity image, these centroid coordinates need to be projected onto the image plane, and the average reflection intensity of the voxel is used. As pixel values, a two-dimensional intensity image is formed.

[0076] Subsequently, a Graph Neural Network (GNN) is used to match keypoint descriptors between the two modalities. The GNN treats keypoints as nodes in a graph and the relationships between keypoints as edges. Through a multi-layer message passing mechanism, it learns the association features between nodes and then calculates the matching probability between the LiDAR and camera keypoint descriptors. A matching threshold of 0.05 is set, retaining only point pairs with matching probabilities higher than this threshold to filter low-reliability matches and ensure spatial consistency of corresponding point pairs. This embodiment automatically learns the mapping relationship of cross-modal features through a deep learning model, improving matching accuracy by more than 30% in complex scenarios such as low illumination and low reflectivity compared to traditional handcrafted features (such as SIFT and ORB). The output keypoint matching pairs are directly used as input to the rotation RANSAC algorithm, and their matching accuracy determines the reliability of the initial rotation matrix.

[0077] Furthermore, an initial rotation matrix estimation is performed between the lidar and camera coordinate systems. The initial estimation object is the rotation matrix from the lidar coordinate system to the camera coordinate system. This method describes the attitude transformation relationship between two sensors in space. The core objective is to quickly solve for the initial rotation relationship between the two coordinate systems using cross-modal correspondence points provided by feature matching pairs. This provides an iterative starting point for subsequent fine-grained registration and NID optimization, preventing the optimization process from getting trapped in local optima. Based on keypoint matching pairs, the RANSAC algorithm is used for initial estimation, estimating the rotation matrix from the LiDAR coordinate system to the camera coordinate system. Using the Cauchy kernel function Construct an error function to quantify the deviation between the coordinates of the LiDAR points projected onto the camera coordinate system and the actual camera point coordinates: (4); In formula (4), The Cauchy kernel function is used for robust estimation, reducing the impact of outliers on the calculation results. It is the rotation matrix from the lidar coordinate system to the camera coordinate system, which is continuously optimized during calculation to find the optimal rotation relationship. This is the optimized rotation matrix, obtained by minimizing the error in subsequent formulas. Indicates the first The coordinates of a lidar point in its own coordinate system Indicates the first The coordinates of each LiDAR point in the camera coordinate system are M, where M is the number of corresponding point pairs involved in the calculation. This formula calculates the distance error between the LiDAR point after rotation matrix transformation and projection and the corresponding point in the camera coordinate system. It then uses the Cauchy kernel function for robust estimation to find the rotation matrix that minimizes the error. This formula calculates the distance error between the LiDAR point after rotation matrix transformation and projection and the corresponding point in the camera coordinate system. It then uses the Cauchy kernel function for robust estimation to find the rotation matrix that minimizes the error.

[0078] S213, Fine-grained registration, i.e., NID optimization; The fine registration step uses feature matching and the rotation matrix obtained from the initial estimation. Based on this, the effective point cloud voxel features are first derived using the optimal rotation matrix. The corresponding point cloud coordinates are projected onto the camera image plane. The nearest point corresponding to each pixel is retained through a depth buffer test, and occluded invalid points are filtered out to output a set of visible point clouds. (including three-dimensional coordinates) (and reflection intensity), this operation reduces the amount of point cloud data by about 40%, focusing on the effective area that can be matched with the camera image.

[0079] Then, with visible point clouds and image enhancement Taking semantic features (including orbital boundaries, object outlines, etc.) as input, the normalized information distance is calculated using the following formula: (5); In formula (5), For entropy, For mutual information; This is the lidar point cloud after visible point removal, retaining only the visible points from the camera's view, including 3D coordinates. Based on the reflection intensity, occluded invalid points are filtered out, focusing on the effective area that matches the image features.

[0080] Finally, the Nelder-Mead optimization algorithm is employed, with the NID value as the optimization objective, to iteratively adjust the rotation matrix. Translation vector The initial extrinsic parameter matrix is ​​used until the NID value converges to its minimum, and the optimal extrinsic parameter matrix for achieving millimeter-level spatial registration is output. The extrinsic parameter matrix includes the rotation matrix. Translation vector .

[0081] S22. Time alignment compensation, also known as synchronous triggering compensation, such as Kalman filtering; Automatic extrinsic calibration based on SuperGlue solves the problem of spatial coordinate system alignment between LiDAR and camera, while temporal alignment compensation compensates for the time sampling difference between the two. Together, they form a complete closed loop of "spatiotemporal synchronization". Spatial registration accuracy is the basis of temporal alignment. If the spatial coordinate system is not aligned, even if the time synchronization is accurate, cross-modal features will still fail to fuse due to spatial deviation.

[0082] The inputs to time alignment compensation are the extrinsic parameter matrix output by "Automatic Extrinsic Parameter Calibration Based on SuperGlue" and the original timestamps of multiple sensors. The extrinsic parameter matrix provides a reference for spatial coordinate transformation for time alignment. The original timestamps of multiple sensors are the hardware timestamps collected by the LiDAR and the camera, reflecting the differences in the acquisition time of the sensor data. The outputs are the synchronized timestamps and the compensated multimodal data.

[0083] Synchronization timestamps are achieved through hardware triggering and Kalman filtering, reducing multi-sensor timestamp errors to <1ms; compensated multimodal data refers to LiDAR point clouds and camera images with unified timestamps, which serve as input for "view perception query generation".

[0084] The automatic extrinsic calibration based on SuperGlue achieves spatial alignment through "dynamic point cloud preprocessing - feature matching - fine registration". Temporal alignment compensation is based on the spatial alignment result and achieves temporal alignment through "hardware-triggered coarse synchronization - Kalman filter fine compensation". Finally, it outputs spatiotemporally aligned multimodal data, providing a spatiotemporally consistent data source for subsequent feature fusion.

[0085] Specifically, the time alignment achieved through "hardware-triggered coarse synchronization - Kalman filter fine compensation" includes: S221, Hardware-triggered coarse synchronization: First, a preliminary unification of the time base is achieved through hardware-triggered coarse synchronization. Specifically, the LiDAR and the camera are connected using a hardware synchronization trigger signal line. When the sensor collects data, a synchronization trigger signal is sent, which initially reduces the original timestamp error between the two from the 500μs level to the 1ms level, laying the foundation for subsequent fine compensation.

[0086] S222, Kalman filter fine compensation: After hardware coarse compensation, Kalman filtering fine compensation is performed. A first-order Kalman filter model is used to iteratively smooth the residual time error. The core operation is to iteratively smooth the residual time error based on the Kalman filter model, so that the timestamp error of the multi-sensor is stabilized at <1ms, ensuring that the LiDAR point cloud and the camera image are strictly synchronized in the time dimension.

[0087] The formula for predicting timestamps using a first-order Kalman filter is as follows: (6); In formula (6), This represents the synchronized timestamp predicted at time k, which is a weighted fusion result of the current measurement and historical estimates. This is the sensor's raw timestamp at time k, directly acquired from the hardware clock of the LiDAR or camera. This is the synchronized timestamp predicted at time k-1, which is the time estimate of the previous time. =0.95 is the filter coefficient, used to balance current measurements with historical estimates. This is the frame interval, typically 0.1 seconds. It represents the current sensor's original timestamp. Synchronization timestamp predicted from the previous moment By performing weighted fusion and iterative adjustments, the timestamp error of the multiple sensors was eventually stabilized at <1ms.

[0088] The two operations, S221 and S222, are progressive. Hardware-triggered coarse synchronization solves the consistency problem of the time reference and provides reliable initial conditions for Kalman filter fine compensation. Kalman filter fine compensation further eliminates the time offset caused by the difference in sensor sampling frequency through dynamic prediction and error correction. Together, they form a complete closed loop of "coarse synchronization-fine compensation", ensuring that the LiDAR point cloud and the camera image are strictly synchronized in the time dimension, providing a spatiotemporally consistent data source for subsequent feature fusion.

[0089] S3. Sparse feature fusion: Based on spatiotemporally aligned multimodal data, camera semantic features and lidar geometric features are fused to obtain fused features. To address the challenge of sparse features in orbital scenarios, this embodiment designs a three-level processing architecture of view-aware query generation, adaptive feature diffusion, and uncertainty-aware fusion to achieve efficient complementarity and noise suppression of cross-modal features.

[0090] After spatiotemporal synchronization registration and synchronization trigger compensation, the LiDAR and camera data have achieved sub-millisecond time synchronization and millimeter-level spatial registration, forming spatiotemporally aligned multimodal data. Based on this, the viewpoint-aware query generation stage begins, where a lightweight detection model establishes a direct correlation between the 2D semantic features of the camera images and the 3D geometric features of the LiDAR. Sparse feature fusion is based on the spatiotemporal synchronization and calibration of LiDAR and high-definition camera, which are closely linked through spatiotemporal alignment accuracy and coordinate transformation logic. The spatiotemporal synchronization and calibration process provides sub-millisecond-level time synchronization (timestamp error <1ms) and millimeter-level spatial registration (extrinsic parameter matrix accuracy <5mm) for sparse feature fusion, ensuring that the LiDAR point cloud and camera image are aligned in the temporal dimension and have a unified coordinate system in the spatial dimension. The extrinsic parameter matrix obtained from the calibration is the rotation matrix. With translation vector It is also the core basis for operations such as three-dimensional coordinate transformation, laying a spatiotemporal benchmark for cross-modal feature fusion.

[0091] Furthermore, S3 specifically includes: S31, View Perception Query Generation, i.e., Lightweight View Detection; The view perception query generation adopts lightweight view detection, takes spatiotemporally aligned multimodal data with time alignment compensation as input, and outputs the three-dimensional coordinates of the object in the lidar coordinate system and the corresponding camera semantic features through 2D detection, monocular 3D detection and three-dimensional coordinate transformation, providing an accurate cross-modal data foundation for sparse feature fusion.

[0092] 2D detection aims to quickly locate foreign objects in the image plane using a lightweight FCOS detector; monocular 3D detection infers the three-dimensional position of the foreign object in space based on the 2D results; 3D coordinate transformation uses an external parameter matrix to unify the monocular 3D coordinates to the lidar coordinate system, ensuring consistency of the spatial reference of multimodal data, thereby realizing the initial mapping from image semantics to spatial geometry, and preparing for subsequent high-precision fusion and alarm decision-making.

[0093] The viewpoint-aware query generation specifically includes: S311, 2D inspection; Using a lightweight FCOS detector, the two-dimensional bounding box of foreign objects is quickly detected based on spatiotemporally aligned multimodal data, enabling preliminary localization of foreign objects on the image plane. The output includes 2D detection results containing category and confidence, providing a semantic feature basis for subsequent 3D localization.

[0094] S312, Monocular 3D Inspection; Based on the 2D detection results, combined with camera intrinsic parameters and depth estimation models, the three-dimensional spatial location information of the foreign object is deduced from the monocular image, and the three-dimensional outline of the foreign object in space is initially constructed, extending the 2D information of the foreign object to 3D space.

[0095] S313, Three-dimensional coordinate transformation; Based on the optimal extrinsic parameter matrix obtained from calibration, the coordinates obtained from monocular 3D detection are transformed from the camera coordinate system to the lidar coordinate system, achieving a unified expression of multimodal data under the same spatial reference. The 2D coordinates are converted into 3D spatial points using the camera's intrinsic and extrinsic parameter matrices, and the intrinsic parameter inverse matrix is ​​used to... Image coordinates The depth d is converted to normalized coordinates in the camera coordinate system, and then converted to the inverse extrinsic matrix. After removing the influence of camera extrinsic parameters, the three-dimensional coordinates of the target in the camera coordinate system are finally obtained. The formula is as follows: (7); In formula (7), The target's three-dimensional coordinates in the lidar coordinate system. The inverse matrix of the camera's extrinsic parameters. The inverse matrix of the camera's intrinsic parameters. , where are the image coordinates of the center point of the 2D detection box, and d is the target depth value predicted by monocular 3D detection, output by monocular 3D detection.

[0096] It is the inverse of the camera extrinsic matrix, used to transform points from the world coordinate system to the camera coordinate system. This is the inverse of the camera intrinsic matrix, used to convert image plane coordinates into normalized coordinates in the camera coordinate system. , where are the image coordinates of the center point of the 2D detection box, and d is the target depth value predicted by monocular 3D detection, output from the monocular 3D detection step.

[0097] The coordinates are then transformed to the lidar coordinate system using the calibrated optimal extrinsic parameter matrix, achieving cross-modal data spatial alignment. This step establishes a direct correlation between the semantic features of the image features and the geometric features of the point cloud. Through this transformation, the system achieves a mapping from "image semantic localization" to "real space localization," ensuring the consistency of the physical meaning of the subsequently fused features.

[0098] S32, Dynamic Feature Diffusion; After completing the lightweight view detection, the system proceeds to the dynamic feature diffusion step. Using the 3D coordinates of the object in the LiDAR coordinate system and the corresponding camera semantic features as input, the system dynamically adjusts the diffusion radius and sparse convolution operations to obtain the diffused dense point cloud features. This dynamic adjustment of the diffusion radius and sparse convolution operations addresses the feature discontinuity problem caused by the sparse LiDAR point cloud, providing a more complete cross-modal feature representation for subsequent uncertainty perception fusion. Due to the non-uniform distribution of sparse point clouds, this system designs a dynamic feature diffusion mechanism to enhance feature propagation capabilities. This mechanism consists of two progressive feature enhancement steps: diffusion range calculation and 3D sparse convolution. First, the spatial range of feature diffusion is dynamically determined based on the target's physical size and distance to address the problem of insufficient feature coverage due to sparse point clouds at long distances. Then, by performing sparse convolution operations within this range, the features of the center point are propagated to neighboring voxels, achieving "local feature densification." Finally, the system outputs dense point cloud features containing spatial context information.

[0099] Furthermore, the dynamic feature diffusion specifically includes: S321. Calculation of diffusion range; The diffusion radius is dynamically adjusted based on the target size and distance, using the 3D coordinates of the foreign object in the lidar coordinate system generated by the view-perception query and the corresponding camera semantic features as input. The calculation formula is as follows: (7); In formula (7),, The feature diffusion radius of the i-th target determines the neighborhood range of the 3D sparse convolution. The average size of the target is used to ensure that the diffusion range matches the physical size of the target, with an exponential decay coefficient. This automatically shrinks the diffusion range of distant targets, reducing redundant calculations. d represents the straight-line distance between the target and the lidar, which comes from the three-dimensional coordinate transformation result in the step, i.e., the Z-axis coordinate component in the lidar coordinate system. S322, 3D sparse convolution: The sparse point cloud features are diffused using a 3×3×3 sparse convolution, as follows: (8); In formula (8), The characteristics of the dense point cloud after diffusion. The input is sparse point cloud features, which are effective point cloud voxel features. It is obtained through the spatial mapping relationship of S313. Indicates the dynamic diffusion radius, ⊕ indicates feature splicing. The operation is a 3×3×3 sparse convolution. While preserving the original sparsity, it diffuses the features of the center point to the neighboring voxels, thus solving the problem of feature loss caused by the sparsity of distant point clouds.

[0100] S33, Modal uncertainty quantification; Based on the diffused dense point cloud features and camera semantic features, the fusion weights are dynamically allocated by quantifying the localization uncertainty of the two types of sensors, and finally the fused features are output. .

[0101] Steps S32 and S33 are closely linked through a feature transfer and fusion strategy: Dynamic feature diffusion, through adaptive diffusion radius calculation and 3D sparse convolution calculation, diffuses the sparse point cloud features of the LiDAR (such as the geometric features of orbital obstructions) to neighboring voxels, generating dense point cloud features containing spatial context information. It also preserves the semantic associations corresponding to the three-dimensional coordinates of the dynamic feature diffusion output, such as the foreign object category.

[0102] Step modal uncertainty quantization is based on the dense point cloud features output by the step dynamic feature diffusion. By combining the camera semantic features after the transformation of the viewpoint detection in the lightweight step, the localization uncertainty of the two types of sensors is quantified, such as the distance between the camera detection box and the ground truth box, and the spatial distribution error of the LiDAR point cloud. Then, the fusion weights are dynamically assigned, and the fused features are finally output. .

[0103] Furthermore, to address multi-sensor noise interference, a confidence-driven dynamic fusion strategy is designed, wherein the modal uncertainty quantification specifically includes: S331. Uncertainty Modeling: The core objective of uncertainty modeling is to quantify the positioning uncertainty of the camera and lidar, providing a confidence basis for dynamic weighted fusion.

[0104] The object of lidar modal modeling is the diffused lidar point cloud features output by the dynamic feature diffusion step. The spatial distance between the point cloud geometric features and the target's true location (BEV plane Euclidean distance) The system assesses the reliability of localization. The input is the dense point cloud features output from the step-by-step dynamic feature diffusion; the ground truth bounding box coordinates B are derived from labeled data; and the output is the LiDAR modal confidence weights. The value ranges from [0,1], and the larger the value, the more reliable the positioning.

[0105] The lidar modes are as follows: (9); In formula (9), For the confidence weight of lidar, Features of lidar This represents the Euclidean distance of the lidar point cloud on the BEV (bird's-eye view) plane. Let B be the regression function for camera features, and B be the coordinates of the ground truth bounding box. Camera modal modeling is as follows: (10); In formula (10), This represents the confidence weight of the camera mode, with a value range of [0,1]. A larger value indicates that the camera positioning is more reliable. This is a regression function for camera features, which converts camera semantic features into predicted bounding box coordinates. The camera semantic features derived from the step-lightweight viewpoint detection include information such as the 2D detection bounding box, category, and confidence score of the foreign object. B represents the coordinates of the ground truth bounding box, derived from the labeled data or the baseline coordinates of the preceding detection results. This represents the Euclidean distance of the LiDAR point cloud on the BEV bird's-eye view plane. It is the Euclidean distance on the BEV plane based on the predicted bounding box and the ground truth bounding box. To quantify the uncertainty of positioning, the closer the distance, the higher the confidence level.

[0106] S332, Weighted Fusion: The core objective of weighted fusion is to dynamically allocate weights based on sensor confidence levels, fuse camera semantic features and LiDAR geometric features, and improve detection robustness.

[0107] The input is camera semantic features. and lidar features Weights are dynamically assigned based on sensor confidence levels, and camera semantic features and LiDAR geometric features are fused together, with camera features as the input. and lidar features : (11); In formula (11), To achieve feature fusion, semantic features (such as the "foreign object" label) and geometric features (such as 3D dimensions and location) are integrated. The next step in weighted fusion is to... Input Steps: Center Heatmap Generation: Center heatmap generation involves generating a target heatmap using a sparse detection head and combining it with a dynamic safety distance model to achieve tiered early warning. This invention dynamically allocates weights based on the sensor's current confidence level. In low-reflectivity object detection scenarios, the camera's contribution weight can be increased to 65%, significantly enhancing detection robustness.

[0108] S4. Intrusion Detection and Early Warning: After completing the multimodal feature fusion, a target heatmap is generated through a sparse detection head, and a hierarchical early warning is achieved by combining a dynamic safety distance model, ensuring that protection commands are output within 50ms.

[0109] Furthermore, S4 specifically involves: fusing features Target heatmaps are generated using sparse detectors, and graded early warnings are achieved by combining them with a dynamic safety distance model. S41, Central heat map generation; To address the challenges of perceiving sparse features, this invention designs a center heatmap generation algorithm based on a dynamic Gaussian kernel. The input consists of fused feature voxels output by step modal uncertainty quantization and target physical size parameters obtained through step-lightweight perspective detection-dynamic feature diffusion. The output is a two-dimensional center heatmap. Its pixel value range is [0,1], which represents the probability density of the corresponding position being the geometric center of the target.

[0110] First, an algorithm for generating center heatmaps based on dynamic Gaussian kernels is designed, and the Gaussian kernel parameters are dynamically adjusted: The input consists of the target width W and length L obtained from monocular 3D detection and feature diffusion; the output is the Gaussian kernel standard deviation. Among them, W and L depend on the spatiotemporally synchronized extrinsic parameter matrix and the S313-S332 cross-modal feature fusion results; Calculate the Gaussian kernel standard deviation based on the target physical dimensions: (12); In formula (12), and The design sets the target width and length; this design matches the Gaussian kernel coverage to the actual size of the target, avoiding the blurring or oversharpening problems caused by traditional fixed kernel parameters.

[0111] Then, a center heatmap generation algorithm based on dynamic Gaussian kernels is used to generate the heatmap, resulting in the center heatmap. Abstract fusion features The probability distribution is converted into a center heatmap, which is a two-dimensional heatmap with pixel values ​​ranging from [0,1]. Each pixel value represents the probability that the corresponding location is the center of the target.

[0112] S42, Tiered early warning strategy; Based on the dynamic relationship between the actual distance between the target and the track and the train's operating speed, combined with the probability distribution of the target's location in the central heat map, a tiered early warning system is implemented using a preset safe distance threshold. Dynamic safety distance: safe distance (13); In formula (13), The safety protection threshold is dynamically adjustable based on the train's real-time speed at different speeds. The train speed V is input in real time from the track signal system. The model comprehensively considers the train braking distance and emergency avoidance space to ensure that the safety protection threshold is dynamically adjustable at different speeds. The train speed V is input in real time from the track signal system, forming a cross-system data linkage with the acquired data and data processing.

[0113] The three-level early warning thresholds set in this embodiment are as follows: Yellow alert: Target distance <20m and speed >50km / h (approximately 13.9m / s) → Trigger audio-visual alert to remind monitoring personnel to pay attention; Orange alert: Target distance <10m and speed >30km / h (approximately 8.3m / s) → Activate the braking system to precharge air and shorten subsequent braking response time; Red alert: Target distance <5m → Directly trigger emergency braking to force the train to stop to avoid collision.

[0114] The above three warning thresholds are set based on a multiple relationship of the safe distance, namely, yellow warning threshold = 4 × safe distance = 20m, orange warning threshold = 2 × safe distance = 10m, and red warning threshold = safe distance = 5m. .

[0115] For example, when the train is running at 100km / h (approximately 27.8m / s): the safe distance = max(5m, 0.1×27.8+0.5) = 3.28m, take 5m; the orange warning threshold is set to 10m, at which time the system starts the brake pre-charge air to ensure that the train has (10-5) / 27.8≈0.18s of braking preparation time at a speed of 27.8m / s; the yellow warning threshold is set to 20m, providing the monitoring personnel with (20-5) / 27.8≈0.54s of response time.

[0116] Target distance is determined by the following method: high-response areas generated from the central heatmap (e.g., The target position is extracted from the LiDAR coordinate system (>0.7), and the 3D coordinates in the LiDAR coordinate system output by the lightweight view detection are combined and spatially mapped to the orbital plane for calculation. Specifically, millimeter-level spatiotemporal synchronization accuracy is achieved by using automatic extrinsic parameter calibration technology based on SuperGlue to ensure accurate spatial mapping between the 3D coordinates and the heat map position, thereby ensuring the reliability of the target distance calculation.

[0117] The generated center heatmap provides a probability distribution of target locations for multi-threshold decision-making, replacing the traditional "single-point detection" mode. It improves the robustness of detection in complex scenes (such as partially occluded targets) through probability density weighting. Multi-threshold decision-making then transforms the spatial information in the heatmap into executable safety instructions, serving as the final link in the "feature fusion-localization-decision" chain. Center Heatmap It is the core input of the graded early warning strategy. The high-response areas in the heat map are mapped in three-dimensional space and the actual risk level is calculated in combination with the train speed.

[0118] Example 2 This embodiment provides a radar-visual fusion intrusion prevention and control monitoring system for track scenes based on sparse feature fusion. The system includes a hardware module and a software module. The hardware modules include: a heterogeneous sensor combination of LiDAR and high-definition camera, and an edge computing unit. LiDAR: Deployed on both sides of the track to acquire high-precision 3D point cloud data, with a maximum effective detection range of 200 meters; HD camera: Equipped with a high dynamic range imaging module, a resolution of 2560×1440, and covering a 120° field of view; Edge computing unit: Integrates a high-performance GPU to support real-time processing and fusion computing of sensor data; The software modules include five core modules: data acquisition, data preprocessing, spatiotemporal synchronization and calibration, sparse feature fusion, and intrusion detection and early warning. The system comprises the following modules: a data acquisition module, which acquires relevant data collected by the hardware module; a data preprocessing module, which receives data from the data acquisition module and performs preprocessing; a spatiotemporal synchronization and calibration module, which ensures that the timestamp error of multiple sensors is <1ms through hardware triggering and Kalman filtering compensation, and further compensates with Kalman filtering to achieve a final timestamp error of <0.5ms; a sparse feature fusion module, which performs hierarchical extraction of LiDAR voxel features and camera image features respectively, and combines sparse convolution and uncertainty perception mechanisms to achieve cross-modal feature complementarity; and an intrusion detection and early warning module, which generates a center heatmap based on a sparse detection head and outputs protection commands in conjunction with a hierarchical early warning strategy.

[0119] The implementation steps of a radar-visual fusion intrusion prevention and control monitoring system for track scenes based on sparse feature fusion are as follows: (a) Hardware deployment and parameter configuration; LiDAR and high-definition cameras were symmetrically installed on both sides of the track to be tested. The voxelization parameters of the LiDAR were configured as follows: voxel size 0.1m for close range (d<30m), 0.2m for medium range (30m≤d<60m), and 0.3m for long range (d≥60m). The adaptive voxel drop strategy of SAFDNet was enabled, and the maximum detection distance was 200 meters. The resolution of the high-definition camera was 2560×1440, and the mode was HDR.

[0120] The edge computing unit uses a domestically developed platform and integrates a high-performance GPU to support real-time processing and fusion computing of sensor data. (ii) Acquisition and preprocessing of multi-sensor data; The data acquisition module is responsible for acquiring relevant data collected by the hardware module; then, the data preprocessing module is responsible for receiving the data from the data acquisition module and performing preprocessing, specifically including: LiDAR processing: Noise-resistant submanifold convolution NRConv is used to denoise voxel features. Noise is suppressed by 3D-2D joint convolution, retaining 92% of the effective features. After dynamic voxelization, the point cloud density is increased by 35% compared with traditional methods.

[0121] Camera processing: Four layers of feature maps (resolution 1 / 4 to 1 / 32) were extracted using the Swin Transformer, and adaptive histogram equalization (clip limit=3.0, tile grid size=16×16) and Retinex enhancement (sigma=15) were performed sequentially, improving the feature quality by 40% under low light conditions.

[0122] (III) Spatiotemporal synchronization and calibration; External parameter calibration: The spatiotemporal synchronization and calibration module adopts an automatic calibration algorithm based on SuperGlue, which improves the point cloud density through dynamic point cloud integration technology and achieves millimeter-level spatial registration by combining NID optimization, with an average error of <5mm; Time alignment: Hardware triggering reduces the time error to within 1ms, and further compensation is achieved by combining Kalman filtering 5 with a filter coefficient of α=0.9, resulting in a final timestamp error of <0.5ms.

[0123] (iv) Sparse feature fusion and detection; The sparse feature fusion module includes: view-aware query generation, dynamic feature diffusion, and uncertainty-aware fusion; Viewpoint-aware query generation: 2D image coordinates are converted into 3D spatial points using the FCOS3D monocular detection algorithm, with a mean absolute error of depth estimation of <0.3m; Dynamic feature diffusion: The diffusion radius is dynamically adjusted based on target size and distance, and feature propagation is enhanced through 3×3×3 sparse convolution; Uncertainty-aware fusion: The fusion weights of the camera and LiDAR are dynamically allocated according to confidence levels, increasing the camera weight to 65% in low-reflectivity scenes and improving the detection rate by 2.8%; (v) Intrusion detection and early warning: The sparse detection head generates a dynamic Gaussian kernel heat map, which is combined with a graded early warning strategy for judgment.

[0124] Example 3 This embodiment provides a low-light condition sensing scheme for a track scene radar-visual fusion intrusion prevention monitoring system based on sparse feature fusion, which differs from Embodiment 2 in that: (a) The difference between hardware deployment and parameter configuration lies in; The deployment parameters of the lidar and camera are the same as in Example 1. The test environment is a twilight scene with an illumination intensity of 50 lux. The camera is set to HDR mode and equipped with an infrared fill light module to enhance low-light imaging.

[0125] (ii) The difference between the acquisition and preprocessing of multi-sensor data lies in: Camera processing: Adaptive histogram equalization and Retinex enhancement are adopted, which improves contrast by 50% and reduces noise by 30%; edge-preserving convolution is introduced to enhance target edge features and improve edge localization accuracy by 25%.

[0126] LiDAR processing: Mid-range voxelization (0.2m) combined with NRConv effectively suppresses rain noise, and the effective feature retention rate of point cloud reaches 90%.

[0127] (iii) The difference between spatiotemporal synchronization and calibration lies in: External parameter calibration: The feature matching algorithm based on SuperGlue still maintains a matching accuracy of 95% under low light conditions, and the spatial registration error after NID optimization is <3mm; Uncertainty perception fusion: The camera confidence weight is dynamically adjusted to 55%, and the LiDAR weight is 45%, which improves the detection robustness in low light scenes by 30%.

[0128] (iv) Sparse feature fusion and detection and (v) Intrusion detection and early warning are consistent with Example 2.

[0129] Example 4 This embodiment provides a radar-visual fusion detection scheme for low reflectivity objects. The specific implementation steps differ from those in Embodiment 2 in that: (a) The difference between hardware deployment and parameter configuration lies in; The parameters of the lidar and camera are the same as in Example 2. The test target is a black plastic board (reflectivity <5%) at a distance of 80 meters. The camera is set to high gain mode, and the lidar is set to close-range voxelization (0.1m) to enhance detail capture.

[0130] (ii) The difference between the acquisition and preprocessing of multi-sensor data lies in: Camera processing: After multi-scale feature extraction, edge features are enhanced by a depth gradient enhancement module, increasing the edge gradient magnitude by 40%; LiDAR processing: Near-range voxelization combined with dynamic feature diffusion improves point cloud feature coverage by 20%; Uncertainty perception fusion: Camera weight is increased to 70%, LiDAR weight is 30%, and the feature response value of low reflectivity targets is improved by 35%.

[0131] (iii) Spatiotemporal synchronization and calibration are the same as in Example 2; (iv) The difference between sparse feature fusion and detection lies in; When the sparse detection head generates a heatmap, the standard deviation of the dynamic Gaussian kernel is set to 0.5 (the target size is 0.8m×0.5m), and the heatmap threshold is 0.7. Detection results: The black plastic board is accurately identified at 80 meters with a detection rate of 95.8%, which is 17.5% higher than the traditional method. The system inference time meets the real-time requirements.

[0132] (v) The intrusion detection and early warning are consistent with Example 2; This embodiment significantly improves the detection capability of low-reflectivity objects through dynamic weight allocation and feature enhancement, providing an effective solution for monitoring complex targets in orbital scenarios.

[0133] Example 5 This embodiment also provides a computer-readable storage medium storing executable instructions, which, when executed, cause the machine to perform the above-described method for anti-intrusion monitoring of orbital scenes based on sparse feature fusion.

[0134] Specifically, a system or apparatus equipped with a readable storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer or processor of the system or apparatus can read and execute the instructions stored in the readable storage medium.

[0135] In this case, the program code itself, which can be read from the readable medium, can perform the functions of any of the above embodiments, and therefore the computer-readable code and the readable storage medium storing the computer-readable code constitute a part of this specification.

[0136] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer or the cloud via a communication network.

[0137] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A sparse feature fusion-based track scene radar and visual fusion intrusion monitoring method, characterized in that, The method comprises: S1, a laser radar acquires three-dimensional point cloud data and a high-definition camera acquires visible light image data, and dynamic voxelization processing and contrast enhancement processing are respectively performed on the three-dimensional point cloud and the visible light image; S2, automatic calibration and optimization of external parameters of the laser radar and the high-definition camera and time alignment compensation are performed, millimeter-level spatial registration and sub-millisecond-level time synchronization of the laser radar and the high-definition camera are realized, and multi-modal data in space-time alignment is obtained; S3, based on the multi-modal data in space-time alignment, camera semantic features and laser radar geometric features are fused to obtain fused features; S4, after completing the multi-modal feature fusion, a target heat map is generated through a sparse detection head, and hierarchical early warning is realized in combination with a dynamic safety distance model. 2.The rail scene radar and visual fusion intrusion monitoring method based on sparse feature fusion according to claim 1, characterized in that, The dynamic voxelization processing and the contrast enhancement processing on the three-dimensional point cloud and the visible light image respectively comprise: S11, voxelizing the three-dimensional point cloud data, using an adaptive voxel discarding strategy of SAFDNet, and dynamically adjusting the voxel size according to the distance to perform hierarchical voxelization strategy: near distance d<30m: voxel size 0.1m; medium distance 30m≤d<60m: voxel size 0.2m; far distance d≥60m: voxel size 0.3m; Then the centroid coordinates, average reflection intensity and point cloud density of the point cloud in the voxel are calculated to generate a voxel-level feature vector by splicing , , C is the feature dimension, N is the number of voxels, and R is the real set; S12, introduce noise-resistant submanifold convolution NRConv, laser radar voxel features with the voxel coordinates projected to the image plane input, suppress noise by 3D-2D joint convolution, retain effective features, and obtain effective point cloud voxel features: （1）； S13, sequentially performing adaptive histogram equalization and Retinex enhancement on the visible light image to obtain an enhanced image i.e. valid image data: Adaptive histogram equalization enhances local details through adaptive histogram equalization of limiting contrast: （2）； In formula (2), is the input visible light image, is the image after adaptive histogram equalization processing; Then Retinex enhancement is performed with the parameter of Gaussian kernel standard deviation = 15, the final enhanced image i.e. the effective image: （3）； In equation (3), is the input visible light image, is the standard deviation of the Gaussian kernel.

3. The rail scene radar and vision fusion intrusion monitoring method based on sparse feature fusion according to claim 2, characterized in that, The S2 specifically comprises: S21, spatial registration: based on SuperGlue, automatic calibration and optimization of external parameters of the laser radar and the high-definition camera are performed to realize millimeter-level spatial registration, comprising: S211, rough correction of laser radar pose; input three-dimensional point cloud data, dynamically integrate multiple frames of point cloud through continuous time ICP algorithm, then adopt linear iVox structure to construct a dynamic point cloud model, optimize the laser radar pose by minimizing the distance between the scanning points and the model points, output the optimized laser radar pose transformation matrix, and filter the dynamic noise points and retain the static scene features of the dynamic point cloud model; S212, feature matching and initial estimation: first, extract the centroid coordinates and average reflection intensity of the effective point cloud voxel, select a projection model according to the field of view FoV of the laser radar, map the three-dimensional centroid coordinates to a two-dimensional gray image according to the reflection intensity, the pixel value in the two-dimensional gray image corresponds to the reflection intensity of the point cloud, and a point index map is generated to quickly query the three-dimensional coordinates corresponding to each pixel; project the centroid coordinates to the image plane, and generate a two-dimensional intensity image by taking the average reflection intensity of the voxel as the pixel value; after generating the laser radar two-dimensional intensity image, use the SuperPoint algorithm to detect the key points in the laser radar intensity image and the camera image respectively, and generate a local descriptor for each key point; then, the key point descriptors of the two modal images are matched through a graph neural network GNN, the key points are regarded as nodes of the graph, the relationship between the key points is regarded as edges, the correlation features between the nodes are learned through a multi-layer message passing mechanism, and then the matching probability of the key point descriptors of the laser radar and the camera is calculated, and the key point matching pairs with a matching probability higher than a preset threshold are retained; Based on the key point matching pair, an initial estimation is made by a RANSAC algorithm, and it is estimated that the rotation matrix of the object is from the laser radar coordinate system to the camera coordinate system : （4）； In formula (4), is a Cauchy kernel function, is a rotation matrix from the LiDAR coordinate system to the camera coordinate system, is the optimized optimal rotation matrix, represents the coordinates of the th LiDAR point in its own coordinate system, represents the point coordinates of the th LiDAR point in the camera coordinate system, and M is the number of corresponding point pairs participating in the calculation. Robust estimation is performed using the Cauchy kernel function to find the rotation matrix that minimizes the error i.e., the optimal rotation matrix; S213, fine registration, i.e., NID optimization; Effective point cloud voxel features are derived based on the optimal rotation matrix. The corresponding point cloud coordinates are projected onto the camera image plane. The nearest point corresponding to each pixel is retained through a depth buffer test, and occluded invalid points are filtered out to output a set of visible point clouds. ; Then, with the visible point cloud and the augmented image as input, the normalized information distance is computed: （5） In equation (5), is the entropy, is the mutual information; Finally, the Nelder-Mead optimization algorithm is adopted to iteratively adjust the initial extrinsic parameter matrix including the rotation matrix and the translation vector with the NID value as the optimization target until the NID value converges to the minimum value, and the optimal extrinsic parameter matrix achieving the millimeter-level spatial registration is output, the extrinsic parameter matrix including the rotation matrix and the translation vector . S22, time alignment compensation: The time alignment compensation is based on the spatial alignment result, inputs the optimal external parameter matrix and the original timestamps of the multiple sensors, and realizes time alignment through "hardware trigger coarse synchronization-Kalman filter fine compensation", and finally outputs the multi-modal data aligned in time and space.

4. The rail scene radar and vision fusion intrusion monitoring method based on sparse feature fusion according to claim 3, characterized in that, The time alignment through "hardware trigger coarse synchronization-Kalman filter fine compensation" specifically includes the following steps. S221, hardware trigger coarse synchronization: The hardware synchronization trigger signal line is used to connect the laser radar and the camera, and the synchronization trigger signal is sent when the sensor collects data, so that the original timestamp error of the two is preliminarily reduced from the 500us level to the 1ms level; S222, Kalman filter fine compensation After the hardware coarse compensation, Kalman filter fine compensation is performed, that is, the residual time error is iteratively smoothed based on the Kalman filter model, so that the timestamp error of the multiple sensors is finally stabilized at <1ms, and the laser radar point cloud and the camera image are strictly synchronized in the time dimension; The calculation formula of the first-order Kalman filter for predicting the timestamp is as follows: （6）； In formula (6), is the predicted synchronization timestamp at the kth moment, which is the weighted fusion result of the current measurement and the historical estimation, is the raw timestamp of the sensor at the kth moment, directly collected from the hardware clock of the laser radar or camera, is the predicted synchronization timestamp at the k-1th moment, i.e. the time estimation value at the previous moment, =0.95 is the filter coefficient for balancing the current measurement and the historical estimation, is the frame interval time.

5. The rail scene radar and vision fusion intrusion monitoring method based on sparse feature fusion according to claim 3 or 4, characterized in that, The S3 specifically includes the following steps. S31, view angle perception query generation: With the multi-modal data aligned in time and space as input, the three-dimensional coordinates of the foreign matter in the laser radar coordinate system and the corresponding camera semantic features are obtained through 2D detection, monocular 3D detection and three-dimensional coordinate conversion; S32, dynamic feature diffusion; With the three-dimensional coordinates of the foreign matter in the laser radar coordinate system as input, the dense point cloud features after diffusion are extracted through dynamic adjustment of the diffusion radius and sparse convolution operation; S33, uncertainty perception fusion; Based on the dense point cloud features after diffusion and camera semantic features, the positioning uncertainty of the two types of sensors is quantified, the fusion weight is dynamically allocated, and the fusion features are finally output .

6. The rail scene radar and vision fusion intrusion monitoring method based on sparse feature fusion according to claim 5, characterized in that, The view angle perception query generation specifically includes the following steps. S311, 2D detection; The lightweight FCOS detector is used to quickly detect a two-dimensional bounding box of the foreign matter based on spatiotemporal alignment multi-modal data, to realize preliminary positioning of the foreign matter in an image plane, to output a 2D detection result including a category and a confidence, and to output an intermediate layer semantic feature in a corresponding position of the image according to the two-dimensional bounding box , which is a camera semantic feature S312, monocular 3D detection; Based on the 2D detection result, the three-dimensional spatial position information of the foreign matter is calculated from the monocular image by combining the camera internal parameter and the depth estimation model, the three-dimensional profile of the foreign matter in space is preliminarily constructed, the 2D information of the foreign matter is extended to the 3D space, and the 3D information of the foreign matter is obtained; S313, three-dimensional coordinate conversion; According to the optimal external parameter matrix obtained through calibration, the 3D information of the foreign matter obtained through monocular 3D detection is converted from the camera coordinate system to the laser radar coordinate system; The 2D coordinates are converted into 3D space points through the camera internal and external parameter matrix, and the formula is as follows: （7）； In formula (7), is the three-dimensional coordinate of the target in the laser radar coordinate system, is the inverse matrix of the camera extrinsic parameter, is the inverse matrix of the camera intrinsic parameter, are the image coordinates of the center points of the 2D detection frame respectively, d is the depth value of the monocular 3D detection target, which is output by the monocular 3D detection.

7. The rail scene radar and vision fusion intrusion monitoring method based on sparse feature fusion according to claim 5, characterized in that, The dynamic feature diffusion specifically includes the following steps. S321, diffusion range calculation: With the three-dimensional coordinates of the foreign matter in the laser radar coordinate system and the corresponding camera semantic features output by the view angle perception query generation as input, the diffusion radius is dynamically adjusted according to the target size and distance, and the calculation formula is as follows: （7）； In formula (7), represents the feature diffusion radius of the i-th target, which determines the neighborhood range of 3D sparse convolution, is the average size of the target, is the exponential attenuation coefficient, and d is the straight-line distance between the target and the laser radar; S322, 3D sparse convolution; The sparse point cloud features are diffused through 3*3*3 sparse convolution, and the operation is as follows: （8）； In formula (8), is the dense point cloud feature after diffusion, is the input sparse point cloud feature, the sparse point cloud feature is an effective point cloud voxel feature is obtained through the spatial mapping relationship of S313, denotes a dynamic diffusion radius, and denotes feature splicing, is a 3*3*3 sparse convolution.

8. The rail scene radar and vision fusion intrusion monitoring method based on sparse feature fusion according to claim 6, characterized in that, The modal uncertainty quantification specifically includes the following steps. S331, uncertainty modeling, that is, laser radar modal modeling and camera modal modeling are performed respectively; The laser radar modal modeling object is a diffuse laser radar point cloud feature output after diffusion for step dynamic feature diffusion , the positioning reliability is evaluated by the spatial distance between the point cloud geometric feature and the target real position, wherein the input is the laser radar feature, that is, the dense point cloud feature output after step dynamic feature diffusion; the real box coordinate B comes from the labeled data, and the output is the laser radar modal confidence weight , the numerical range is [0, 1] The laser radar modal is as follows: （9）； In equation (9), is the lidar confidence weight, is the lidar feature, represents the Euclidean distance of the lidar point cloud on the BEV plane, is the regression function of the camera feature, and B is the real box coordinate. The camera modal modeling is as follows: （10）； In formula (10), is the confidence weight of the camera modality, with a value range of [0, 1], is the regression function of the camera feature, is the camera semantic feature, B is the real box coordinate, represents the Euclidean distance of the laser radar point cloud on the BEV plane; S332, weighted fusion; According to the sensor confidence degree, weights are dynamically assigned, camera semantic features and lidar geometric features are fused, and the input is camera semantic features and lidar features : （11）； In equation (11), is the fused feature.

9. The rail scene radar and vision fusion intrusion monitoring method based on sparse feature fusion according to claim 8, characterized in that, The S4 is specifically: fusing features A target heat map is generated by a sparse detection head, and hierarchical early warning is realized in combination with a dynamic safety distance model. S41, center heat map generation; Firstly, a center heat map generation algorithm based on a dynamic Gaussian kernel is designed, and the Gaussian kernel parameter is dynamically adjusted: The input is the target width W and length L obtained by monocular 3D detection and feature diffusion, and the output is a Gaussian kernel standard deviation wherein W and L depend on the extrinsic parameter matrix of space-time synchronous calibration and the cross-modal feature fusion result S313-S332. The Gaussian kernel standard deviation is calculated according to the target physical size: （12）； In equation (12), and W and L are the target width and length; Then, a center heatmap generation algorithm based on dynamic Gaussian kernels is used to generate the heatmap, resulting in the center heatmap. Abstract fusion features The probability distribution is converted into a center heatmap, which is a two-dimensional heatmap with pixel values ​​ranging from [0,1]. Each pixel value represents the probability that the corresponding location is the center of the target. S42, hierarchical early warning strategy; According to the dynamic relationship between the actual distance of the target and the track and the train running speed, combined with the target position probability distribution in the center heat map, the hierarchical early warning is realized through the preset safety distance threshold: Dynamic safety distance: （13）； In formula (13), The safety protection threshold is dynamically adjustable at different speeds for real-time speed of the train, Is input in real time by the track signal system; Three-level early warning thresholds are set: Yellow warning: target distance < yellow warning threshold and speed > 50 km / h → trigger sound and light prompt, remind the monitoring personnel to pay attention; Orange warning: target distance < orange warning threshold and speed > 30 km / h → start the brake system pre-charge air, shorten the subsequent brake response time; Red warning: target distance < red warning threshold → directly trigger emergency braking, force the train to stop to avoid collision; The three-level early warning thresholds are set based on the safety distance in the multiple relationship, that is, the yellow warning threshold = 4 × safety distance, the orange warning threshold = 2 × safety distance, and the red warning threshold = safety distance.

10. A system for realizing a sparse feature fusion-based track scene radar and visual fusion intrusion monitoring method, characterized in that, The system includes hardware modules and software modules: The hardware modules include a heterogeneous sensor combination of laser radar and high-definition camera, and an edge computing unit: Laser radar: deployed on both sides of the track, used to obtain high-precision three-dimensional point cloud data, with a maximum effective detection distance of 200 meters; High-definition camera: equipped with a high dynamic range imaging module, with a resolution of 2560×1440 and a field of view of 120°; Edge computing unit: integrated with high-performance GPU, supporting real-time processing and fusion computing of sensor data; The software modules include five core modules: data acquisition module, data preprocessing module, space-time synchronization and calibration module, sparse feature fusion module, and intrusion detection and early warning module: Space-time synchronization module: through hardware triggering and Kalman filter compensation, ensure that the timestamp error of multiple sensors is <1ms; Multi-scale feature extraction: hierarchical extraction of laser radar voxel features and camera image features; Dynamic point cloud fusion: combined with sparse convolution and uncertainty perception mechanism, realize cross-modal feature complementation; Intrusion judgment module: based on the sparse detection head to generate the center heat map, combined with the hierarchical early warning strategy to output protection instructions.

Images