A railway train operation safety early warning method based on image, point cloud and inertial navigation information fusion

Abstract

The application discloses a kind of railway train operation safety early warning method based on image, point cloud and inertial navigation information fusion, comprising: establishing unified multi-sensor perception reference coordinate system;Target detection is carried out, laser radar point cloud is projected to image plane and is associated with detection target, and the spatial position and relative distance information of target are obtained by screening;The pose information output as input is fused and state estimation is carried out, the continuous prediction of target state is maintained when the measurement is missing, and the smooth and continuous target distance information is output;According to target detection result, continuous distance estimation result and its time sequence stability, dynamic risk assessment is carried out on operation target, and alarm signal is generated or directly output to intervention system.The application improves the target perception stability, ranging accuracy and early warning reliability under shielding, bad light and weather conditions by multi-source information deep fusion and collaborative mutual verification, and provides high-robustness safety guarantee for railway shunting operation.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation and driving behavior analysis, and in particular to a railway train operation safety early warning method based on the fusion of image, point cloud and inertial navigation information. Background Technology

[0002] In recent years, with the continuous expansion of my country's railway network and the increasing demands for transportation efficiency, the density of station operations has been constantly increasing, placing higher requirements on on-site operational safety assurance capabilities. Railway shunting operations occur extensively within stations, marshalling yards, and dedicated lines, involving coupling, uncoupling, marshalling, track switching, and propulsion. These operations are characterized by frequent low-speed vehicle movement, close personnel involvement, limited working space, severe obstruction, and susceptibility to environmental factors such as backlighting, nighttime conditions, rain, snow, and dust, resulting in a high concentration of risks. Currently, many railway station shunting operations still rely primarily on manual observation, radio communication, and experience-based judgment, with relatively insufficient intelligent sensing and proactive early warning capabilities. Accident cases demonstrate that in situations such as personnel intrusion, track or equipment malfunctions, or misjudgment of signal targets, the lack of timely and reliable risk identification and early warning mechanisms can easily lead to serious consequences. Therefore, existing technologies are beginning to incorporate onboard or yard-level video intelligent analysis, radar, or lidar detection to improve the level of railway operational safety assurance. Intelligent early warning systems in railway scenarios can typically be abstracted into a processing chain of perception, localization, scene understanding, risk assessment, and alarm / intervention. Existing implementation schemes mainly include image-based methods and point cloud-based methods.

[0003] (a) Image-based methods

[0004] Image-based methods typically deploy visible light cameras, infrared cameras, or binocular cameras at the vehicle or field end. They utilize target detection and tracking networks to identify targets such as personnel, vehicles, coupler areas, switches, signals, and clearance signs from images. Combined with monocular depth estimation, binocular parallax, or scene-calibrated ranging methods, they estimate the relative distance and motion trend between the target and the locomotive / vehicle, thereby triggering intrusion warnings or collision risk warnings. To meet real-time requirements, lightweight detectors, region of interest constraints, and rule-based threshold strategies are commonly used in engineering implementations. However, these methods have the following shortcomings: (1) Image detection results are two-dimensional observations, making it difficult to directly and stably reflect the true three-dimensional spatial position of the target; (2) Visual ranging results are easily affected by changes in camera posture, viewing angle, target scale, and lighting conditions; (3) Under conditions of backlight, nighttime, rain, snow, dust, and occlusion, false detections, missed detections, and unstable ranging are prone to occur.

[0005] (ii) Point cloud-based methods

[0006] Point cloud-based methods typically deploy lidar at the locomotive end or key operating areas to perform ground or track surface segmentation, clustering, and target detection based on point clouds. For example, they employ 3D bounding box regression or detection algorithms based on voxels or bird's-eye view representations to output the target's 3D position, velocity, or inter-frame displacement and occupied space information. This allows for the calculation of the spatial relationship between the target and the vehicle's operating envelope and the issuance of early warnings. Although this type of method has advantages in geometric measurement accuracy, it still has the following problems: (1) The lidar sampling frequency is usually lower than that of a visual camera, making it difficult to output target distance change information at high frequencies; (2) In cases of occlusion, rain, snow, dust, and unstable reflectivity of the target material, point cloud echoes are prone to sparseness, breakage, or loss; (3) In complex station environments, there are many background objects, and there is a risk of misassociation in target point cloud extraction and matching.

[0007] (III) Shortcomings of existing multi-sensor methods

[0008] While some existing multi-sensor solutions incorporate multiple information sources such as video, radar, fiber optics, and track circuits, many solutions in engineering applications still rely primarily on a single sensor or a single information source, lacking effective multi-source verification mechanisms. This makes it difficult to stably output target distances and risk indicators that can be used for safety decision-making. Other solutions focus more on area intrusion detection or security alarms, providing intrusion locations or alarm conclusions, but lacking the ability to continuously measure distances and conduct dynamic risk assessments for railway shunting operations. Furthermore, existing vision-based ranging methods typically rely on monocular depth estimation or scene calibration, and their distance accuracy is significantly affected by environment and attitude. While existing lidar point cloud-based measurement methods possess high geometric accuracy, under conditions of limited onboard computing power, sparse point clouds, and occlusion, they often only output stably at a low frequency, failing to meet the demands of railway shunting operations for high-frequency, continuous, and predictable distance change monitoring.

[0009] Therefore, existing technologies still lack a safety early warning method that can collaboratively integrate image, point cloud, and inertial or odometer information under complex railway shunting conditions, thereby achieving stable target identification, reliable ranging, continuous state estimation, and risk assessment output. Summary of the Invention

[0010] In view of this, the purpose of this invention is to provide a railway train operation safety early warning method based on the fusion of image, point cloud and inertial navigation information. Through deep fusion and collaborative verification of multi-source information, it significantly improves the target perception stability, ranging accuracy and early warning reliability under occlusion, poor lighting and weather conditions, and provides a highly robust safety guarantee for railway shunting operations.

[0011] The present invention solves the technical problem by adopting the following technical solution:

[0012] A railway train operation safety early warning method based on the fusion of image, point cloud, and inertial navigation information includes the following steps:

[0013] Step S1: Establishing a spatiotemporal reference and calibrating multiple sensors. Time synchronization and spatial registration are performed on the visual camera, lidar, and inertial measurement unit to establish a unified multi-sensor perception reference coordinate system.

[0014] Step S2, target detection, point cloud projection and distance measurement: target detection is performed using visual camera images, the LiDAR point cloud is projected onto the image plane and associated with the detected target, and the spatial position and relative distance information of the target are obtained through clustering and robust filtering.

[0015] Step S3, State estimation and fusion: Using the pose information output by the inertial measurement unit or inertial odometry as input, the measurement results obtained in step S2 are fused and the state is estimated through a filter. When the measurement is missing, the continuous prediction of the target state is maintained, and smooth and continuous target distance information is output.

[0016] Step S4, Risk Assessment and Alarm / Intervention Output: Based on the target detection results, continuous distance estimation results and their temporal stability, a dynamic risk assessment is performed on the operation target, and an alarm signal is generated or directly output to the intervention system.

[0017] Furthermore, in step S1, the method for establishing the intrinsic parameter model of the visual camera includes:

[0018] To achieve the projection of a 3D point of the LiDAR onto the image pixel plane, it is necessary to establish the mapping relationship between the camera coordinate system, the image coordinate system, and the pixel coordinate system, and then calculate the camera intrinsic parameter matrix accordingly.

[0019] Let the camera coordinate system be... The image coordinate system is focal length is Based on the relationship of similar triangles, the camera coordinate system points Points in the image coordinate system satisfy: Its matrix form is written as: ;

[0020] Digital images are considered to be of size A two-dimensional matrix; let the pixel coordinate system be... The pixel is located at the first Column, No. Line, origin of the image coordinate system Corresponding to the principal point position in the pixel coordinate system ;set up , Each pixel in the image coordinate system , Given a unit length in the direction, the relationship between the image coordinate system and the pixel coordinate system is: The corresponding matrix form is:

[0021] Combining the above relationships, we obtain the projection relationship from the camera coordinate system to the pixel coordinate system: ;

[0022] in for The camera intrinsic parameter related projection matrix, defining the intrinsic parameter matrix. : ;in , .

[0023] Furthermore, the visual camera intrinsic parameter model employs the following distortion correction: ;in, For the corrected coordinates, For ideal imaging coordinates, The radial distortion coefficient is... Here, r represents the tangential distortion coefficient. The distance from the origin of the coordinate system, and satisfying: .

[0024] Furthermore, the extrinsic parameter calibration method for LiDAR and vision camera is as follows:

[0025] In the process of projecting the 3D points of the LiDAR radar onto the pixel coordinates of the image, it is necessary to obtain the LiDAR coordinate system. To the camera coordinate system The rigid body transformation relationship, i.e., the extrinsic parameters; the extrinsic parameters are represented by the rotation matrix and the translation vector: ;

[0026] Let {C} be the three-dimensional coordinates of the lidar point in the camera coordinate system. Let be the rotation matrix from the lidar coordinate system {L} to the camera coordinate system {C}. Let {L} be the three-dimensional coordinates of the same spatial point in the lidar coordinate system. Let be the translation vector from the lidar coordinate system {L} to the camera coordinate system {C}. Let be a special orthogonal group in three dimensions, representing the set of rotation matrices in three-dimensional space. It is a three-dimensional real vector space;

[0027] Common features between the radar and camera are extracted using a checkerboard calibration board, and extrinsic parameters are solved with the goal of minimizing reprojection error; in the first... In a single synchronized data pair: the point cloud side obtains the set of three-dimensional coordinates of the corner points within the chessboard grid. ;

[0028] The image side detected the corresponding set of two-dimensional pixel corner points: ;

[0029] in, The total number of corner points on the chessboard. Let be the number of interior corner points in the horizontal direction of the chessboard grid. This represents the number of interior corner points in the vertical direction of the chessboard grid. Let be the two-dimensional pixel coordinates of the i-th corner point of the chessboard grid in the k-th frame image. This refers to the vertical pixel coordinates of the corner point in the image coordinate system;

[0030] To avoid ambiguity in numbering caused by the symmetry of the chessboard, and Perform consistent sorting to ensure that the same index Both sides represent the same physical corner point; , , These are the coordinate components of the corner point along the X, Y, and Z axes in the lidar coordinate system, respectively. For any lidar corner point... :

[0031] Transform to camera coordinate system: ;

[0032] Projected onto the normalized image plane: ;

[0033] in, Let be the homogeneous coordinates of the i-th corner point on the normalized image plane. Let be the lateral coordinate of the i-th corner point on the normalized image plane. Let be the vertical coordinate of the i-th corner point on the normalized image plane;

[0034] Mapped to pixel coordinates via intrinsic parameter matrix: ;

[0035] in, To predict the horizontal coordinates of a pixel, i.e., its column coordinates in the image, This is to predict the vertical coordinates of the pixels, i.e., the row coordinates in the image;

[0036] For the Set of data, define a single-point reprojection error vector : ;

[0037] in, Let be the true horizontal pixel coordinates of the i-th corner point detected in the k-th image group. Let be the true vertical pixel coordinates of the i-th corner point detected in the k-th image group;

[0038] Corresponding pixel distance error for: ;

[0039] The extrinsic parameter estimation is then expressed as a least-squares problem summing over all poses and all corner points, where the solution obtained from RANSAC estimation is the initial value, and the optimization formula is: ;

[0040] in, This represents a combined function of coordinate transformation, perspective projection, and intrinsic parameter mapping. To determine the total number of data sets.

[0041] Furthermore, in step S2, the specific methods for target detection, point cloud projection, and distance measurement include:

[0042] Step S21: Perform target detection on the image acquired by the vision camera, and output the target category, confidence score, and two-dimensional bounding box;

[0043] Step S22: Using the intrinsic and extrinsic parameters obtained in step S1, project all three-dimensional points in a frame of lidar point cloud onto the image pixel plane.

[0044] Step S23: For each detected target, select a subset of point clouds whose projected coordinates fall within its two-dimensional bounding box to form a candidate frustum point set.

[0045] Step S24: Perform three-dimensional spatial clustering on the candidate point set of the view frustum, remove noise points and background points, and select the target cluster with the most points as the true echo point set of the target;

[0046] Step S25: Based on the statistical median of the distances of all points in the target cluster, calculate and output the current distance measurement of the target relative to the lidar.

[0047] Furthermore, in step S3, the specific methods for state estimation and fusion include:

[0048] Step S31: Define a state vector that includes the carrier pose and the target global position;

[0049] Step S32: Establish a kinematic prediction model, use the velocity and angular velocity information of the carrier provided by the inertial measurement unit or inertial odometry to drive the prediction update of the state vector, and treat the target as stationary or in constant motion during the prediction stage.

[0050] Step S33: Establish a measurement update model. When step S2 provides a valid target distance measurement, the measurement value is converted to the state space, and the innovation quantity between the predicted measurement value and the actual value is calculated.

[0051] Step S34: Use the extended Kalman filter algorithm to perform state fusion. When measurements are available, combine prediction and measurements to perform state correction. When measurements are missing, only the prediction step in step S32 is executed.

[0052] Step S35: Calculate and output the continuous distance between the target and the carrier in real time based on the filtered state vector.

[0053] Furthermore, in step S34, before measurement, the Normalized Squared Innovation (NIS) is used as a consistency gating test. When the NIS value exceeds a preset threshold, the current measurement is determined to be an outlier and rejected, and only the prediction result is retained.

[0054] Furthermore, the risk assessment in step S4, including enhanced assessment rules for specific targets such as traffic signals, includes the following processing logic:

[0055] Establishment and confirmation mechanism: The candidate traffic light target is continuously observed for multiple frames. Only after the color is stable, the distance is within the effective range, and a certain number of frames are continuously detected can it be confirmed as the current valid traffic light and its tracking trajectory is initialized.

[0056] Prediction-measurement consistency check: Compare the radar direct ranging value of the current frame with the filter predicted distance value. If the difference is greater than the threshold, it is determined that the measurement is inconsistent, the update is rejected and the output of the previous frame is maintained; if they are consistent, the update is accepted.

[0057] Missed detection retention and track estimation: In the event of a short-term missed detection, the current signal light track is not immediately deleted. Instead, the filter pure prediction mode is used to maintain continuous distance and color status output.

[0058] False detection elimination and clearing: In the candidate stage, if the confirmation conditions cannot be met for multiple consecutive frames, the candidate is eliminated; in the stable tracking stage, if the detection fails for multiple consecutive frames, the current trajectory is cleared.

[0059] Exit mechanism: When the filter predicts a distance less than the preset proximity threshold, it is determined that the train has passed the signal light, and the system automatically clears the trajectory and resets the status;

[0060] Color switching confirmation: A hysteresis update strategy is adopted for the output color status. Color switching output is only performed when the new color is observed stably for multiple consecutive frames, in order to suppress single-frame color flicker.

[0061] The present invention discloses a railway train operation safety early warning method based on the fusion of image, point cloud and inertial navigation information, which has the following beneficial effects:

[0062] 1. Significantly improves the stability and robustness of target perception under complex conditions: This invention establishes a unified spatiotemporal benchmark and deeply fuses high-semantic image information and high-precision point cloud geometric information, achieving collaborative perception of targets guided by images and distance measured by point clouds. Compared to the performance degradation of single-image methods under conditions such as backlighting, rain, and snow, and the echo loss of single-point cloud methods under occlusion and dust, this method ensures that at least one main sensor can provide effective information through multi-source mutual verification and redundant design, thereby maintaining stable locking and reliable perception of safety-critical targets in various harsh environments.

[0063] 2. Achieved high-frequency, continuous, and predictable target distance output: This invention creatively introduces inertial information as the driving force for state prediction, constructing an EKF-based motion estimation and fusion framework. This design fundamentally solves the pain point of low-frequency and intermittent missing LiDAR measurements. When radar measurements are briefly lost, the system is no longer helpless, but seamlessly switches to inertial estimation mode, outputting smooth and continuous predicted distances, forming an uninterrupted sensing flow. This provides temporally dense and logically coherent input for subsequent risk assessment, completely eliminating the information blind spot caused by measurement interruptions.

[0064] 3. Significantly reduces the risks of false alarms, missed alarms, and miscorrelation: At the risk assessment and decision-making level, this invention designs a sophisticated time-series logic processing mechanism, including multi-frame confirmation, prediction-measurement consistency gating, hysteresis update, missed detection retention, and intelligent exit. This mechanism incorporates the uncertainty of single-frame detection into a short-term situation assessment framework, effectively filtering transient noise, background interference, and target miscorrelation, resulting in highly deterministic and reliable alarm information output. It successfully upgrades the traditional point-based, fragmented alarm mode to a continuous, logical situation assessment mode, significantly reducing production interruptions caused by false alarms and security risks caused by missed alarms.

[0065] 4. Enhanced overall effectiveness of railway shunting operation safety early warning: By integrating the above advantages into a complete closed-loop chain, this invention provides railway shunting operations with unprecedented perception accuracy, status continuity, and decision-making logic. This enables the entire system to detect risks earlier, more accurately, and more stably, and to issue alarms or intervene in a more reasonable manner, thereby providing a more robust and intelligent safety barrier for personnel, vehicles, and goods. It facilitates a leapfrog transformation from human-based to technology-based safety measures, possessing significant practical value and social benefits. Attached Figure Description

[0066] Figure 1 This is the overall flowchart of the method of the present invention;

[0067] Figure 2 This is a schematic diagram of multi-sensor time synchronization;

[0068] Figure 3 This is a diagram showing the camera intrinsic parameter calibration results;

[0069] Figure 4 This is a diagram showing the calibration results of the lidar-camera extrinsic parameters;

[0070] Figure 5 This is the output trajectory diagram of a laser inertial odometer;

[0071] Figure 6 This is a schematic diagram of the LiDAR points projected onto the image plane;

[0072] Figure 7 This is a demonstration diagram of the integrated positioning algorithm and early warning output. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] To address the shortcomings of existing technologies, the technical problems to be solved by this invention include: how to achieve time synchronization and spatial alignment between images, point clouds, and inertial information in railway shunting operation scenarios to establish a unified multi-sensor perception benchmark; how to reliably correlate target detection results in images with lidar point clouds to obtain the spatial position and relative distance information of targets; how to continuously output smooth, stable, and predictable target distance information even when point cloud measurements are low-frequency, sparse, or intermittently missing; and how to perform dynamic risk assessment of operational targets based on multi-source fusion results and generate stable and reliable alarm or intervention outputs, thereby reducing the risks of false alarms, missed alarms, and miscorrelation.

[0075] refer to Figure 1 This invention proposes a railway train operation safety early warning method based on the fusion of image, point cloud, and inertial navigation information. This method comprehensively utilizes pose information output from visual cameras, lidar, and inertial measurement units (IMUs) or laser inertial odometers (LIOs) to detect, locate, continuously estimate distances, assess risks, and output early warnings for targets in railway shunting operations. The method includes the following steps:

[0076] Step S1, establishing a spatiotemporal reference and calibrating multiple sensors, involves time synchronization and spatial registration of the visual camera, LiDAR, and inertial measurement unit to establish a unified multi-sensor perception reference coordinate system. This step is used to achieve time synchronization and spatial registration between the visual camera, LiDAR, and inertial information, providing a foundation for subsequent point cloud projection, target matching, and fusion estimation.

[0077] Time Synchronization: Since the sampling frequency of LiDAR is usually lower than that of the camera, the LiDAR point cloud frame is used as the synchronization matching benchmark. Within a LiDAR data cycle, image data acquired within that cycle is buffered using a queue. When the queue length reaches a preset maximum value or the accumulated time reaches one LiDAR cycle, the nearest image frame in time is matched using the current LiDAR point cloud timestamp. If no new image data is received within a certain LiDAR data cycle, that LiDAR frame and its corresponding buffered image data are discarded, and matching begins with the next frame. After each matching is completed, a synchronization data pair is output, the queue is cleared, and the timer is reset to achieve time synchronization between image data and point cloud data.

[0078] Methods for establishing intrinsic parameter models for visual cameras include:

[0079] To achieve the projection of a 3D point of the LiDAR onto the image pixel plane, it is necessary to establish the mapping relationship between the camera coordinate system, the image coordinate system, and the pixel coordinate system, and then calculate the camera intrinsic parameter matrix accordingly.

[0080] Let the camera coordinate system be... The image coordinate system is focal length is Based on the relationship of similar triangles, the camera coordinate system points Points in the image coordinate system satisfy: Its matrix form is written as: ;

[0081] Digital images are considered to be of size A two-dimensional matrix; let the pixel coordinate system be... The pixel is located at the first Column, No. Line, origin of the image coordinate system Corresponding to the principal point position in the pixel coordinate system ;set up , Each pixel in the image coordinate system , Given a unit length in the direction, the relationship between the image coordinate system and the pixel coordinate system is: The corresponding matrix form is: ;

[0082] Combining the above relationships, we obtain the projection relationship from the camera coordinate system to the pixel coordinate system: ;

[0083] in for The camera intrinsic parameter related projection matrix, defining the intrinsic parameter matrix. : ;in , .

[0084] However, actual imaging deviates from the ideal pinhole model, mainly due to radial distortion (caused by lens shape) and tangential distortion (caused by assembly errors, etc.). The intrinsic parameter model of the vision camera uses the following distortion correction: ;in, For the corrected coordinates, For ideal imaging coordinates, The radial distortion coefficient is... Let be the tangential distortion coefficient, and r be the distance from point (x, y) to the origin, satisfying: .

[0085] The extrinsic parameter calibration methods for LiDAR and vision cameras are as follows:

[0086] In the process of projecting the 3D points of the LiDAR radar onto the pixel coordinates of the image, it is necessary to obtain the LiDAR coordinate system. To the camera coordinate system The rigid body transformation relationship, i.e., the extrinsic parameters; the extrinsic parameters are represented by the rotation matrix and the translation vector: ;

[0087] in, Let {C} be the three-dimensional coordinates of the lidar point in the camera coordinate system {C}. Let be the rotation matrix from the lidar coordinate system {L} to the camera coordinate system {C}. Let {L} be the three-dimensional coordinates of the same spatial point in the lidar coordinate system. Let be the translation vector from the lidar coordinate system {L} to the camera coordinate system {C}. Let be a special orthogonal group in three dimensions, representing the set of rotation matrices in three-dimensional space. It is a three-dimensional real vector space.

[0088] Common features between the radar and camera are extracted using a checkerboard calibration board, and extrinsic parameters are solved with the goal of minimizing reprojection error; in the first... In a single synchronized data pair: the point cloud side obtains the set of three-dimensional coordinates of the corner points within the chessboard grid. ;

[0089] The image side detected the corresponding set of two-dimensional pixel corner points: ;

[0090] in, The total number of corner points on the chessboard. This represents the number of interior corner points in the horizontal direction of the chessboard. This represents the number of interior corner points in the vertical direction of the chessboard grid. Let be the two-dimensional pixel coordinates of the i-th corner point of the chessboard grid in the k-th frame image. This represents the vertical pixel coordinates of the corner point in the image coordinate system.

[0091] To avoid ambiguity in numbering caused by the symmetry of the chessboard, and Perform consistent sorting to ensure that the same index Both sides represent the same physical corner point; , , These are the coordinate components of the corner point along the X, Y, and Z axes in the lidar coordinate system, respectively. For any lidar corner point... :

[0092] Transform to camera coordinate system: ;

[0093] Projected onto the normalized image plane: ;

[0094] in, Let be the homogeneous coordinates of the i-th corner point on the normalized image plane. Let be the lateral coordinate of the i-th corner point on the normalized image plane. Let be the vertical coordinate of the i-th corner point on the normalized image plane;

[0095] Mapped to pixel coordinates via intrinsic parameter matrix: ;

[0096] in, To predict the horizontal coordinates of a pixel, i.e., its column coordinates in the image, This is to predict the vertical coordinates of the pixels, i.e., the row coordinates in the image;

[0097] For the Set of data, define a single-point reprojection error vector : ;

[0098] in, Let be the true horizontal pixel coordinates of the i-th corner point detected in the k-th image group. Let be the true vertical pixel coordinates of the i-th corner point detected in the k-th image group;

[0099] Corresponding pixel distance error for: ;

[0100] The extrinsic parameter estimation is then expressed as a least-squares problem summing over all poses and all corner points, where the solution obtained from RANSAC estimation is the initial value, and the optimization formula is: ;

[0101] in, This represents a combined function of coordinate transformation, perspective projection, and intrinsic parameter mapping. To determine the total number of data sets.

[0102] Step S2, target detection, point cloud projection and distance measurement: target detection is performed using visual camera images, the LiDAR point cloud is projected onto the image plane and associated with the detected target, and the spatial position and relative distance information of the target are obtained through clustering and robust filtering.

[0103] Specific methods for target detection, point cloud projection, and distance measurement include:

[0104] Step S21: Perform target detection on the image acquired by the vision camera, and output the target category, confidence score, and two-dimensional bounding box;

[0105] Step S22: Using the intrinsic and extrinsic parameters obtained in step S1, project all three-dimensional points in a frame of lidar point cloud onto the image pixel plane.

[0106] Step S23: For each detected target, select a subset of point clouds whose projected coordinates fall within its two-dimensional bounding box to form a candidate frustum point set.

[0107] Step S24: Perform three-dimensional spatial clustering on the candidate point set of the view frustum, remove noise points and background points, and select the target cluster with the most points as the true echo point set of the target;

[0108] Step S25: Based on the statistical median of the distances of all points in the target cluster, calculate and output the current distance measurement of the target relative to the lidar.

[0109] 1. Image object detection

[0110] After the images acquired by the vision camera are processed by the target detection algorithm, the target category, confidence score, and two-dimensional bounding box can be output in pixel coordinate system. Target categories may include, but are not limited to, personnel, vehicles, obstacles, signals, equipment targets, or other targets related to railway shunting safety.

[0111] In a preferred embodiment, YOLOv5 is used as the object detection model, and deployment optimization is performed using ONNX and TensorRT. Multiple bounding boxes can be output per frame, in the form of Bboxes. ;

[0112] in The coordinates of the top left pixel. These are the pixel coordinates of the bottom right corner.

[0113] 2. Projection of point cloud onto image plane

[0114] Let the lidar point be in the lidar coordinate system as Camera-radar extrinsic parameters are The camera intrinsic parameter matrix is The projection of a point cloud onto a pixel plane can be represented as: ;

[0115] Then, the camera intrinsic parameter matrix is ​​used to map it to the pixel plane: ;

[0116] in, Points located behind the camera should be directly eliminated; in addition, pixels falling outside the image boundary should not participate in the matching.

[0117] 3. Construct a candidate set of point clouds from the target bounding boxes.

[0118] For each detection box (bbox) in the image From all projectible points, select points that satisfy the following conditions as the candidate point set for the target: ;

[0119] Recorded as This set corresponds to a frustum-shaped cutoff region in three-dimensional space. Since the frustum may simultaneously contain: the target's true echo, background object echoes, sparse noise points, etc., directly... Taking the mean can lead to unstable distance estimation, so further clustering and robust screening are necessary.

[0120] 4. Point cloud clustering and robust distance estimation

[0121] For each target, the set of candidate frustum points DBSCAN was used for clustering in three-dimensional space to obtain multiple clusters. DBSCAN does not require pre-defined cluster numbers, making it suitable for scenarios where the number of target points is not fixed and noise points exist.

[0122] Set DBSCAN input: Parameter: Neighborhood radius With minimum number of points Output: After obtaining clusters by cluster labels and noise points (labeled as -1), filtering is performed using the maximum support priority principle. ;in, The index for the final selected target cluster, For the j-th cluster;

[0123] The target cluster was finally selected. Then, output the target distance using the median of its distance statistics: Where p is the target cluster Any three-dimensional point in the array;

[0124] Step S3, State estimation and fusion: Using the pose information output by the inertial measurement unit or inertial odometry as input, the measurement results obtained in step S2 are fused and the state is estimated through a filter. When the measurement is missing, the continuous prediction of the target state is maintained, and smooth and continuous target distance information is output.

[0125] This step is used to continuously estimate the target state based on inertial or odometry information in the case of low-frequency, sparse or intermittent missing point cloud measurements, so as to obtain a smooth and continuous distance output.

[0126] In perception systems, although step S2 can provide spatial measurements of the target, these measurements are often low-frequency and intermittently available due to limitations such as sparse point clouds, occlusion, and unstable reflections. Therefore, this invention introduces an extended Kalman filter (EKF) for target motion estimation and multi-source information fusion.

[0127] Its core logic lies in using the pose increment output by the laser inertial odometry (LIO) as the core driving force for state evolution, establishing a joint state model between the vehicle and the target. When measurements are missing, state updates are maintained through odometry calculations; when measurements are valid, cumulative drift is corrected using lidar observations. This mechanism ensures that the system can still output smooth and continuous target distance information even in complex environments.

[0128] Specific methods for state estimation and fusion include:

[0129] Step S31: Define a state vector that includes the carrier pose and the target global position;

[0130] Step S32: Establish a kinematic prediction model, use the velocity and angular velocity information of the carrier provided by the inertial measurement unit or inertial odometry to drive the prediction update of the state vector, and treat the target as stationary or in constant motion during the prediction stage.

[0131] Step S33: Establish a measurement update model. When step S2 provides a valid target distance measurement, the measurement value is converted to the state space, and the innovation quantity between the predicted measurement value and the actual value is calculated.

[0132] Step S34: Use the extended Kalman filter algorithm to perform state fusion. When measurements are available, combine prediction and measurements to perform state correction. When measurements are missing, only the prediction step in step S32 is executed.

[0133] Step S35: Calculate and output the continuous distance between the target and the carrier in real time based on the filtered state vector.

[0134] In a preferred embodiment, in step S34, before measurement, the Normalized Innovative Square (NIS) is used as a consistency gating test. When the NIS value exceeds a preset threshold, the current measurement is determined to be an outlier and rejected, and only the prediction result is retained.

[0135] 1. Definition of State Vector

[0136] Define the system state vector as follows: ;

[0137] in, Let t represent the two-dimensional position and heading angle of the carrier in the world coordinate system. Let t be the position of the target (or the target's representative point) in the world coordinate system at time t.

[0138] 2. Forecasting Phase

[0139] Using velocity and angular velocity provided by inertial / odometry as inputs, a constant-velocity kinematic model is used for state prediction: ;

[0140] in, The time interval between two adjacent frames or between two adjacent filter updates. The velocity components of the target or platform in the x-direction of the world coordinate system, provided by inertial / odometry. The velocity components of the target or platform in the y-direction of the world coordinate system, provided by inertial / odometry. Angular velocity;

[0141] For a short period of time, the target position is approximated as stationary: ;

[0142] Covariance prediction is: ;

[0143] in For process noise covariance, Let be the predicted state covariance matrix at time t, used to describe the uncertainty of the predicted state. This is the state transition matrix; in implementations where velocity / angular velocity is used as external input, we can take... .

[0144] 3. During the update phase

[0145] The measurement is given by the two-dimensional position of the target output by the lidar in its coordinate system: ;

[0146] in, Let be the target two-dimensional position measurement vector output by the lidar at time t. Let the target's coordinates at time t be the axes of the lidar coordinate system. Position coordinates in the direction, Let be the target in the lidar coordinate system at time t. Position coordinates along the axis;

[0147] To match variables in different coordinate systems, the predicted state is projected onto the lidar measurement space. Let the target's displacement relative to the carrier in the world frame be: ;

[0148] set up This represents the rotation of a vector from the world coordinate system to the carrier coordinate system. The external parameters from radar to carrier (satisfying) ,in, Let these be the coordinates of the point in the carrier coordinate system. (where the coordinates of the point are in the lidar coordinate system), then the measurement prediction is: ;

[0149] The innovation quantity is defined as: ;

[0150] in, For measuring the actual two-dimensional position of the target output by the lidar;

[0151] And the linearized measurement matrix is ​​used Constructing innovation covariance: ;

[0152] in, To measure the noise covariance of lidar.

[0153] To suppress estimation divergence caused by mismatches and outlier measurements, consistency gating (NIS) can be used: ;

[0154] when If the threshold is exceeded, the update is rejected and only the prediction result is retained; when the gate is passed, the Kalman gain is updated according to the standard EKF and the state and covariance are corrected.

[0155] 4. Distance Output

[0156] At the output layer, the distance between the target and the carrier. It can be calculated directly from the estimated state: ;

[0157] In actual operation, prediction and updates are performed to correct drift when measurements are available; when measurements are missing (e.g., insufficient point cloud within the target bounding box resulting in empty measurements), prediction-only is performed to maintain distance output continuity; when the target changes, the first frame of valid measurements can be used to adjust the distance output. Reinitialization and resetting of relevant covariance are performed to avoid state crosstalk between different objectives.

[0158] Step S4, Risk Assessment and Alarm / Intervention Output: Based on the target detection results, continuous distance estimation results and their temporal stability, a dynamic risk assessment is performed on the operation target, and an alarm signal is generated or directly output to the intervention system.

[0159] This step is used to assess the risk of a target based on the target detection results, distance estimation results, and their temporal stability, and to output corresponding alarm or intervention information.

[0160] Considering that targets in railway shunting operations can include personnel, vehicles, obstacles, signals, and equipment, this invention designs the risk assessment module using a combination of general target trajectory management and specific target enhancement rules. Specifically, for signal targets, color status and switching stability can be further considered in the processing.

[0161] The risk assessment in step S4 includes enhanced assessment rules for specific targets such as traffic signals, and its processing logic includes:

[0162] Establishment and confirmation mechanism: The candidate traffic light target is continuously observed for multiple frames. Only after the color is stable, the distance is within the effective range, and a certain number of frames are continuously detected can it be confirmed as the current valid traffic light and its tracking trajectory is initialized.

[0163] Prediction-measurement consistency check: Compare the radar direct ranging value of the current frame with the filter predicted distance value. If the difference is greater than the threshold, it is determined that the measurement is inconsistent, the update is rejected and the output of the previous frame is maintained; if they are consistent, the update is accepted.

[0164] Missed detection retention and track estimation: In the event of a short-term missed detection, the current signal light track is not immediately deleted. Instead, the filter pure prediction mode is used to maintain continuous distance and color status output.

[0165] False detection elimination and clearing: In the candidate stage, if the confirmation conditions cannot be met for multiple consecutive frames, the candidate is eliminated; in the stable tracking stage, if the detection fails for multiple consecutive frames, the current trajectory is cleared.

[0166] Exit mechanism: When the filter predicts a distance less than the preset proximity threshold, it is determined that the train has passed the signal light, and the system automatically clears the trajectory and resets the status;

[0167] Color switching confirmation: A hysteresis update strategy is adopted for the output color status. Color switching output is only performed when the new color is observed stably for multiple consecutive frames, in order to suppress single-frame color flicker.

[0168] 1. State and Input Modeling

[0169] The system maintains only one current traffic light track (denoted as Track) externally, and receives the following inputs in each frame: visual detection bounding boxes and detection confidence scores, color classification results (distinguishing only red / yellow / green), and two range measurement information streams: direct point cloud range measurement (preferred) and filter-predicted / estimated distance (output by EKF). Internally, Track contains: distance filtering status, color stabilization status, and several counters used to evaluate the reliability and risk level of the current output. For ease of description, the EKF predicted distance is denoted as... The updated estimated distance is denoted as Direct ranging of point clouds is denoted as Color output is denoted as .

[0170] 2. Current target establishment and multi-frame confirmation

[0171] Upon first detecting a candidate traffic light, the system does not immediately recognize it as the current traffic light. Instead, it employs multi-frame stability constraints to suppress false detections of no-light situations and transient jitter. A candidate target must simultaneously meet the following conditions to enter the stable tracking phase:

[0172] (1) Color stability: The color determination does not change for three consecutive frames;

[0173] (2) Availability and range constraints: When point cloud ranging is available, the following requirements apply: To avoid high errors and misjudgments caused by distant small targets;

[0174] (3) Continuous observation: There are valid detection results in three consecutive frames.

[0175] When the above conditions are met, the system confirms the candidate as the current traffic light, initializes the EKF, and enters stable output mode. If any condition is not met, the system remains in the candidate stage or is directly judged as an unreliable target, and no confirmed traffic light result is output.

[0176] Upon initial detection of a candidate target, the system does not immediately confirm it as a stable tracking target. Instead, a multi-frame confirmation mechanism is employed to suppress transient false detections and jitter. A candidate target can only enter the stable tracking phase after meeting at least the following conditions:

[0177] 3. Risk assessment and anomaly handling model

[0178] (1) Prediction-measurement consistency test

[0179] In multi-signal scenarios, a common failure mode is the misdetection of other signal lights or similar targets as the current signal light. Under a single output constraint, once an incorrect target is accepted, it will directly lead to false alarms in color and distance outputs. Therefore, this paper uses the consistency between direct ranging from point clouds and EKF predicted distance as the main risk criterion.

[0180] When point cloud ranging When available, calculate the predicted measurement difference: ;

[0181] like If the measurement exceeds a preset threshold, it is determined that the measurement in this frame is inconsistent with the existing trajectory, belonging to a high-risk observation (suspected false association / preemption). The system adopts a measurement rejection strategy: it does not use the observation in this frame to update the EKF, only performs prediction propagation, and continues to output the color and predicted distance of the previous stable signal light. Conversely, when If the distance does not exceed the threshold, the measurement is considered to be consistent with the trajectory, and EKF update is performed and the updated distance estimate is output.

[0182] (2) Missed detection retention and track estimation

[0183] In actual driving, traffic lights may experience short-term missed detections due to obstruction, motion blur, or detector instability. To ensure output continuity, the system does not immediately clear the current trajectory when a missed detection occurs, but instead employs a pure prediction-and-hold strategy: when there are no effective detections or available point cloud measurements in the current frame, the EKF only performs prediction propagation, and the distance output directly uses... Color output retains the previous stable color. This strategy enables the system to provide continuous distance information and stable color output even during short periods of missed detection, avoiding frequent switching between output presence and absence.

[0184] (3) False detection rejection and clearing logic

[0185] In scenarios without lights, occasional false detections may occur. To prevent false detections from being incorrectly identified as the current traffic light, the system uses three-frame confirmation as a necessary condition for entering stable tracking during the candidate phase. Simultaneously, if no stable observations support a valid result is generated for three consecutive frames (e.g., discontinuous detection or failure to meet stability conditions), the candidate is directly eliminated, and the system maintains a "no" or "unknown" output. This strategy complements the three-frame confirmation: the former prevents false detections from entering, while the latter ensures that false detections do not persist for extended periods.

[0186] (4) Exit mechanism: Reset rules at close range

[0187] After a vehicle approaches and passes through an intersection, continuing to use the trajectory of the previous light may result in meaningless output or confusion with the next light. This invention employs a distance-based exit rule: when the EKF predicted distance meets... If the signal light information is deemed invalid, the system removes the current trajectory, resets the initial EKF value, and proceeds to wait for confirmation of the next signal light. This rule is used to explicitly terminate the output of signal lights that have already passed the control point, reducing the risk of historical trajectory residue.

[0188] 4. Color Switch Confirmation: Lag Update Strategy

[0189] Actual traffic light switching (e.g., from blue to white) typically occurs while vehicles are stationary and waiting, but visual classification can still exhibit jitter within a single frame. This paper employs a hysteresis strategy for color updates, prioritizing color changes while confirming the switch over three frames.

[0190] (1) When the detected color changes but the trajectory remains reliable (indicated by a small difference between EKF prediction and measurement or continuous availability of measurement), the system does not immediately change the output color, but continues to output the previous stable color and accumulates the color switching count.

[0191] (2) When a new color appears consecutively for three frames and the distance consistency remains good (the change between prediction and measurement is small), then the actual color switch is confirmed and the output color attribute is updated.

[0192] This strategy distinguishes between color flickering misjudgments and genuine color switching: the former is difficult to suppress because it does not meet the condition of consistency for three consecutive frames, while the latter can be identified and the output updated in a short time.

[0193] Through a closed-loop design of confirmation-consistency gating-holding-rejection / reset-switch confirmation, the system can achieve stability and controllability in external output under typical operating conditions such as false detection of color changes, other lights preempting, missed detection, and false detection of no lights, thereby reducing the risk of erroneous signal light information spreading to downstream links.

[0194] S5. Case Verification and System Verification

[0195] First, prepare a checkerboard calibration board, selecting an 8×6 checkerboard with a square side length of 100mm. Fix the camera, move the checkerboard, and take 30 photos of the checkerboard from different positions and angles. Select and fix the focal length of the visual camera. Design a calibration scheme based on the performance characteristics of the visual camera used, and collect calibration data, such as... Figure 3 and Figure 4 The intrinsic parameters of the camera were calibrated using the MATLAB calibration toolbox; point cloud data from the lidar were collected, and joint extrinsic parameters were calculated from accumulated point cloud frames captured under static conditions. The results of the intrinsic and extrinsic parameters are shown below:

[0196] Camera internal parameters: ;

[0197] The camera distortion coefficient is: ;

[0198] Joint external reference: ;

[0199] (1) Using the Kitti autonomous driving dataset, distance measurement was simulated based on the vehicle's operational status data and multi-source information. The presence of actual errors was checked; if any were found, the fusion positioning parameters needed to be adjusted until the optimal distance measurement effect was achieved. The algorithm results are shown in Table 1.

[0200] 1 2 3 4 5 6 7 8 9 10 algorithm 22.96 22.19 21.28 20.39 19.52 18.67 17.86 16.34 15.64 14.00 actual 23.48 22.40 21.33 20.26 19.20 18.18 17.16 16.16 15.16 14.18

[0201] Table 1

[0202] To enable those skilled in the art to more clearly understand the technical solution of this application, the technical solution of this application will be further described below in conjunction with specific railway line embodiments. To illustrate the feasibility of the design method, it is verified using actual railway line data; the actual line selection data is based first on time synchronization of multiple sensors, such as... Figure 2 As shown, low-frequency signal ranging is achieved based on previously calculated intrinsic and extrinsic parameters, where the point cloud cone is used to achieve... Figure 5 As shown, the train's trajectory and attitude information are obtained through a laser-inertial odometry system. Figure 6 As shown. Based on this. Figure 7 To achieve integrated positioning and output of early warning information, the algorithm can provide real-time safety warning indications to the early warning system by providing sensor data in real time.

[0203] (2) Using real vehicle tests, verify whether the actual train operation data is consistent with the simulation results. If there is a difference, the parameters of the algorithm need to be corrected until the best safety and stability effect is achieved.

[0204] This invention proposes a safety early warning method for railway shunting operations based on multi-source fusion of images, point clouds, and IMU data. Addressing typical challenges in station shunting and track operations such as occlusion, backlighting / nighttime conditions, rain, snow, dust, limited space, and complex operational targets, a closed-loop early warning link is constructed through multi-source verification, spatiotemporal alignment, and filtering fusion. Compared to existing solutions that rely solely on video or single information sources like LiDAR, this method helps improve the level of train safety operations.

[0205] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A railway train operation safety early warning method based on the fusion of image, point cloud, and inertial navigation information, characterized in that, Includes the following steps: Step S1: Establishing a spatiotemporal reference and calibrating multiple sensors. Time synchronization and spatial registration are performed on the visual camera, lidar, and inertial measurement unit to establish a unified multi-sensor perception reference coordinate system. Step S2, target detection, point cloud projection and distance measurement: target detection is performed using visual camera images, the LiDAR point cloud is projected onto the image plane and associated with the detected target, and the spatial position and relative distance information of the target are obtained through clustering and robust filtering. Step S3, State estimation and fusion: Using the pose information output by the inertial measurement unit or inertial odometry as input, the measurement results obtained in step S2 are fused and the state is estimated through a filter. When the measurement is missing, the continuous prediction of the target state is maintained, and smooth and continuous target distance information is output. Step S4, Risk Assessment and Alarm / Intervention Output: Based on the target detection results, continuous distance estimation results and their temporal stability, a dynamic risk assessment is performed on the operation target, and an alarm signal is generated or directly output to the intervention system.

2. The railway train operation safety early warning method based on the fusion of image, point cloud, and inertial navigation information according to claim 1, characterized in that, In step S1, the method for establishing the intrinsic parameter model of the visual camera includes: To achieve the projection of a 3D point of the LiDAR onto the image pixel plane, it is necessary to establish the mapping relationship between the camera coordinate system, the image coordinate system, and the pixel coordinate system, and then calculate the camera intrinsic parameter matrix accordingly. Let the camera coordinate system be... The image coordinate system is focal length is Based on the relationship of similar triangles, the camera coordinate system points Points in the image coordinate system satisfy: Its matrix form is written as: ; Digital images are considered to be of size A two-dimensional matrix; let the pixel coordinate system be... The pixel is located at the first Column, No. Line, origin of the image coordinate system Corresponding to the principal point position in the pixel coordinate system ;set up , Each pixel in the image coordinate system , Given a unit length in the direction, the relationship between the image coordinate system and the pixel coordinate system is: ; The corresponding matrix form is: ; Combining the above relationships, we obtain the projection relationship from the camera coordinate system to the pixel coordinate system: ; in for The camera intrinsic parameter related projection matrix, defining the intrinsic parameter matrix. : ;in , .

3. The railway train operation safety early warning method based on the fusion of image, point cloud, and inertial navigation information according to claim 2, characterized in that, The intrinsic parameter model of the visual camera uses the following distortion correction: ;in, For the corrected coordinates, For ideal imaging coordinates, The radial distortion coefficient is... Here, r represents the tangential distortion coefficient. The distance from the origin of the coordinate system, and satisfying: .

4. The railway train operation safety early warning method based on the fusion of image, point cloud, and inertial navigation information according to claim 3, characterized in that, The extrinsic parameter calibration methods for LiDAR and vision cameras are as follows: In the process of projecting the 3D points of the LiDAR radar onto the pixel coordinates of the image, it is necessary to obtain the LiDAR coordinate system. To the camera coordinate system The rigid body transformation relationship, i.e., the extrinsic parameters; the extrinsic parameters are represented by the rotation matrix and the translation vector: ;in, Let {C} be the three-dimensional coordinates of the lidar point in the camera coordinate system. Let be the rotation matrix from the lidar coordinate system {L} to the camera coordinate system {C}. Let {L} be the three-dimensional coordinates of the same spatial point in the lidar coordinate system. Let be the translation vector from the lidar coordinate system {L} to the camera coordinate system {C}. Let be a special orthogonal group in three dimensions, representing the set of rotation matrices in three-dimensional space. It is a three-dimensional real vector space; Common features between the radar and camera are extracted using a checkerboard calibration board, and extrinsic parameters are solved with the goal of minimizing reprojection error; in the first... In a single synchronized data pair: the point cloud side obtains the set of three-dimensional coordinates of the corner points within the chessboard grid. ; The image side detected the corresponding set of two-dimensional pixel corner points: ; in, The total number of corner points on the chessboard. This represents the number of interior corner points in the horizontal direction of the chessboard. This represents the number of interior corner points in the vertical direction of the chessboard grid. Let be the two-dimensional pixel coordinates of the i-th corner point of the chessboard grid in the k-th frame image. This refers to the vertical pixel coordinates of the corner point in the image coordinate system; To avoid ambiguity in numbering caused by the symmetry of the chessboard, and Perform consistent sorting to ensure that the same index Both sides represent the same physical corner point; , , These are the coordinate components of the corner point along the X, Y, and Z axes in the lidar coordinate system, respectively. For any lidar corner point... : Transform to camera coordinate system: ; Projected onto the normalized image plane: ; in, Let be the homogeneous coordinates of the i-th corner point on the normalized image plane. Let be the lateral coordinate of the i-th corner point on the normalized image plane. Let be the vertical coordinate of the i-th corner point on the normalized image plane; Mapped to pixel coordinates via intrinsic parameter matrix: ; in, To predict the horizontal coordinates of a pixel, i.e., its column coordinates in the image, This is to predict the vertical coordinates of the pixels, i.e., the row coordinates in the image; For the first Set of data, define a single-point reprojection error vector : ; in, Let be the true horizontal pixel coordinates of the i-th corner point detected in the k-th image group. Let be the true vertical pixel coordinates of the i-th corner point detected in the k-th image group; Corresponding pixel distance error for: ; The extrinsic parameter estimation is then expressed as a least-squares problem summing over all poses and all corner points, where the solution obtained from RANSAC estimation is the initial value, and the optimization formula is: ; in, This represents a combined function of coordinate transformation, perspective projection, and intrinsic parameter mapping. To determine the total number of data sets.

5. A railway train operation safety early warning method based on the fusion of image, point cloud, and inertial navigation information according to claim 4, characterized in that, Step S2, the specific methods for target detection, point cloud projection, and distance measurement include: Step S21: Perform target detection on the image acquired by the vision camera, and output the target category, confidence score, and two-dimensional bounding box; Step S22: Using the intrinsic and extrinsic parameters obtained in step S1, project all three-dimensional points in a frame of lidar point cloud onto the image pixel plane. Step S23: For each detected target, select a subset of point clouds whose projected coordinates fall within its two-dimensional bounding box to form a candidate frustum point set. Step S24: Perform three-dimensional spatial clustering on the candidate point set of the view frustum, remove noise points and background points, and select the target cluster with the most points as the true echo point set of the target. Step S25: Based on the statistical median of the distances of all points in the target cluster, calculate and output the current distance measurement of the target relative to the lidar.

6. The railway train operation safety early warning method based on the fusion of image, point cloud, and inertial navigation information according to claim 5, characterized in that, Step S3, the specific methods for state estimation and fusion include: Step S31: Define a state vector that includes the carrier pose and the target global position; Step S32: Establish a kinematic prediction model, use the velocity and angular velocity information of the carrier provided by the inertial measurement unit or inertial odometry to drive the prediction update of the state vector, and treat the target as stationary or in constant motion during the prediction stage. Step S33: Establish a measurement update model. When step S2 provides a valid target distance measurement, the measurement value is converted to the state space, and the innovation quantity between the predicted measurement value and the actual value is calculated. Step S34: Use the extended Kalman filter algorithm to perform state fusion. When measurements are available, combine prediction and measurements to perform state correction. When measurements are missing, only the prediction step in step S32 is executed. Step S35: Calculate and output the continuous distance between the target and the carrier in real time based on the filtered state vector.

7. A railway train operation safety early warning method based on the fusion of image, point cloud, and inertial navigation information according to claim 6, characterized in that, In step S34, before measurement, normalized innovation square (NIS) is used as a consistency gating test. When the NIS value exceeds the preset threshold, the current measurement is determined to be an outlier and rejected, and only the prediction result is retained.

8. A railway train operation safety early warning method based on the fusion of image, point cloud, and inertial navigation information according to claim 7, characterized in that, The risk assessment in step S4 includes enhanced assessment rules for specific targets such as traffic signals, and its processing logic includes: Establishment and confirmation mechanism: The candidate traffic light target is continuously observed for multiple frames. Only after the color is stable, the distance is within the effective range, and a certain number of frames are continuously detected can it be confirmed as the current valid traffic light and its tracking trajectory is initialized. Prediction-measurement consistency check: Compare the radar direct ranging value of the current frame with the filter predicted distance value. If the difference is greater than the threshold, it is determined that the measurement is inconsistent, the update is rejected and the output of the previous frame is maintained; if they are consistent, the update is accepted. Missed detection retention and track estimation: In the event of a short-term missed detection, the current signal light track is not immediately deleted. Instead, the filter pure prediction mode is used to maintain continuous distance and color status output. False detection elimination and clearing: In the candidate stage, if the confirmation conditions cannot be met for multiple consecutive frames, the candidate is eliminated; in the stable tracking stage, if the detection fails for multiple consecutive frames, the current trajectory is cleared. Exit mechanism: When the filter predicts a distance less than the preset proximity threshold, it is determined that the train has passed the signal light, and the system automatically clears the trajectory and resets the status; Color switching confirmation: A hysteresis update strategy is adopted for the output color status. Color switching output is only performed when the new color is observed stably for multiple consecutive frames, in order to suppress single-frame color flicker.

Images