Safety monitoring system for light rail vehicles based on multi-sensor fusion

Abstract

This invention discloses a light rail vehicle safety monitoring system based on multi-sensor fusion, belonging to the field of light rail vehicle safety monitoring technology. The system consists of several functional modules, including: a multimodal perception module, which captures real-time images of the light rail's surrounding environment and delineates visually marked areas; performs 3D scanning of the visually marked areas to generate environmental point cloud data in real time, and uses a Doppler compensation algorithm to acquire the distance to dynamic objects; a data processing module, which uses optical flow and a CNN deformation detection network to calculate object deformation between adjacent frames; performs relative motion calculations based on object shape analysis; outputs object contours based on MaskR-CNN instance segmentation; simultaneously uses a dynamic target clustering algorithm to perform multi-target tracking of the object contours, and outputs the object's 3D coordinates and motion vectors in conjunction with environmental point cloud data; and a fusion and analysis module, which performs time synchronization based on the PTP protocol.

Description

Technical Field

[0001] This invention relates to the field of light rail vehicle safety monitoring technology, specifically a light rail vehicle safety monitoring system based on multi-sensor fusion. Background Technology

[0002] As the core of urban public transportation, the safety of light rail is directly related to passenger lives and the order of urban operations. With the acceleration of urbanization, the passenger flow carried by the light rail system continues to grow, becoming an important link connecting various areas of the city. However, behind the efficient operation of light rail lies a complex safety challenge. According to the analysis of historical accident cases, the safety hazards of light rail mainly stem from factors such as equipment failure, human error, natural disasters, and external interference. According to the "Regulations on the Operation and Management of Urban Rail Transit", flying drones within 100 meters on both sides of the track is illegal, but due to the lack of real-time monitoring methods, similar incidents are repeatedly prohibited but not stopped.

[0003] Existing technologies rely heavily on manual inspections and basic sensors, which have problems such as limited coverage and reliance on experience-based judgment, making it difficult to respond to sudden risks in real time. Traditional methods often use single sensors and lack the ability to fuse and analyze multimodal data, resulting in insufficient accuracy in identifying dynamic objects and a high false alarm rate. In recent years, the frequent occurrence of unauthorized drone flights has exposed the weak real-time capture capability of existing monitoring systems for low-altitude dynamic targets and their lack of adaptability to complex environments. Summary of the Invention

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] The multimodal perception module captures images of the environment around the light rail in real time and delineates visually marked areas; it performs 3D scanning of the visually marked areas, generates environmental point cloud data in real time, and uses a Doppler compensation algorithm to obtain the distance to dynamic objects.

[0006] The data processing module uses optical flow and a CNN deformation detection network to calculate the object deformation between adjacent frames; it performs relative motion calculation based on object shape analysis; it outputs the object contour based on instance segmentation using Mask R-CNN; and it uses a dynamic target clustering algorithm to perform multi-target tracking on the object contour, and outputs the object's 3D coordinates and motion vectors by combining environmental point cloud data.

[0007] The fusion and analysis module uses the PTP protocol for time synchronization and the checkerboard calibration method for joint laser vision calibration. It employs DS evidence theory for multimodal decision-making, and outputs fusion decisions based on input evidence sources. The evidence sources include deformation analysis confidence, laser ranging reliability, and classification network probability.

[0008] The control and response module acquires fusion decisions and implements a tiered early warning mechanism and response.

[0009] Furthermore, the process of defining the visually marked region is as follows:

[0010] A combination of median filtering and bilateral filtering algorithms is used to eliminate high-frequency noise in environmental images. Adaptive histogram equalization is performed based on Retinex theory, lens distortion parameters are pre-calibrated using a calibration method, offset is corrected by online calibration, and an adaptive threshold segmentation algorithm is used to delineate visually marked regions.

[0011] Furthermore, the process of generating environmental point cloud data in real time is as follows:

[0012] LiDAR uses a multi-beam laser emitter to simultaneously emit laser pulses at different vertical angles. Each beam is spaced at a certain angle, and 360° horizontal coverage is achieved through rotating mirrors or electronic scanning. Combined with the pose data of the IMU, the three-dimensional coordinates of each point are calculated.

[0013] Furthermore, the process of obtaining the distance to the dynamic object is as follows:

[0014] Within a single frame scan cycle, the motion trajectory is calculated by numerical integration of data from the IMU and wheel odometer. After achieving microsecond-level time synchronization between the laser point and sensor data using the PTP protocol, linear interpolation is used to distribute the IMU pre-integration results to each laser point to compensate for motion distortion.

[0015] Using the Doppler compensation algorithm, the relative velocity and distance of a dynamic object are calculated by analyzing the frequency shift of the laser echo signal; the reflected signal is collected at the receiving end, and the time difference between transmission and reception and the signal phase change are recorded; the frequency shift is extracted by performing spectrum analysis on the echo signal through fast Fourier transform; the target velocity is calculated based on the frequency shift, and the velocity change is tracked in real time using a Kalman filter.

[0016] Furthermore, the process of calculating object deformation between adjacent frames is as follows:

[0017] S201: Input two consecutive frames of light rail environment images; analyze the displacement of all pixels using an optical flow algorithm; output an optical flow field map;

[0018] S202: The optical flow field map obtained by the thinning optical flow method is used to extract feature maps by inputting adjacent frames and passing them through a pre-trained convolutional neural network;

[0019] S203: Use a specific network structure to compare the feature differences between adjacent frames; the network outputs the deformation ratio of key regions of objects in each frame, and combines it with optical flow results to perform advantage fusion.

[0020] Furthermore, the process of calculating the relative motion of the object is as follows:

[0021] For each detected object region, calculate the variance of the deformation ratio between its adjacent frames, and set a motion threshold to determine motion:

[0022] If the deformation ratio variance is less than or equal to the motion threshold, the object is considered to be relatively stationary.

[0023] If the deformation ratio variance is greater than the motion threshold, then it is marked as a dynamic object;

[0024] Simultaneously, by combining the motion vectors in the optical flow field, the relative velocity and direction of motion of the object are calculated.

[0025] Furthermore, the process of outputting the object contour, as well as the object's three-dimensional coordinates and motion vectors, is as follows:

[0026] Input the environment image and dynamic object candidate regions, and generate candidate boxes through the RPN module of Faster R-CNN;

[0027] Based on the dynamic target clustering algorithm, the object contour data of adjacent frames are input, and the center point coordinates, velocity vector and shape features of each object are extracted; clustering is performed, and the DBSCAN algorithm is used to cluster the targets: according to spatial distance and velocity similarity, objects in consecutive frames are divided into the same trajectory cluster; Kalman filtering is applied to the clustering results to associate the trajectories and predict the target position in the next frame;

[0028] The clustered target trajectory and LiDAR point cloud data are used to calibrate the intrinsic and extrinsic parameters of the camera and LiDAR using a deep learning-based calibration algorithm.

[0029] The 2D center point of the object's outline is matched with the 3D point cloud in the point cloud data using a perspective projection model; the 3D coordinates of the object are calculated using calibration parameters and projection formulas; the 3D velocity vector of the object is calculated by combining the time series of the point cloud data to assess its motion direction and threat level; the output dynamic object list includes the category, 2D outline mask, 3D coordinates and motion vector of each object, as well as the trajectory information of the object in consecutive frames.

[0030] Furthermore, the laser vision joint calibration process is as follows:

[0031] Using the corner points of a checkerboard calibration board as 2D / 3D feature points, the camera intrinsic parameters are calculated using the Zhang Zhengyou calibration method, and the camera extrinsic parameters are determined based on multi-view images. Point cloud data is obtained by scanning the checkerboard with a LiDAR, and its installation pitch angle θ and height h are inferred. The point cloud is transformed to the camera coordinate system, and the rotation and translation matrices of the LiDAR to the camera are constructed. The matrix parameters are optimized by minimizing the projection error between the point cloud and the image corner points. The overlap is verified by projecting the point cloud onto the image plane, achieving multimodal spatial alignment.

[0032] Furthermore, the output fusion decision process is as follows:

[0033] By transforming CNN deformation confidence, laser ranging stability, and MaskR-CNN classification probability into a BPA function and calculating the conflict factor K, the Dempster rule is used to synthesize evidence and generate a comprehensive confidence level, thereby enabling graded decision-making for high-threat, potential-threat, and safe objects.

[0034] Furthermore, the tiered early warning mechanism and response process are as follows:

[0035] The threat threshold is dynamically adjusted, and the mapping rules are optimized based on historical data; the fusion decision results are monitored in real time and adjusted accordingly.

[0036] The light rail vehicle safety monitoring system based on multi-sensor fusion provided by this invention has the following beneficial effects:

[0037] (1) This invention achieves spatial alignment between LiDAR and camera through checkerboard calibration method, and eliminates motion distortion by combining Doppler compensation algorithm to ensure that the three-dimensional coordinates and motion vector calculation accuracy of dynamic objects, such as UAVs, reaches the centimeter level; it uses optical flow algorithm to analyze pixel-level motion vectors, and combines pre-trained CNN network to extract feature differences to accurately identify the deformation ratio of objects and distinguish between static backgrounds and dynamic threats; by converting CNN deformation confidence, laser ranging stability and MaskR-CNN classification probability into BPA function and calculating conflict factor K, evidence is synthesized using Dempster rule to generate comprehensive confidence level; this method can effectively reduce false alarm rate and realize graded decision-making for high-threat objects, such as UAVs approaching the orbit.

[0038] (2) This invention optimizes the threat level classification based on the environment to avoid misjudgment caused by sensor performance fluctuations; it uses a combination of median filtering and bilateral filtering algorithms to eliminate high-frequency noise, and combines CLAHE adaptive histogram equalization to improve image quality in low visibility scenarios such as tunnels, rain, fog, and night; it achieves microsecond-level time synchronization based on the PTP protocol and combines IMU for motion distortion compensation to ensure the spatiotemporal consistency of point cloud data in high-speed scenarios; it processes data in real time through an embedded system, such as an ARM processor, and combines a cloud computing platform to achieve historical data learning and model iteration to continuously optimize threat identification rules. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the system flow of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0041] Example 1

[0042] Please see Figure 1 Embodiment 1 of this application provides a light rail vehicle safety monitoring system based on multi-sensor fusion, the system comprising:

[0043] The multimodal perception module captures images of the environment around the light rail in real time and delineates visually marked areas; it performs 3D scanning of the visually marked areas, generates environmental point cloud data in real time, and uses a Doppler compensation algorithm to obtain the distance to dynamic objects.

[0044] Real-time capture of images of the environment surrounding the light rail:

[0045] Multiple sets of high-resolution industrial cameras are deployed at the front and sides of the light rail car to cover the forward track and the trackside areas on both sides of the light rail.

[0046] Front-facing main camera: Equipped with optical image stabilization and a telephoto lens with a focal length of ≥50mm, supporting HDR mode and adapting to sudden changes in lighting inside and outside the tunnel;

[0047] Side-facing wide-angle camera: horizontal field of view ≥120°, used to capture dynamic targets such as pedestrians and vehicles beside the track;

[0048] Infrared auxiliary camera: integrates an uncooled infrared detector with a wavelength of 8-14μm to compensate for rain, fog, and low visibility at night.

[0049] Among them, PTP is used to achieve μs-level time synchronization of multiple cameras, ensuring the spatiotemporal consistency of environmental data at the same moment, and the triggering frequency is dynamically adjusted according to vehicle speed;

[0050] A combination of median filtering and bilateral filtering algorithms is used to eliminate high-frequency noise such as raindrops and snowflakes; adaptive histogram equalization is performed based on Retinex theory to solve the problems of strong backlight at tunnel entrances and exits and low illumination inside the tunnel; the lens distortion parameters are pre-calibrated using a calibration method, and online calibration is used to correct the offset caused by mechanical vibration.

[0051] For example:

[0052] When light rail travels in the rain, the camera feed is filled with high-frequency noise dots. Median filtering removes these randomly distributed noise points by replacing each pixel value with the median of all pixel values ​​within its neighborhood window, effectively removing isolated noise points without affecting edge information. However, median filtering alone may not completely eliminate all types of noise, especially when the noise is dense. Therefore, bilateral filtering is further applied. Bilateral filtering considers not only spatial distance but also pixel value differences, meaning it can smooth the image while preserving important edge features. Even in conditions of dense raindrops or snowflakes, it ensures the clarity of key object outlines in the image, providing high-quality input for subsequent object detection and classification.

[0053] Define visual marker areas:

[0054] Real-time images of the light rail surroundings are captured using high-resolution cameras, and an adaptive threshold segmentation algorithm is used to delineate visually marked areas, with a focus on key areas such as the trackside, signal equipment, and dynamic objects, such as drones and pedestrians.

[0055] Median filtering or Gaussian filtering is used to preprocess environmental images to eliminate salt-and-pepper noise and Gaussian noise caused by vibration and lighting changes. For example, 5×5 Gaussian kernel smoothing is used to preserve edge information along the track and for drone targets. To address the uneven lighting problem during light rail operation, histogram equalization or grayscale stretching is used to enhance image contrast and improve the robustness of subsequent threshold segmentation. For example, at the tunnel entrance, the CLAHE algorithm is used for contrast-limited adaptive histogram equalization to improve details in dark areas.

[0056] The image is divided into N×N local regions, such as 11×11 or 15×15 pixel blocks. The average gray value of the pixels in the neighborhood is calculated as the baseline threshold using the adaptiveThreshold function of OpenCV. A Gaussian kernel (σ=0.5) is used to sum the neighborhood pixels in a weighted manner to reduce the edge blurring effect.

[0057] A compensation factor C (usually between 2 and 10) is introduced based on the benchmark threshold, as shown in the formula:

[0058] Formula 1: T(x,y)=μlocal(x,y)-C

[0059] Formula 2: T(x,y)=Gσ*I(x,y)+C

[0060] In the formula, T(x,y) is the dynamic threshold at position (x,y), used for tasks such as image segmentation or feature detection to distinguish between foreground and background; μ local(x,y) represents the local average gray value at position (x,y), obtained by calculating the average gray value of pixels within a certain area surrounding this point; the local average gray value reflects the image brightness characteristics near this position; C is a compensation factor, a constant, typically ranging from 2 to 10, with the threshold dynamically adjusted according to the scene to adapt to different lighting conditions; increasing the C value can suppress background interference when the light is strong; decreasing the C value can prevent target misses when the light is low; G σ σ represents the Gaussian kernel function, where σ is the standard deviation, used to control the smoothing effect of the Gaussian filter. Gaussian filtering is a commonly used image smoothing technique that reduces image noise while preserving edge information; * represents the convolution operator, indicating the Gaussian kernel function G. σ Perform a convolution operation with the image I(x,y); the result of the convolution is the image after smoothing; I(x,y) is the gray value of the original image at position (x,y);

[0061] Among them, C is dynamically adjusted according to the scene: when the light is strong, the C value is increased to suppress background interference, and when the light is low, the C value is decreased to avoid missing the target;

[0062] The grayscale value of each pixel is compared with its corresponding threshold to generate a binarized image.

[0063] If the pixel value is greater than or equal to the threshold, it is marked as foreground, and the candidate region is visually labeled with a value of 255.

[0064] If the pixel value is less than the threshold, mark it as background and assign a value of 0;

[0065] The segmentation results are closed by first dilating and then eroding to fill the holes, and small noise areas, such as isolated points with an area of ​​less than 50 pixels, are removed by connected component analysis. Finally, a stable visual marker region outline is output.

[0066] Perform a 3D scan of the visually marked region:

[0067] High-precision environmental point cloud data is generated in real time using LiDAR multi-beam scanning technology; and point cloud deformation errors caused by high-speed movement of light rail are eliminated by using motion distortion compensation algorithm based on incremental motion model of IMU / wheel odometer.

[0068] Multi-beam laser scanning and point cloud generation:

[0069] LiDAR uses multi-beam laser emitters, such as 16-line, 32-line, and 128-line laser emitters, to simultaneously emit laser pulses at different vertical angles, such as -15° to +15°. For example, the Velodyne VLP-16 LiDAR covers 30° vertically, with each beam spaced 2° apart. It achieves 360° horizontal coverage through rotating mirrors or electronic scanning, with a single frame scan time of approximately 0.1 seconds. After the laser beam strikes an object and reflects, the receiver records the echo time difference and intensity information. Combined with pose data from an IMU and a wheeled odometry system, the three-dimensional coordinates of each point are calculated. The raw data is parsed and stored in LAS, XYZ, or PCD formats, containing location, reflection intensity, and timestamp information. For example, binary data packets transmitted via UDP protocol need to be decoded into a processable point cloud structure.

[0070] Motion distortion compensation algorithm flow:

[0071] Within a single frame scan cycle, using IMU accelerometer and gyroscope data, combined with wheel odometry pulse counting, incremental motion trajectory, position, attitude, and velocity are calculated through numerical integration; precise timestamps are assigned to each laser point, based on the scan angle or hardware triggering, and synchronized with IMU / wheel odometry data at the microsecond level via the PTP protocol; linear interpolation or spherical linear interpolation is used to distribute the IMU pre-integration results to each laser point according to the timestamp;

[0072] The local coordinate system of each laser point, i.e. the LiDAR coordinate system, is transformed to the global coordinate system. Factor graph optimization, such as the GTSAM framework, is used to jointly optimize the LiDAR point cloud matching residual, IMU pre-integration constraint and wheel odometry error, so as to minimize the joint error of global pose and motion distortion.

[0073] Distance acquisition for dynamic objects:

[0074] By combining the Doppler compensation algorithm and analyzing the frequency shift of the laser echo signal, the relative velocity and distance of dynamic objects can be accurately calculated with centimeter-level accuracy. This algorithm can effectively distinguish between static backgrounds and dynamic obstacles, solving the error accumulation problem of traditional ranging methods in high-speed scenarios.

[0075] The receiver acquires the reflected signal and records the time difference between transmission and reception, as well as the signal phase change. It should be noted that in high-speed scenarios, the laser echo signal may experience a Doppler frequency shift due to target motion, requiring adjustment of the received signal. The echo signal is then subjected to spectral analysis using Fast Fourier Transform or Wavelet Transform to extract the frequency shift Δf. For example, the phase difference least squares algorithm can be used to model the phase difference of multiple subcarriers and solve for the Doppler frequency shift. The target velocity v is calculated based on the frequency shift, and a Kalman filter or adaptive filter is used to track the velocity change in real time. For instance, in satellite communication systems, Kalman filtering is used to control the Doppler error within ±5Hz.

[0076] The data processing module uses optical flow and a CNN deformation detection network to calculate the object deformation between adjacent frames; it performs relative motion calculation based on object shape analysis; it outputs the object contour based on instance segmentation using Mask R-CNN; and it uses a dynamic target clustering algorithm to perform multi-target tracking on the object contour, and outputs the object's 3D coordinates and motion vectors by combining environmental point cloud data.

[0077] Calculate object deformation between adjacent frames:

[0078] Calculate pixel-level motion vectors between consecutive image frames, that is, the direction and magnitude of movement of each point between two frames;

[0079] S201: Input two consecutive frames of light rail environment images; analyze the displacement of all pixels using optical flow algorithm; output optical flow field map, which is represented as a two-dimensional motion vector of each pixel, reflecting the motion changes between adjacent frames;

[0080] S202: Refine the optical flow field map obtained by the optical flow method to improve the ability to analyze nonlinear deformation in complex scenes; extract feature maps by inputting adjacent frames into a pre-trained convolutional neural network;

[0081] S203: Use a specific network structure to compare the feature differences between adjacent frames; the network outputs the deformation ratio of key regions of the object in each frame, and combines the optical flow results to integrate the advantages of the two methods and improve the deformation detection accuracy.

[0082] The advantages of the two methods are: combining the precise pixel-level motion vector information provided by optical flow and the advanced feature difference analysis capabilities provided by CNN, so as to achieve the goal of accurately capturing minute displacement changes and effectively identifying and quantifying the nonlinear deformation of objects, significantly improving the overall accuracy of deformation detection, especially in complex and ever-changing practical application scenarios, particularly in the process of monitoring the environment around light rail vehicles.

[0083] Calculation of relative motion of objects:

[0084] Determine whether an object is stationary or moving dynamically relative to the light rail; for each detected object region, calculate the variance of the deformation ratio between adjacent frames, and set a motion threshold (15%):

[0085] If the variance of the deformation ratio is ≤15%, the object is considered to be relatively stationary.

[0086] If the variance of the deformation ratio is greater than 15%, it is marked as a dynamic object;

[0087] Simultaneously, by combining the motion vectors in the optical flow field, the relative velocity and direction of motion of the object are calculated to assist in subsequent tracking;

[0088] Exercise threshold: Considering safety factors, we tend to set a conservative threshold of 15%, and will adjust it in a timely manner based on on-site feedback and effect evaluation.

[0089] Output object outline:

[0090] The input environment image and dynamic object candidate regions are used to generate candidate boxes through the RPN module of Faster R-CNN. The candidate boxes cover the areas that may contain objects, such as drones and pedestrians.

[0091] Classification and mask generation:

[0092] For each candidate bounding box, the MaskR-CNN classifier is used to determine the object category, such as drone or vehicle; a binary mask is generated to accurately segment the object contour; morphological operations such as dilation and erosion are performed on the segmentation results to remove noise and ensure smooth contour boundaries; the mask image and category label of each object are output.

[0093] Multi-target tracking and 3D coordinate calculation:

[0094] Based on the dynamic target clustering algorithm, the input is the object contour data (mask image) and optical flow motion vector of adjacent frames; the center point coordinates, velocity vector and shape features of each object are extracted; clustering is performed, and the DBSCAN algorithm is used to cluster the targets: according to spatial distance (Euclidean distance) and velocity similarity, the objects in consecutive frames are divided into the same trajectory cluster, and noise points are filtered.

[0095] Kalman filtering is applied to the clustering results to correlate trajectories and predict the target location in the next frame.

[0096] Based on a dynamic target clustering algorithm, objects in consecutive frames are divided into different trajectory clusters according to their spatial distance and velocity similarity. For each trajectory cluster, information such as its center point coordinates, velocity vector, and shape features are extracted as an initial state vector. A Kalman filter is then used to predict the state of each trajectory cluster. The Kalman filter is a recursive algorithm that uses the state estimate from the previous moment and the observed value at the current moment to update the current state estimate. During each frame processing, a state transition model is first used to predict the position and velocity of the next state. For example, assuming the target is moving at a constant velocity in a straight line, the position in the next frame can be predicted based on the current position and velocity. The actual observed target position is compared with the predicted value, and the difference between the two is calculated. The predicted value is adjusted based on this difference to obtain a more accurate target position estimate. By continuously repeating the above prediction and update steps, the Kalman filter not only provides a prediction of the target's future position but also effectively reduces position estimation errors caused by sensor noise or external interference, significantly improving the stability and accuracy of target tracking. The corrected state estimate is then used to predict the target's position in the next frame.

[0097] 3D coordinate and motion vector calculation:

[0098] After clustering, the target trajectory and LiDAR point cloud data are used to calibrate the intrinsic and extrinsic parameters of the camera and LiDAR using a deep learning-based calibration algorithm to ensure coordinate system consistency. The 2D center point of the object contour is matched with the 3D point cloud in the point cloud data using a perspective projection model. The 3D coordinates of the object are calculated using calibration parameters and projection formulas. Combined with the time series of the point cloud data, the 3D velocity vector of the object is calculated to assess its motion direction and threat level. The output is a dynamic object list containing the category, 2D contour mask, 3D coordinates and motion vector of each object, as well as the trajectory information of the object in consecutive frames.

[0099] The fusion and analysis module uses the PTP protocol for time synchronization and the checkerboard calibration method for joint laser vision calibration. It employs DS evidence theory for multimodal decision-making, and outputs fusion decisions based on input evidence sources. The evidence sources include deformation analysis confidence, laser ranging reliability, and classification network probability.

[0100] Time synchronization based on the PTP protocol:

[0101] Time synchronization based on the PTP protocol: The master clock sends a Sync message to the slave clock and records the sending timestamp t1; after receiving the Sync message, the slave clock records the receiving timestamp t2; the slave clock sends a Delay_Req message to the master clock, the master clock records the receiving timestamp t3, and sends a Delay_Resp message to reply to the slave clock, including t3; after receiving the Delay_Resp message, the slave clock records the receiving timestamp t4; and calculates the average path delay and time offset. The slave clock corrects its local time based on the calculated Delay and Offset, so that the master and slave clocks are synchronized to sub-microsecond accuracy.

[0102] Laser vision joint calibration:

[0103] A transformation relationship between the LiDAR coordinate system and the camera coordinate system was established to achieve spatial alignment of multimodal data. A checkerboard calibration board was used, with its black and white grid corner points serving as precise 2D / 3D feature points. Corner point coordinates were extracted from the checkerboard image, and camera intrinsic parameters were calculated using the Zhang Zhengyou calibration method. The rotation matrix and translation vector of the camera relative to the world coordinate system were determined using multi-view images. Point cloud data was acquired by scanning the checkerboard calibration board with the LiDAR, and the installation elevation angle θ and installation height h of the LiDAR were inferred from the point cloud data. The LiDAR point cloud data was transformed to the camera coordinate system to obtain the rotation and translation matrices from the LiDAR to the camera. The values ​​of the rotation and translation matrices from the LiDAR to the camera were optimized by minimizing the projection error between the point cloud data and the image corner points. The LiDAR point cloud data was projected onto the camera image plane using calibration parameters to verify the overlap.

[0104] Output fusion decision:

[0105] Deformation analysis confidence: from the CNN deformation detection network in the data processing module; m1 represents the confidence of the deformation ratio D of an object between adjacent frames;

[0106] Laser ranging reliability: derived from lidar distance measurement; representing the stability m2 of lidar point cloud data;

[0107] Classification network probability: derived from Mask R-CNN instance segmentation; representing the predicted probability m3 of object category (e.g., drone, pedestrian);

[0108] The confidence level of each evidence source is transformed into a BPA function, the conflict factor K between evidence sources is calculated, and evidence is synthesized using the Dempster rule; a fusion decision is generated based on the overall confidence level.

[0109] High trust level (>0.8): Marked as a high-threat object, such as a drone approaching its orbit;

[0110] Medium trust level (0.5–0.8): Marked as a potential threat, triggering an alert;

[0111] Low trust level (<0.5): classified as a safe object, such as a static obstacle;

[0112] The control and response module acquires fusion decisions and implements a tiered early warning mechanism and response.

[0113] Acquire fused decision results, spatiotemporally aligned data, and environmental context information to classify threat levels:

[0114] Table 1: Threat Level Classification Table

[0115]

[0116] Threat thresholds can be adjusted by combining environmental context information. For example, the reliability of lidar decreases in rainy or snowy weather, so the m threshold needs to be increased. Mapping rules can be optimized by learning from historical data. For example, if the false alarm rate of drones is high in certain areas, the threat level can be dynamically reduced.

[0117] Through real-time monitoring, the fusion decision result m is continuously received and compared with the preset threshold; the threshold is dynamically adjusted according to the current environment (such as nighttime, strong light); for example, if the reliability of the lidar decreases at night, m ​​needs to be increased to 0.8 to trigger a level 3 warning.

[0118] Low threat (Level 1 alert):

[0119] Operation: Logs are recorded, requiring no manual intervention.

[0120] Technical implementation: Quickly write to the local database using an embedded system (such as an ARM processor);

[0121] Medium Threat (Level 2 Alert):

[0122] Operation: Audible and visual alarm + operator interface prompts;

[0123] Technical Implementation:

[0124] Audible and visual alarm: The buzzer and LED indicator are controlled via GPIO;

[0125] User interface: Displays real-time images and locations of threat targets on the HMI (Human Machine Interface);

[0126] High threat (Level 3 alert):

[0127] Operation: Emergency braking + automatic intervention (such as laser interference, radio frequency blocking);

[0128] Technical Implementation:

[0129] Emergency braking: Sends an emergency stop command to the light rail control system via the CAN bus;

[0130] Automatic intervention:

[0131] Laser interference: Activate the high-energy laser module and carry out the attack according to the preset strategy, such as a three-step destruction.

[0132] Radio frequency jamming: Directional interference with the drone's communication link, forcing it to return or land.

[0133] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0134] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0135] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A light rail vehicle safety monitoring system based on multi-sensor fusion, characterized in that, The system includes: The multimodal perception module captures images of the environment around the light rail in real time and delineates visually marked areas; it performs 3D scanning of the visually marked areas, generates environmental point cloud data in real time, and uses a Doppler compensation algorithm to obtain the distance to dynamic objects. The data processing module employs optical flow and a CNN deformation detection network to calculate object deformation between adjacent frames; it performs relative motion calculation based on object shape analysis; it outputs object contours through instance segmentation based on Mask R-CNN; and it simultaneously uses a dynamic target clustering algorithm to perform multi-target tracking on the object contours, combining environmental point cloud data to output the object's 3D coordinates and motion vectors. The process of calculating the relative motion of the object is as follows: For each detected object region, the deformation ratio variance between adjacent frames is calculated, and a motion threshold is set to determine motion: if the deformation ratio variance is less than or equal to the motion threshold, the object is considered to be relatively stationary; if the deformation ratio variance is greater than the motion threshold, it is marked as a dynamic object; at the same time, the relative velocity and direction of motion of the object are calculated by combining the motion vector in the optical flow field. The process of outputting the object contour, as well as the object's three-dimensional coordinates and motion vectors, is as follows: Input the environment image and dynamic object candidate regions, and generate candidate boxes through the RPN module of Faster R-CNN; Based on the dynamic target clustering algorithm, the object contour data of adjacent frames are input, and the center point coordinates, velocity vector and shape features of each object are extracted; clustering is performed, and the DBSCAN algorithm is used to cluster the targets: according to spatial distance and velocity similarity, objects in consecutive frames are divided into the same trajectory cluster; Kalman filtering is applied to the clustering results to associate the trajectories and predict the target position in the next frame; The clustered target trajectory and LiDAR point cloud data are used to calibrate the intrinsic and extrinsic parameters of the camera and LiDAR using a deep learning-based calibration algorithm. The 2D center point of the object's outline is matched with the 3D point cloud in the point cloud data using a perspective projection model; the 3D coordinates of the object are calculated using calibration parameters and projection formulas; the 3D velocity vector of the object is calculated by combining the time series of the point cloud data to assess its motion direction and threat level; the output dynamic object list includes the category, 2D outline mask, 3D coordinates and motion vector of each object, as well as the trajectory information of the object in consecutive frames. The fusion and analysis module uses the PTP protocol for time synchronization and the checkerboard calibration method for joint laser vision calibration. It employs DS evidence theory for multimodal decision-making, and outputs fusion decisions based on input evidence sources. The evidence sources include deformation analysis confidence, laser ranging reliability, and classification network probability. The control and response module acquires fusion decisions and implements a tiered early warning mechanism and response.

2. The light rail vehicle safety monitoring system based on multi-sensor fusion according to claim 1, characterized in that, The process of defining the visual marker region is as follows: A combination of median filtering and bilateral filtering algorithms is used to eliminate high-frequency noise in environmental images. Adaptive histogram equalization is performed based on Retinex theory, lens distortion parameters are pre-calibrated using a calibration method, offset is corrected by online calibration, and an adaptive threshold segmentation algorithm is used to delineate visually marked regions.

3. The light rail vehicle safety monitoring system based on multi-sensor fusion according to claim 2, characterized in that, The process of generating environmental point cloud data in real time is as follows: LiDAR uses a multi-beam laser emitter to simultaneously emit laser pulses at different vertical angles. Each beam is spaced at a certain angle, and the lasers cover 360° horizontally using a rotating mirror or electronic scanning method. Combined with the pose data of the IMU, the three-dimensional coordinates of each point are calculated.

4. The light rail vehicle safety monitoring system based on multi-sensor fusion according to claim 3, characterized in that, The process of obtaining the distance to the dynamic object is as follows: Within a single frame scan cycle, the motion trajectory is calculated by numerical integration of IMU and wheel odometer data. After time synchronization of laser point and sensor data using the PTP protocol, linear interpolation is used to distribute the IMU pre-integration results to each laser point. Using the Doppler compensation algorithm, the relative velocity and distance of a dynamic object are calculated by analyzing the frequency shift of the laser echo signal; the reflected signal is collected at the receiving end, and the time difference between transmission and reception and the signal phase change are recorded; the frequency shift is extracted by performing spectrum analysis on the echo signal through fast Fourier transform; the target velocity is calculated based on the frequency shift, and the velocity change is tracked in real time using a Kalman filter.

5. The light rail vehicle safety monitoring system based on multi-sensor fusion according to claim 1, characterized in that, The process of calculating object deformation between adjacent frames is as follows: S201: Input two consecutive frames of light rail environment images; analyze the displacement of all pixels using an optical flow algorithm; output an optical flow field map; S202: The optical flow field map obtained by the thinning optical flow method is used to extract feature maps by inputting adjacent frames and passing them through a pre-trained convolutional neural network; S203: Compare feature differences between adjacent frames using a specific network structure; The network outputs the deformation ratio of key areas of the object in each frame, and combines it with the optical flow results for optimal fusion.

6. The light rail vehicle safety monitoring system based on multi-sensor fusion according to claim 1, characterized in that, The laser vision joint calibration process is as follows: Using the corner points of a checkerboard calibration board as 2D / 3D feature points, the camera intrinsic parameters are calculated using the Zhang Zhengyou calibration method, and the camera extrinsic parameters are determined based on multi-view images. Point cloud data is obtained by scanning the checkerboard with a LiDAR, and its installation pitch angle θ and height h are inferred. The point cloud is transformed to the camera coordinate system, and the rotation and translation matrices of the LiDAR to the camera are constructed. The matrix parameters are optimized by minimizing the projection error between the point cloud and the image corner points. The point cloud is projected onto the image plane to verify the overlap and perform multimodal spatial alignment.

7. The light rail vehicle safety monitoring system based on multi-sensor fusion according to claim 1, characterized in that, The process of output fusion decision-making is as follows: By transforming CNN deformation confidence, laser ranging stability, and MaskR-CNN classification probability into a BPA function and calculating the conflict factor K, Dempster rules are used to synthesize evidence and generate a comprehensive confidence level for classifying high-threat, potential-threat, and safe objects.

8. The light rail vehicle safety monitoring system based on multi-sensor fusion according to claim 1, characterized in that, The tiered early warning mechanism and response process are as follows: The threat threshold is dynamically adjusted, and the mapping rules are optimized based on historical data; the fusion decision results are monitored in real time and adjusted accordingly.

Images