Railway crossing intrusion foreign object trajectory intelligent prediction and early warning system and method

Abstract

The application discloses a railway crossing intrusion foreign matter trajectory intelligent prediction and early warning system and method, and belongs to the technical field of railway traffic operation. The method comprises the following steps: collecting visible light images and infrared images based on a joint sensing device, detecting and identifying intrusion foreign matters based on the visible light images and the infrared images, obtaining foreign matter detection results, performing intrusion foreign matter coordinate inversion based on the foreign matter monitoring results and the external parameters of the visible light camera and the infrared camera, obtaining the world coordinates of the intrusion foreign matters, performing trajectory prediction based on the change of the world coordinates of the intrusion foreign matters, obtaining future trajectory prediction results, generating early warning information based on the foreign matter monitoring results and the future trajectory prediction results, and performing early warning through a field warning device. The application realizes accurate foreign matter detection and motion trend prediction, converts passive detection into active protection by combining a partitioned and graded early warning mechanism, and significantly improves the intelligent early warning capability and emergency response efficiency of the crossing.

Description

Technical Field

[0001] This invention belongs to the field of railway traffic operation technology, specifically relating to an intelligent prediction and early warning system and method for the trajectory of foreign objects intruding into railway crossings. Background Technology

[0002] In recent years, accidents at railway crossings caused by foreign object intrusion have occurred frequently. The current technological system faces three core contradictions: First, the detection methods are limited. Existing vision-based deep learning models mainly rely on object detection and tracking technologies, which can only perform static detection based on location. They cannot effectively predict the future coordinates of foreign objects and are unable to provide timely warnings of potential risks.

[0003] Second, there is a lack of temporal information modeling. Foreign objects at level crossings often exhibit complex movement patterns, such as animals crossing, vehicles running through checkpoints, and plastic bags being blown by the wind. Relying solely on visual information from a single frame or a short time window cannot fully depict the evolution of the target's trajectory, making it difficult for the system to accurately predict the timing and trend of foreign object intrusion.

[0004] Third, there is a lag in emergency response. Because existing detection methods primarily rely on analyzing target movement based on historical frames, they cannot make forward-looking predictions, resulting in a significant time lag between incident occurrence and response, impacting the efficiency of emergency decision-making. Given this current technological situation, how to combine visual detection with temporal networks to achieve sequential prediction of foreign object trajectories and improve the predictive capability for intruding foreign objects has become a current research bottleneck. In the future, level crossing monitoring technology needs to break through the traditional location-based static detection model and construct a comprehensive prediction framework integrating deep learning, temporal modeling, and dynamic early warning to enhance the proactive protection capabilities of railway level crossings.

[0005] With the improvement of computing power and optimization of computational models, vision-based time-series prediction methods have gradually become an important research direction in the field of rail transit safety monitoring. Analyzing video sequence data using deep learning models to predict the trajectory of foreign objects is considered a feasible solution to improve the early warning capability of railway crossing intrusions.

[0006] Currently, research mainly focuses on two types of methods: First, trajectory prediction methods based on traditional temporal modeling. These methods typically rely on statistical methods such as Kalman filtering and Hidden Markov Models (HMMs) to infer the future position of a target based on its historical trajectory. However, these methods have limited ability to model the complexity of target motion patterns and are ill-suited to the uncertain motion characteristics of foreign objects in railway crossing environments, such as the random movement of animals or the nonlinear trajectory of plastic bags blown by the wind. Second, temporal prediction methods based on deep learning. In recent years, temporal neural networks such as Long Short-Term Memory (LSTM), Gated Recurrent Units (GRUs), and Transformers have been introduced into trajectory prediction tasks. Combined with Convolutional Neural Networks (CNNs) to extract target features, they enable the prediction of the future trajectory of foreign objects. Compared to traditional methods, these deep learning models can learn complex motion patterns from large-scale trajectory data, improving prediction accuracy.

[0007] However, this method still has the following shortcomings: (1) Unable to directly model physical constraints. Existing LSTM or Transformer architectures only perform time series prediction based on data-driven methods, lacking the ability to model the physical environment and the motion laws of the target, which may lead to the prediction results deviating from the actual motion trajectory.

[0008] (2) Poor adaptability to abnormal motion. Deep learning models rely on historical trajectory data during training. When encountering rare motion patterns (such as sudden acceleration, sudden stop or non-inertial motion), they are prone to prediction bias.

[0009] (3) High computational cost and limited real-time performance. Complex deep learning models have a large computational load during the inference stage, especially on resource-constrained edge devices, which may make it difficult to meet the low-latency response requirements of railway crossing safety monitoring.

[0010] To address the aforementioned issues, there is an urgent need for an intelligent prediction and early warning system and method for the trajectory of foreign objects intruding into railway crossings, which can solve the problems existing in traditional methods. Summary of the Invention

[0011] The purpose of this invention is to provide an intelligent prediction and early warning system and method for the trajectory of foreign objects intruding into railway crossings. Through multimodal perception, an improved YOLO network, and adaptive weighted trajectory prediction, it achieves accurate detection and prediction of the movement trend of foreign objects. Combined with a zoned and hierarchical early warning mechanism, it transforms passive detection into proactive protection, significantly improving the intelligent early warning capability and emergency response efficiency of railway crossings.

[0012] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A railway crossing intrusion foreign object trajectory intelligent prediction and early warning system includes: a joint sensing device, an edge computing device, an on-site warning device, and a railway central server. The joint sensing device is connected to the edge computing device, and the edge computing device is connected to the on-site warning device and the railway central server. The joint sensing device includes a visible light camera, an infrared camera, and supporting installation equipment. The visible light camera and infrared camera are installed at the railway crossing through the supporting installation equipment, and the visible light camera and infrared camera are pointed at the key monitoring area of ​​the railway crossing to collect visible light images and infrared images of the key monitoring area of ​​the railway crossing. The visible light camera is used for high-definition foreign object detection in daytime environments, and the infrared camera is used for thermal imaging detection at night or in severe weather conditions. The visible light camera, infrared camera, and supporting installation equipment are connected to the edge computing device. The edge computing device is used to acquire visible light images and infrared images, and based on the visible light images and infrared images, it uses an improved YOLO v11 network to detect and identify intruding foreign objects, obtain foreign object detection results, and use an LSTM model combined with a kinematic model to predict the trajectory, thereby obtaining the future trajectory prediction results. The on-site warning equipment includes an on-site LED screen and a loudspeaker. The edge computing device is connected to the on-site LED screen and the loudspeaker. The on-site LED screen is fixed to a conspicuous position at the railway crossing by bolts and is used to display foreign object detection results, future trajectory prediction results, and warning information. The loudspeaker is installed on the matching equipment and is used for sound and light alarms and voice broadcasts.

[0013] Furthermore, the supporting installation equipment includes a level crossing monitoring bracket, a pan-tilt unit, and a power supply system. The level crossing monitoring bracket is installed at the railway level crossing, and the pan-tilt unit is installed on the level crossing monitoring bracket. The visible light camera and infrared camera are installed on the pan-tilt unit. The power supply system and a loudspeaker are installed on the level crossing monitoring bracket. The power supply system is used to supply power to the system, and the pan-tilt unit is connected to the edge computing device.

[0014] Furthermore, the gimbal consists of a horizontal rotation mechanism, a pitch adjustment mechanism, and a gyroscope stabilization system, used for horizontal rotation adjustment, pitch angle adjustment, and stabilization adjustment.

[0015] Furthermore, the power supply system consists of a solar-assisted power supply module, a mains power supply module, and a high-capacity lithium battery, and the solar-assisted power supply module, the mains power supply module, and the high-capacity lithium battery are installed on the crossing monitoring bracket.

[0016] Furthermore, the edge computing device is connected to the railway central server via a network cable and a wireless communication module for data transmission using 5G / 4G cellular networks or fiber optic Ethernet.

[0017] This invention also provides a method for intelligent prediction and early warning of the trajectory of foreign objects intruding into railway crossings, applied to the aforementioned intelligent prediction and early warning system for the trajectory of foreign objects intruding into railway crossings, comprising: Step 1: Acquire visible light and infrared images using a joint sensing device; Step 2: The edge computing device detects and identifies intruding foreign objects based on visible light and infrared images, and obtains the foreign object detection results; Step 3: The edge computing device performs coordinate inversion of the intruding foreign object based on the foreign object monitoring results and the extrinsic parameters of the visible light camera and infrared camera to obtain the world coordinates of the intruding foreign object. Based on the changes in the world coordinates of the intruding foreign object, the device performs trajectory prediction to obtain the future trajectory prediction result. Step 4: The edge computing device generates early warning information based on the foreign object monitoring results and future trajectory prediction results, and issues the warning through on-site warning devices.

[0018] Furthermore, in step 2, the edge computing device detects and identifies intruding foreign objects based on visible light and infrared images, obtaining foreign object detection results, specifically: Edge computing devices use an improved YOLO v11 network based on visible light and infrared images to detect and identify intruding foreign objects.

[0019] Furthermore, improvements were made to the YOLO v11 network, specifically: Dynamic snake convolution is introduced into the backbone of the original YOLO v11 network; A BiFPN module is introduced into the neck network of the original YOLO v11 network; A channel-spatial attention module is introduced into the network detection head of the original YOLO v11 network.

[0020] Furthermore, in step 3, the edge computing device performs coordinate inversion of the intruding foreign object based on the foreign object monitoring results and the extrinsic parameters of the visible light camera and infrared camera to obtain the world coordinates of the intruding foreign object, specifically: Obtain the installation positions of the infrared camera and the visible light camera in the world coordinate system, as well as the position of a pixel in the camera image in the world coordinate system after the gimbal rotates to the current angle; Based on the obtained location, the position of the intruding foreign object in the world coordinate system is determined, and the world coordinates of the intruding foreign object are obtained.

[0021] Furthermore, in step 3, trajectory prediction is performed based on the changes in the world coordinates of the intruding foreign object to obtain the future trajectory prediction result, specifically: Obtain the world coordinates of the intruding foreign object in each frame of infrared or visible light image; Trajectory prediction is performed using an LSTM model combined with a kinematic model based on all world coordinates.

[0022] In summary, the present invention has at least one of the following beneficial technical effects: 1. This invention is based on an improved YOLO v11 network, which improves the accuracy and adaptability of foreign object detection. It adopts dynamic snake convolution to optimize the backbone network of YOLO v11, which improves the feature extraction capability for foreign objects with complex shapes. It adopts BiFPN to enhance multi-scale feature fusion, which improves the detection capability of small foreign objects. It adopts a spatial-channel attention mechanism to optimize the detection head, which improves the attention to key foreign object regions.

[0023] 2. This invention employs LSTM combined with a kinematic model for trajectory prediction. After detecting a foreign object, it uses historical coordinate sequences for time-series regression to improve the accuracy of future trajectory prediction. Simultaneously, the railway crossing area is divided into safety zones, warning zones, and restricted zones, enabling the system to issue early warnings before foreign objects enter critical areas, reducing the probability of accidents. Compared to traditional single-frame detection methods, this method significantly improves the system's real-time warning capability.

[0024] 3. This invention employs a multi-channel warning system, classifying alerts based on the type, trajectory, and threat level of foreign objects. The system can reach relevant personnel in the shortest possible time through various measures, improving the efficiency of handling abnormal situations at level crossings and reducing the probability of accidents. Attached Figure Description

[0025] Figure 1 This is a schematic diagram illustrating the overall concept of the system and method of the present invention; Figure 2 This is a block diagram of the system structure of the present invention; Figure 3 A schematic diagram of the improved YOLO v11 network training and application process; Figure 4 This is a flowchart illustrating the trajectory prediction process based on an LSTM model combined with a kinematic model. Figure 5 This is a schematic diagram of the early warning process; Figure 6 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0027] The overall concept of the system and method of this invention is as follows: Figure 1 As shown, the system will now be described in detail, as follows: Figure 2 As shown, the present invention provides an intelligent prediction and early warning system for the trajectory of foreign objects intruding into railway crossings, comprising: a joint sensing device, an edge computing device, an on-site warning device, and a railway central server. The joint sensing device is connected to the edge computing device, and the edge computing device is connected to the on-site warning device and the railway central server. The joint sensing device includes a visible light camera, an infrared camera, and supporting installation equipment. The visible light camera and infrared camera are installed at the railway crossing through the supporting installation equipment, and the visible light camera and infrared camera are pointed at the key monitoring area of ​​the railway crossing to collect visible light images and infrared images of the key monitoring area of ​​the railway crossing. The visible light camera, infrared camera, and supporting installation equipment are connected to the edge computing device. The edge computing device is used to acquire visible light images and infrared images, and based on the visible light images and infrared images, it uses an improved YOLO v11 network to detect and identify intruding foreign objects, obtain foreign object detection results, and use an LSTM model combined with a kinematic model to predict the trajectory, thereby obtaining the future trajectory prediction results. The on-site warning equipment includes an on-site LED screen and a loudspeaker. The on-site LED screen is bolted to a conspicuous position at the railway crossing and is used to display foreign object detection results, future trajectory prediction results, and warning information. The on-site LED screen is connected to the edge computing device via an HDMI cable. In addition, it has a high-brightness adaptive adjustment function, which can automatically adjust the brightness according to changes in ambient light to ensure clear visibility in strong light or nighttime environments. The loudspeaker is installed on the matching equipment and connected to the edge computing device via an audio interface for sound and light alarms and voice broadcasts. The broadcast content of the loudspeaker can be dynamically adjusted according to the type of foreign object and trajectory prediction, such as "Foreign object intrusion at the crossing, please evacuate immediately!" to improve the warning effect.

[0028] The supporting installation equipment includes a level crossing monitoring bracket, a pan-tilt unit, and a power supply system. The level crossing monitoring bracket is installed at the railway level crossing, and the pan-tilt unit is installed on the level crossing monitoring bracket. The visible light camera and the infrared camera are coaxially fixed on the pan-tilt unit by two 1 / 4-20UNC bolts. By adjusting the horizontal and vertical directions of the pan-tilt unit, the optical axis of the camera is directed to the key monitoring area of ​​the railway level crossing. The visible light camera is used for high-definition foreign object detection in daytime environments, and the infrared camera is used for thermal imaging detection at night or in adverse weather conditions, achieving all-weather monitoring capability. The level crossing monitoring bracket is equipped with the power supply system and a loudspeaker. The power supply system is used to power the system, and the pan-tilt unit is connected to the edge computing device.

[0029] The gimbal consists of a horizontal rotation mechanism, a pitch adjustment mechanism, and a gyroscope stabilization system. It is used for 360° horizontal rotation adjustment, -30° to 90° pitch angle adjustment, and stabilization adjustment to ensure that the camera can still stably capture the target in windy or vibrating environments, thereby improving the reliability of foreign object detection.

[0030] The power supply system consists of a solar-assisted power supply module, a mains power supply module, and a high-capacity lithium battery. The solar-assisted power supply module, mains power supply module, and high-capacity lithium battery are mounted on the crossing monitoring bracket. The high-capacity lithium battery is equipped with a smart power management system (BMS) that can monitor battery status in real time and has overcharge, over-discharge, and short-circuit protection functions. The solar-assisted power supply module optimizes charging efficiency through an MPPT controller, ensuring long-term stable operation of the equipment in remote crossing environments. Under normal circumstances, the mains power supply module is used as the primary power source. If the mains power is interrupted, the system will automatically switch to high-capacity lithium battery power within milliseconds to ensure uninterrupted operation. When the mains power cannot be restored for an extended period, the high-capacity lithium battery is charged via the solar-assisted power supply module. The high-capacity lithium battery uses 36 18650 lithium batteries arranged in three series groups, possessing high energy density and long lifespan characteristics, and outputting 12V DC power. The power protection board of the power supply system has built-in overcharge, over-discharge, and short-circuit protection mechanisms to ensure safe and stable battery operation. The output end adopts the XT60 interface and is equipped with an XT60 to DC 5.5×2.5 adapter cable to adapt to different device interface standards and achieve quick connection and power management.

[0031] The edge computing device is connected to the railway central server via a network cable and a wireless communication module. It is used to transmit data using 5G / 4G cellular networks or fiber optic Ethernet. Its function is to upload the detected foreign object detection results, future trajectory prediction results, and early warning information to the railway central server in real time and connect to the railway dispatching system. In emergency situations, the system can automatically send alarm information to the train driver to remind him to take deceleration or emergency braking measures, thereby improving emergency response efficiency.

[0032] The edge computing device has an efficient heat dissipation system and supports a low-power operating mode to ensure continuous operation for a long time.

[0033] To ensure complete coverage of the entire railway crossing area, this invention installs two sets of joint sensing devices at each railway crossing.

[0034] like Figure 6 As shown, the present invention also provides a method for intelligent prediction and early warning of the trajectory of foreign objects intruding into railway crossings, applied to the aforementioned intelligent prediction and early warning system for the trajectory of foreign objects intruding into railway crossings, comprising: Step 1: Acquire visible light and infrared images using a joint sensing device; Step 2: The edge computing device detects and identifies intruding foreign objects based on visible light and infrared images, and obtains the foreign object detection results; Step 3: The edge computing device performs coordinate inversion of the intruding foreign object based on the foreign object monitoring results and the extrinsic parameters of the visible light camera and infrared camera to obtain the world coordinates of the intruding foreign object. Based on the changes in the world coordinates of the intruding foreign object, the device performs trajectory prediction to obtain the future trajectory prediction result. Step 4: The edge computing device generates early warning information based on the foreign object monitoring results and future trajectory prediction results, and issues the warning through on-site warning devices.

[0035] In step 1, the parameters of the visible light camera and the infrared camera should be calibrated before proceeding. Since this is a conventional technical method, it will not be described in detail here.

[0036] like Figure 3 As shown, in step 2, the edge computing device detects and identifies intruding foreign objects based on visible light and infrared images, obtaining the foreign object detection results, specifically: Edge computing devices utilize visible light and infrared images, employing an improved YOLO v11 network to detect and identify intruding foreign objects. The following section provides a detailed explanation of the improved YOLO v11 network and the specific detection and identification process: This invention addresses the shortcomings of existing technologies by improving the currently advanced YOLO v11 network to adapt it for efficient intrusion detection at railway crossings. The YOLO v11 network employs an end-to-end target detection approach. After an image is input to the network, it is first transformed to a standard size (640, 640) and then normalized to the [0, 1] interval. Next, the image is processed by a backbone network to extract multi-scale features, resulting in multi-channel feature maps of different sizes. Then, the network uses a pyramid-structured neck layer to fuse the obtained multi-scale feature maps, ensuring that large-scale feature maps contain overall target information while small-scale feature maps contain local target information. Next, a target detection head predicts target information, including target category, location, and confidence level. Furthermore, a target segmentation head performs pixel-level segmentation of the target's outline and provides a refined mask. Finally, a non-maximum suppression method is used to remove duplicate detection results, providing the target detection result with the highest confidence level.

[0037] This invention replaces modules in the backbone network of the original YOLO v11 network with dynamic snake convolutions to improve the network's ability to detect and segment diverse targets and complex boundary targets at railway crossings. The computation process of the snake convolution module includes the following steps: (1) Convolution kernel generation: Based on the local structure of the input image, a convolution kernel suitable for the local region is dynamically generated. The shape of the convolution kernel is determined by the changes in the edges and local features in the image, and can be a curve or a polygon. The convolution kernel is not limited to rectangles or squares, but can be bent or stretched according to the image content, so as to better fit the boundaries and details in the image.

[0038] (2) Feature extraction: Based on the generated snake convolution kernel, perform convolution operation with image pixels.

[0039] (3) Dynamic adjustment: During the forward propagation of the entire network, the serpentine convolution kernel will be dynamically adjusted according to different regions of the image. This adjustment not only depends on local image information, but is also affected by the global context, ensuring that the convolution kernel can better adapt.

[0040] The YOLO v11 neck network introduces a Bi-FPN module to better fuse multi-scale target features extracted by the backbone network. In Bi-FPN, each scale feature is assigned learnable weights. During training, the network can learn the contribution of feature maps from different layers to subsequent segmentation networks and automatically adjust the weights. In this process, weighted feature fusion better reflects the characteristics of intruding foreign objects, highlighting information-rich feature layers and suppressing less information-rich feature layers. A fast normalization fusion mechanism is employed in Bi-FPN. The feature process in the Bi-FPN layer after the target features are extracted by the backbone network is as follows: ; In the formula, For the top-down path i Intermediate features of the layer For the bottom-up path i The output features of the layer For learnable weights, Set a small value to avoid numerical instability. Resize( P i ) indicates the network P i Features can be upsampled or downsampled. Conv Depthwise separable convolution can be used here to improve convolution efficiency.

[0041] Furthermore, a channel-spatial attention module is introduced into the network detection head to further improve the head's ability to utilize fused features. The channel-spatial attention module is an attention mechanism module used to enhance the performance of convolutional neural networks, consisting of two key parts: a channel attention module and a spatial attention module. These two modules can be embedded into different layers of the CNN to enhance feature representation. The channel attention module aims to enhance the feature representation of each channel. For the input feature map, global max pooling and global average pooling operations are first performed on each channel to calculate the maximum and average eigenvalues ​​for each channel, generating two vectors containing the number of channels to represent the global maximum and average features of the tunnel appearance defects extracted from each channel. Then, the module inputs the feature vectors obtained from global max pooling and average pooling into a shared fully connected layer to learn the attention weights for each channel. After the network learns the feature importance, the module adaptively determines which channels are more important for the current defect detection task. Finally, the global maximum eigenvector and the average eigenvector are added and normalized using the sigmoid function to obtain the final attention weight vector. The spatial attention module aims to emphasize the importance of different locations in the image. For the feature map enhanced by channel attention, the spatial attention module performs max pooling and average pooling operations along the channel dimension to generate features at different contextual scales. Then, the module concatenates the max-pooled and average-pooled features along the channel dimension to obtain a feature map with contextual information at different scales. This feature map is then processed by convolutional layers to generate spatial attention weights. Similar to the channel attention operation, the network finally uses the sigmoid function to normalize the weights to between 0 and 1.

[0042] This invention provides an embodiment illustrating the construction process of a dataset for training an improved YOLO v11 network. To construct a high-quality dataset of foreign object targets, daytime and nighttime image data are first captured using a visible light camera and an infrared camera, respectively. During the data acquisition phase, a single visible light camera is used to ensure that daytime images are acquired in visible light mode, while an infrared camera is used to acquire images in nighttime and low-light environments, thereby improving the uniformity and efficiency of data acquisition. Foreign object targets are manually arranged in a safe testing area, and a large number of samples of the same type of foreign object are obtained from different perspectives by adjusting the camera angle and the placement of the foreign objects. To enhance the model's generalization ability, additional data needs to be collected under various complex environmental conditions, including low-light environments, strong light interference environments, low-contrast environments, and low-visibility environments. During actual shooting, low-light images are directly acquired at night; strong light is used to illuminate the camera to simulate strong light interference scenarios; and low-contrast and low-visibility environments are created by spraying water mist to ensure the model's robustness under adverse weather and complex lighting conditions.

[0043] like Figure 4 As shown, in step 3, the edge computing device performs coordinate inversion of the intruding foreign object based on the foreign object monitoring results and the extrinsic parameters of the visible light camera and infrared camera to obtain the world coordinates of the intruding foreign object. Based on the changes in the world coordinates of the intruding foreign object, trajectory prediction is performed to obtain the future trajectory prediction result, specifically: Obtain the installation positions of the infrared camera and the visible light camera in the world coordinate system, as well as the position of a pixel in the camera image in the world coordinate system after the gimbal rotates to the current angle; Based on the acquired location, the position of the intruding foreign object in the world coordinate system is determined, and the world coordinates of the intruding foreign object are obtained; Obtain the world coordinates of the intruding foreign object in each frame of infrared or visible light image; Trajectory prediction is performed using an LSTM model combined with a kinematic model based on all world coordinates; The present invention provides the following specific solution process for the above steps: First, set the world coordinate system. W and with ( X w , Y w , Z w () represents the coordinates of a ground point, defining the camera coordinate system. C and with ( X c , Y c , Z c () represents the coordinates from the camera's perspective, defining the pixel coordinate system. Iand with ( u , v () represents pixel coordinates. During camera installation, the camera's position in the world coordinate system can be recorded as () X c0 , Y c0 , Z c0 When the camera is shooting at the corresponding position, the current yaw, pitch, and roll angles of the camera can be determined by the position of the gimbal, which is ( θ y , θ p , θ r Next, spatial position inversion from the camera coordinate system to the world coordinate system is performed. First, the camera's intrinsic parameter matrix is ​​obtained based on the camera's factory calibration information. K , is represented as: ; In the formula, f x as well as f y For the camera x as well as y Focal length in direction, c x as well as c y These are the principal point coordinates of the camera's imaging element. Then, through the intrinsic parameter matrix parameters, it is possible to obtain the coordinates from the pixel coordinates (...). u , v Transformation to the normalized camera coordinate system: ; In the formula, Z c =1 because only directional information can be obtained from a two-dimensional image, but depth information is lacking. Subsequent depth information is calculated based on the assumption of the movement of the foreign object on the ground.

[0044] In the camera coordinate system, a point P c The coordinates can be represented as: ; Next, the coordinates of this point in the camera coordinate system are transformed to the world coordinate system using the camera's spatial position and rotation direction. (Camera rotation matrix) R and translation matrix T It can be defined as follows: ; ; In the formula, the rotation matrix can be determined by the rotation angle obtained from the gimbal, and the translation matrix is ​​the translation amount of the camera relative to the origin of the world coordinate system, that is, the spatial coordinate position of the camera. Based on R and T The transformation from the camera coordinate system to the world coordinate system can be represented as: ; Expanding the above equation, we get: ; Since all points are on the ground, it is possible to determine all foreign object points in the world coordinate system. Z w Since all values ​​are 0, depth information can be obtained, i.e. Z c : ; ; Will Z c Substituting the above transformation formula from camera coordinate system to world coordinate system, we can obtain: ; Based on this, it becomes possible to calculate the pixel value at a point within the camera's field of view. u , v ) to world coordinates ( X w , Y w , Z w The calculation process of ).

[0045] Next, based on the movement of the object's spatial coordinates in the frame sequence captured by the camera, a method is proposed to calculate the object's position in the world coordinate system in each frame using camera parameters and the method described above: ; Based on the assumption of uniform motion, using the nearest... n Frame data is used to calculate the velocity of an object: ; The estimated acceleration is: ; Assuming constant acceleration, the predicted position of the foreign object at the next moment based on the kinematic equations is: .

[0046] The LSTM model predicts the subsequent location of an intruding foreign object by inputting a one-dimensional temporal location vector. It mitigates the vanishing and exploding gradient problems during long sequence training through a gating mechanism. In the LSTM model, the gating mechanism is implemented by training fully connected layers. The calculations for the forget gate, input gate, unit state update, and output gate are shown below: ; ; ; ; ; In the formula, W and b These are the weights and bias matrices of different fully connected layers. C t This is the current cell state. h t This is the current hidden state. x t It is the input one-dimensional matrix. tanh and σ These are the tanh activation function and the sigmoid activation function, and their expressions are as follows: ; ; Set time window T That is, input the past T The spatial coordinates of the frame can be used to obtain the input sequence of the LSTM model, represented as: ; Through the calculation process of the above LSTM model, the next position of the output foreign object can be calculated: ; Then, weighting factors are used. α The location of the foreign object predicted by different methods is adjusted through an adaptive strategy: ; Among them, weighting factors α The prediction error from the previous step is calculated using two methods: ; In the formula, E LTSM and E motion The calculation is as follows: ; ; That is, if the prediction error of the LSTM model is small, its weight is increased, and vice versa.

[0047] This invention also provides an embodiment to illustrate the training process of an LSTM model. This invention trains the LSTM model using artificially generated path data. During data generation, multiple motion modes are comprehensively considered, including uniform linear motion, uniform random curvilinear motion, uniformly accelerated / decelerated linear motion, uniformly accelerated / decelerated random curvilinear motion, random accelerated / decelerated linear motion, and random accelerated / decelerated curvilinear motion. For different motion modes, a series of simulated motion coordinate data are generated, and these paths are scaled to a real-world scale based on camera installation parameters to simulate the data distribution acquired in actual scenarios. Subsequently, training data is acquired according to a set annotation strategy, and this data is used to train the LSTM model to predict the target motion path. This method can effectively improve the prediction accuracy of foreign object trajectories under different motion modes, providing highly reliable data support for subsequent early warning systems. After the network completes training, a traditional kinematic prediction method is combined with a uniform motion model and an acceleration model to calculate the predicted position value at the next moment. The combined method performs preliminary calculations by reading the foreign object sequence position data and sets initial weights to 0.5 and 0.5. Subsequently, as the foreign object location data is updated, the location of the foreign object is further predicted. At the same time, the weights are dynamically adjusted based on the prediction errors of the two methods to make the next prediction.

[0048] like Figure 5 As shown, in step 4, the edge computing device generates early warning information based on the foreign object monitoring results and future trajectory prediction results, and issues the warning through on-site warning devices, specifically: Based on the spatial layout of railway crossings, the area is divided into multiple zones. Based on the foreign object monitoring results and future trajectory prediction results, the possible intrusion location of the foreign object is determined, and early warning information is generated. The early warning information is broadcast through loudspeakers, and the foreign object monitoring results and future trajectory prediction results are displayed on the screen to improve the vigilance of personnel at the crossing.

[0049] The present invention provides an embodiment, specifically as follows: Based on the structure of railway crossings and the safety requirements of train operation, the area surrounding the crossings is divided into four danger levels, and corresponding early warning and response strategies are formulated. Level 1 is designated as the red zone, the track intrusion zone, defined as the area within the track, including the rails, ballast bed, and a certain buffer zone around them. When foreign objects intrude into this area, it may directly affect train operation, posing an extremely high safety risk. For this area, when a potential foreign object intrusion is predicted, the highest level alarm is immediately triggered: the flashing frequency of on-site warning lights increases, loudspeakers continuously broadcast warning information at a high volume, and high-brightness warning information is displayed on the screen in real time. In addition, information is sent to the railway central server via 5G network or Ethernet, which in turn sends emergency warning information to the railway dispatch center, train drivers, and crossing personnel. If the foreign object persists, the system may trigger an emergency braking recommendation. Level 2 is designated as the orange zone, the crossing buffer zone, defined as a buffer zone within 10 meters outside the track. Foreign objects entering this area have a high probability of entering the track, and vigilance should be increased. For this area, when a potential foreign object intrusion is predicted, a high-level warning is triggered. Warning lights flash rapidly, intermittent warnings are played through the loudspeaker, warning information is displayed on the screen, and on-site personnel are notified to investigate and record the object's trajectory to determine further response. Level 3 is the yellow zone, the monitoring and warning zone, defined as a 20-meter radius around the level crossing. It is used to detect potential foreign object intrusions that may threaten railway crossing safety in advance. For this area, when a potential foreign object intrusion is predicted, a medium-level warning is triggered. Warning lights flash at a low frequency, a gentle warning sound is played through the loudspeaker, and information is displayed on the screen. Based on trajectory prediction, if the foreign object shows a tendency to further approach the track, the warning level is increased. Level 4 is the blue zone, the remote monitoring zone. Defined as a 50-meter radius around the railway crossing, it falls within the remote monitoring range. For foreign object intrusions in this area, the system analyzes the remote movement trend of the foreign object and predicts its subsequent movement. For foreign object intrusions in this area, the system records and tracks the foreign object's trajectory but does not actively trigger on-site alarms. If the foreign object is detected moving towards a higher-risk area, the warning level is gradually escalated. Typically, foreign objects tend to gradually approach the crossing area from the outer perimeter. During this process, the warning level tends to increase as the foreign object gets closer. For foreign objects with low risk, the warning level is also lower, reducing the number of times on-site staff need to handle the situation and saving manpower costs.

[0050] Embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0051] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0052] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0053] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0054] Contents not described in detail in this specification are prior art known to those skilled in the art. It is hereby indicated that the above description is intended to help those skilled in the art understand this invention, but does not limit the scope of protection of this invention. Any equivalent substitutions, modifications, improvements, or simplifications of the above descriptions that do not depart from the essential content of this invention fall within the scope of protection of this invention.

Claims

1. A railway crossing intrusion foreign object trajectory intelligent prediction and early warning system, characterized in that, include: The system comprises a combined sensing device, an edge computing device, an on-site warning device, and a railway central server. The combined sensing device is connected to the edge computing device, and the edge computing device is connected to the on-site warning device and the railway central server. The joint sensing device includes a visible light camera, an infrared camera, and supporting installation equipment. The visible light camera and infrared camera are installed at the railway crossing through the supporting installation equipment, and the visible light camera and infrared camera are pointed at the key monitoring area of ​​the railway crossing to collect visible light images and infrared images of the key monitoring area of ​​the railway crossing. The visible light camera is used for high-definition foreign object detection in daytime environments, and the infrared camera is used for thermal imaging detection at night or in severe weather conditions. The visible light camera, infrared camera, and supporting installation equipment are connected to the edge computing device. The edge computing device is used to acquire visible light images and infrared images, and based on the visible light images and infrared images, it uses an improved YOLO v11 network to detect and identify intruding foreign objects, obtain foreign object detection results, and use an LSTM model combined with a kinematic model to predict the trajectory, thereby obtaining the future trajectory prediction results. The on-site warning equipment includes an on-site LED screen and a loudspeaker. The edge computing device is connected to the on-site LED screen and the loudspeaker. The on-site LED screen is fixed to a conspicuous position at the railway crossing by bolts and is used to display foreign object detection results, future trajectory prediction results, and warning information. The loudspeaker is installed on the matching equipment and is used for sound and light alarms and voice broadcasts.

2. The intelligent prediction and early warning system for the trajectory of foreign objects intruding into railway crossings according to claim 1, characterized in that, The supporting installation equipment includes a level crossing monitoring bracket, a pan-tilt unit, and a power supply system. The level crossing monitoring bracket is installed at the railway level crossing. The pan-tilt unit is installed on the level crossing monitoring bracket. The visible light camera and infrared camera are installed on the pan-tilt unit. The power supply system and a loudspeaker are installed on the level crossing monitoring bracket. The power supply system is used to supply power to the system. The pan-tilt unit is connected to the edge computing device.

3. The intelligent prediction and early warning system for the trajectory of foreign objects intruding into railway crossings according to claim 2, characterized in that, The gimbal consists of a horizontal rotation mechanism, a pitch adjustment mechanism, and a gyroscope stabilization system, and is used for horizontal rotation adjustment, pitch angle adjustment, and stabilization adjustment.

4. The intelligent prediction and early warning system for the trajectory of foreign objects intruding into railway crossings according to claim 2, characterized in that, The power supply system consists of a solar-assisted power supply module, a mains power supply module, and a high-capacity lithium battery. The solar-assisted power supply module, the mains power supply module, and the high-capacity lithium battery are installed on the crossing monitoring bracket.

5. The intelligent prediction and early warning system for the trajectory of foreign objects intruding into railway crossings according to claim 1, characterized in that, The edge computing device is connected to the railway central server via a network cable and a wireless communication module, and is used for data transmission using 5G / 4G cellular networks or fiber optic Ethernet.

6. A method for intelligent prediction and early warning of the trajectory of foreign objects intruding into railway crossings, applied to the intelligent prediction and early warning system for the trajectory of foreign objects intruding into railway crossings as described in any one of claims 1-4, characterized in that, include: Step 1: Acquire visible light and infrared images using a joint sensing device; Step 2: The edge computing device detects and identifies intruding foreign objects based on visible light and infrared images, and obtains the foreign object detection results; Step 3: The edge computing device performs coordinate inversion of the intruding foreign object based on the foreign object monitoring results and the extrinsic parameters of the visible light camera and infrared camera to obtain the world coordinates of the intruding foreign object. Based on the changes in the world coordinates of the intruding foreign object, the device performs trajectory prediction to obtain the future trajectory prediction result. Step 4: The edge computing device generates early warning information based on the foreign object monitoring results and future trajectory prediction results, and issues the warning through on-site warning devices.

7. The intelligent prediction and early warning method for the trajectory of foreign objects intruding into railway crossings according to claim 6, characterized in that, In step 2, the edge computing device detects and identifies intruding foreign objects based on visible light and infrared images, obtaining the foreign object detection results, specifically: Edge computing devices use an improved YOLO v11 network based on visible light and infrared images to detect and identify intruding foreign objects.

8. The intelligent prediction and early warning method for the trajectory of foreign objects intruding into railway crossings according to claim 7, characterized in that, Improvements have been made to the YOLO v11 network, specifically: Dynamic snake convolution is introduced into the backbone of the original YOLO v11 network; A BiFPN module is introduced into the neck network of the original YOLO v11 network; A channel-spatial attention module is introduced into the network detection head of the original YOLO v11 network.

9. The intelligent prediction and early warning method for the trajectory of foreign objects intruding into railway crossings according to claim 8, characterized in that, In step 3, the edge computing device performs coordinate inversion of the intruding foreign object based on the foreign object monitoring results and the extrinsic parameters of the visible light camera and infrared camera, to obtain the world coordinates of the intruding foreign object, specifically: Obtain the installation positions of the infrared camera and the visible light camera in the world coordinate system, as well as the position of a pixel in the camera image in the world coordinate system after the gimbal rotates to the current angle; Based on the obtained location, the position of the intruding foreign object in the world coordinate system is determined, and the world coordinates of the intruding foreign object are obtained.

10. The intelligent prediction and early warning method for the trajectory of foreign objects intruding into railway crossings according to claim 9, characterized in that, In step 3, trajectory prediction is performed based on the changes in the world coordinates of the intruding foreign object to obtain the future trajectory prediction result, specifically: Obtain the world coordinates of the intruding foreign object in each frame of infrared or visible light image; Trajectory prediction is performed using an LSTM model combined with a kinematic model based on all world coordinates.

Images