Railway foreign object intrusion detection method and device

Abstract

The application relates to a track foreign matter intrusion detection method and device, wherein the method comprises the following steps: acquiring image data of a fixed camera beside a track, detecting a target in the image, acquiring point cloud data of a vehicle-mounted laser radar on a train, detecting a target in the point cloud data, establishing a space coordinate system, respectively converting the position of the target in the point cloud data and the pixel coordinate of the target in the image into a coordinate position in the space coordinate system, judging whether the two coordinate positions match, if yes, reconstructing the target according to the point cloud data to obtain a reconstruction model, obtaining a depth map point cloud according to the image data, matching the reconstruction model and the depth map point cloud to obtain a complete target point cloud model, extracting the features of the target point cloud model and the features of the image data, fusing the features of the target point cloud model and the features of the image data, and determining whether foreign matter intrusion occurs on the track.

Description

[0001] This application claims priority to Chinese Patent Application No. 202511127406.0, filed on August 12, 2025, entitled "A Method and Apparatus for Detecting Foreign Object Intrusion on an Orbit", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of track foreign object detection, specifically to a method and apparatus for detecting track foreign object intrusion. Background Technology

[0003] With the rapid development of rail transit and the continuous increase in train speeds, track safety has become a key factor in ensuring transportation efficiency and passenger safety. However, on railway and subway lines, foreign object intrusion (such as falling rocks, pedestrians, vehicles, fallen trees, or other obstacles) can lead to serious accidents such as train collisions and derailments, causing significant casualties and economic losses. Therefore, efficient and accurate track foreign object intrusion detection technology is of great importance to ensuring train operation safety.

[0004] Currently, foreign object detection on railway tracks mainly employs two technical approaches: trackside detection and vehicle-mounted detection. Trackside detection systems are typically based on equipment such as video surveillance, infrared sensors, and lidar. While they can achieve monitoring within a certain range, they suffer from inherent drawbacks such as dense equipment deployment, high maintenance costs, and poor environmental adaptability. Especially under adverse weather conditions such as rain, snow, fog, and haze, the performance of optical sensors deteriorates significantly, leading to increased false alarm and false negative rates.

[0005] Vehicle-mounted detection systems primarily rely on equipment such as cameras and millimeter-wave radar installed at the front of the train. While they enable mobile monitoring, they also face numerous technical bottlenecks. Vibrations from high-speed train operation can affect sensor stability, and sudden changes in light when entering or exiting tunnels can cause the vision system to malfunction. More importantly, the detection range of vehicle-mounted systems is limited, often leaving insufficient braking time for the train when obstacles are detected, making it difficult to effectively prevent accidents.

[0006] Most current systems employ a single-platform detection approach, lacking a multi-source information fusion mechanism, making it difficult to meet the detection requirements under all weather and road conditions. Therefore, designing a track foreign object intrusion detection method that utilizes coordinated trackside and onboard systems is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a method and device for detecting foreign object intrusion on tracks, which achieves accuracy, reliability and environmental adaptability in detecting foreign object intrusion on tracks.

[0008] This application first discloses a method for detecting foreign object intrusion into orbital tracks, the specific steps of which are as follows:

[0009] Acquire image data from a fixed camera beside the track, and detect targets in the image based on the image data;

[0010] Acquire point cloud data from the onboard lidar of the train, and detect targets in the point cloud data based on the point cloud data;

[0011] A spatial coordinate system is established, and the position of the target in the point cloud data and the pixel coordinates of the target in the image are converted into coordinate positions in the spatial coordinate system. It is determined whether the two coordinate positions match. If they match, the target is reconstructed based on the point cloud data to obtain a reconstructed model. A depth map point cloud is obtained based on the image data. The reconstructed model and the depth map point cloud are matched to obtain a complete target point cloud model. Features of the target point cloud model and features of the image data are extracted. The features of the target point cloud model and the features of the image data are fused to determine whether there is a foreign object intrusion on the track.

[0012] Preferably, the step of establishing a spatial coordinate system, which converts the position of the target in the point cloud data and the pixel coordinates of the target in the image into coordinate positions in the spatial coordinate system, includes:

[0013] A spatial coordinate system is established with the position of a fixed camera as the origin of the spatial coordinate system, a straight line parallel to the direction of the track extension as the x-axis, and a straight line perpendicular to the direction of the track extension and located on the horizontal plane as the y-axis.

[0014] The current train position is obtained, the point cloud data includes the relative position of the target and the train, the target position is determined based on the relative position and the train position, and the target position is converted into the coordinate position of the spatial coordinate system based on the target position and the position of the fixed camera.

[0015] The position of the fixed camera is obtained, the pixel coordinates of the target are obtained based on the image data, and the pixel coordinates of the target are converted into coordinates in a spatial coordinate system.

[0016] Preferably, determining whether the two coordinate positions match includes:

[0017] Calculate the Euclidean distance between the target coordinates of the fixed camera and the target coordinates of the vehicle-mounted LiDAR. If the Euclidean distance is less than a preset distance threshold, the two coordinate positions are considered to match.

[0018] Preferably, the step of reconstructing the target based on the point cloud data to obtain a reconstruction model includes:

[0019] The point cloud data is preprocessed, and a watertight triangulation network is generated based on the preprocessed point cloud data using the Poisson reconstruction algorithm. The holes in the watertight triangulation network are identified and filled in to obtain the reconstruction model.

[0020] Preferably, obtaining the depth map point cloud based on the image data includes:

[0021] A depth map is obtained from the image data using the MiDas model. A mask for marking the target region is obtained from the image data using the YOLOv8 model. An initial point cloud is obtained based on the depth map and the mask. An orbital plane is fitted from the image data. A global scale factor is calculated using the pixel distance between orbits in the image data and the actual orbital spacing. The initial point cloud is calibrated based on the global scale factor to obtain the depth map point cloud.

[0022] Preferably, the step of matching the reconstructed model and the depth map point cloud to obtain a complete target point cloud model includes:

[0023] The nearest neighbor of each point in the reconstruction model in the depth map point cloud is determined to form a point pair. The local deformation of each point is estimated based on the point pair to generate a deformation field. The deformation field is gradually optimized through multiple iterations until the error is less than a preset threshold, and a complete target point cloud model is output.

[0024] Preferably, extracting features from the target point cloud model and the image data includes:

[0025] The target's volume, reflectivity, and spatial density are obtained by extracting features from the target point cloud model. The target's texture and color are obtained by extracting features from the image data. If the volume is less than a preset volume threshold, the reflectivity is less than a preset reflectivity threshold, and the spatial density is less than a preset density threshold, then the target is determined not to be a foreign object. Otherwise, the features of the target point cloud model and the features of the image data are fused.

[0026] Preferably, the fusion of the features of the target point cloud model and the features of the image data includes:

[0027] The point cloud feature encoder fuses the target's volume, reflectivity, spatial density, and point cloud coordinates of the target's point cloud model to output point cloud fusion features. The image feature encoder fuses the target's texture and color to output image fusion features. The fusion module fuses the point cloud fusion features and the image fusion features to output a fusion feature vector. The fusion feature vector is input to a probability prediction head, which outputs foreign object detection probability, material classification probability, and risk probability.

[0028] Preferably, determining whether a foreign object has intruded into the track includes:

[0029] The foreign object detection probability, the material classification probability, and the risk probability are weighted and summed to obtain the foreign object intrusion probability. The weights are dynamically adjusted according to environmental conditions. If the foreign object intrusion probability is greater than a preset intrusion threshold, it is determined that a foreign object has intruded into the track, and an alarm signal is sent.

[0030] On the other hand, this application also provides a track foreign object intrusion detection device, the device comprising:

[0031] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the above-described method for detecting foreign object intrusion into orbit.

[0032] This application proposes a method and apparatus for detecting foreign object intrusion on railway tracks. It utilizes data from a trackside fixed camera and a vehicle-mounted LiDAR, fusing detection data from both platforms to improve the accuracy and reliability of foreign object intrusion detection and reduce the impact of environmental factors on the detection results. The method matches the target positions detected by the trackside fixed camera and the vehicle-mounted LiDAR, completing pre-screening of the target and reducing unnecessary subsequent data processing. It reconstructs a target model based on image data and point cloud data, resulting in a high-precision, high-resolution target model that captures the target's detailed features. Finally, it extracts and fuses image data and point cloud data to enhance feature representation and improve the system's detection capabilities. Attached Figure Description

[0033] Figure 1 A flowchart of a foreign object intrusion detection method for orbits provided as an exemplary embodiment of the present invention. Detailed Implementation

[0034] The detailed description that follows, taken in conjunction with the accompanying drawings, is intended to describe various exemplary embodiments of the invention and is not intended to represent only embodiments in which the invention can be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form to avoid obscuring the concept of the invention.

[0035] Example 1

[0036] This embodiment provides a method for detecting foreign object intrusion into orbit, such as... Figure 1 As shown, the method provided in this embodiment includes the following steps:

[0037] Step 1: Acquire image data from a fixed camera beside the track, and detect targets in the image based on the image data.

[0038] Fixed trackside cameras are a crucial component of track foreign object intrusion detection systems. They primarily use video surveillance technology to capture real-time images along the track and identify and locate foreign objects that may endanger train safety. Since they do not require direct contact with the track or train, installation and maintenance are relatively simple, and the cameras can monitor a track area ranging from tens to hundreds of meters. However, fixed cameras are significantly affected by changes in lighting conditions, resulting in a noticeable decrease in image quality, and their fixed viewing angle is limited.

[0039] Based on image data, this method detects suspected foreign objects in an image. A pre-trained YOLOv8 model is used to detect the input image data, obtain the target detection results, output the target bounding box, and generate a mask to mark the target region.

[0040] Step 2: Obtain point cloud data from the onboard lidar of the train, and detect targets in the point cloud data based on the point cloud data.

[0041] Among them, the train-mounted lidar emits laser beams and receives reflected signals to detect objects in front of the train in real time. It is used to identify foreign objects such as pedestrians, vehicles, falling rocks, and collapsed objects on the track to avoid collision accidents. Point cloud data is a three-dimensional spatial dataset generated after lidar scanning. It consists of a large number of discrete laser reflection points (each point contains XYZ coordinates, reflection intensity, and other information) and can characterize the surface shape and spatial position of objects.

[0042] Train-mounted lidar achieves centimeter-level ranging accuracy, far exceeding that of millimeter-wave radar and cameras. It is suitable for small target detection, does not rely on ambient light, and can operate stably in low-light environments such as at night and in tunnels. Providing three-dimensional contour information of targets through point cloud data is more beneficial for object classification and trajectory prediction than two-dimensional images. However, high-precision lidar is expensive, and large-scale fixed deployment along tracks presents significant cost pressures. Therefore, acquiring radar data through a train-mounted platform is a relatively efficient, economical, and accurate method.

[0043] Target detection in point cloud data includes: 1. Preprocessing stage: filtering out background point clouds and removing orbital structure point clouds; filtering out dynamic noise, eliminating transient noise such as birds, rain, and snow, and suppressing random noise from sensors; voxelization of point clouds to convert disordered point clouds into regular voxel networks. 2. Target segmentation stage: using PointNet++ network for end-to-end point cloud segmentation, or using the DBSCAN clustering algorithm to separate independent target objects, segmenting targets in the preprocessed point cloud data.

[0044] Step 3: Establish a spatial coordinate system. Convert the position of the target in the point cloud data and the pixel coordinates of the target in the image into coordinate positions in the spatial coordinate system, and determine whether the two coordinate positions match.

[0045] Because fixed cameras cover a wide area along the track, while vehicle-mounted LiDAR focuses on a high-precision local area in front of the train, combining data from both can create a multi-layered detection network for foreign object detection, encompassing both global and approach detection. Furthermore, since cameras provide top / side-view perspectives and LiDAR provides forward-looking 3D data, blind spots associated with single sensors can be eliminated. Simultaneously, using vehicle-mounted LiDAR reduces the need for fixed trackside LiDAR, lowering the cost of foreign object detection. LiDAR also overcomes the image quality degradation caused by rain, snow, and backlighting from fixed cameras, while fixed cameras can provide color and texture features to help distinguish hazardous targets, compensating for the limitations of LiDAR.

[0046] To integrate data from trackside fixed cameras and vehicle-mounted LiDAR, the train sends point cloud data to the server via onboard 5G or a private wireless network, while the trackside fixed cameras send data to the server via fiber optic network or a private wireless network. Step 3 and subsequent processing are all performed on the server. To facilitate rapid processing and feedback, the server can be located in a substation or signal room.

[0047] Because LiDAR data uses its own 3D point cloud coordinates as the original coordinate system, while the camera outputs 2D image pixel coordinates, it's impossible to directly compare their coordinate positions, making it impossible to determine whether they are observing the same target. Therefore, only by converting the point cloud and image data to the same coordinate system can mismatches be avoided, such as misidentifying a tree next to a track as an object on the track. The converted coordinate positions are then checked for a match. If they don't match, it indicates that the camera and LiDAR are actually observing different physical objects, or that the camera or LiDAR is misdetecting. In such cases, the process transitions to single-platform object intrusion detection, without further specific limitations.

[0048] In step 3, the spatial coordinate system is established by taking the position of the fixed camera as the origin of the spatial coordinate system, taking the straight line parallel to the direction of the track extension as the x-axis, and taking the straight line perpendicular to the direction of the track extension and located on the horizontal plane as the y-axis.

[0049] Since a fixed camera has a fixed spatial position, using it as the origin of the coordinate system provides a stable coordinate system for both the train and the camera's detected targets, while also reducing the computational workload of projecting the detected targets to world coordinates. Using a straight line parallel to the track's extension direction as the x-axis and a straight line perpendicular to the track's extension direction and located on the horizontal plane as the y-axis creates a two-dimensional coordinate system parallel to the ground. This transforms the detected targets into their coordinate positions on a two-dimensional coordinate system, reducing the computational workload of transformation and position matching. Using a straight line parallel to the track's extension direction as the x-axis can be done by choosing one of the two rails or by using a straight line extending from the centerline of the two rails.

[0050] The current train position is obtained, and the point cloud data includes the relative position of the target and the train. The target position is determined based on the relative position and the train position. The target position is converted into a coordinate position in the spatial coordinate system based on the target position and the position of the fixed camera.

[0051] The current train position is determined by satellite positioning data, such as the latitude and longitude output in real time from GNSS, BeiDou, and GPS. The point cloud data output by the lidar includes the target's relative position in the lidar coordinate system. Considering the lidar's installation location on the train, this relative position is compensated for and first transformed into the vehicle's coordinate system. Then, the target's coordinates in the vehicle's coordinate system are transformed into the camera's coordinate system.

[0052] The position of the fixed camera is obtained, the pixel coordinates of the target are obtained based on the image data, and the pixel coordinates of the target are converted into coordinates in a spatial coordinate system.

[0053] In step 1, the YOLOv8 model is used to output the target bounding box, and the pixel coordinates (u, v) of the center point of the target bounding box are calculated. The pixel coordinates (u, v) of the center point of the target bounding box represent the pixel coordinates of the target. These pixel coordinates are converted to spatial coordinates based on the camera's intrinsic parameters, as shown below:

[0054]

[0055] In the formula, (u,v) are pixel coordinates, (X,Y) are coordinate positions in the spatial coordinate system, h is the height of the camera, θ is the tilt angle of the camera, and (f x ,f y f is the focal length of the camera. x and f y These represent the magnification factors of the image in the horizontal and vertical directions, respectively. x ,c y ) represents the coordinates of the camera's principal point.

[0056] Calculate the Euclidean distance between the target coordinates of the fixed camera and the target coordinates of the vehicle-mounted LiDAR. If the Euclidean distance is less than a preset distance threshold, the two coordinate positions are considered to match.

[0057] In practical applications, sensor data always contains a certain amount of noise and error. A preset distance threshold can tolerate these errors. By adjusting the threshold, the matching effect can be optimized according to the specific application scenario and sensor accuracy.

[0058] Step 4: If the two coordinate positions match, the target is reconstructed based on the point cloud data to obtain a reconstructed model. A depth map point cloud is obtained based on the image data. The reconstructed model and the depth map point cloud are matched to obtain a complete target point cloud model. Features of the target point cloud model and features of the image data are extracted. The features of the target point cloud model and the features of the image data are fused to determine whether there is a foreign object intrusion on the track.

[0059] Because the relative positions of the train, camera, and foreign object are not fixed, the foreign object may appear between the train and the camera. In this case, the lidar and the camera will capture two opposite sides of the target. However, when the foreign object appears on the same side of the train and the camera, the angles of the train and the camera will be different, resulting in differences in the sides of the target captured. In other words, under any circumstances, there will be gaps in the data comparison between the two. Therefore, in step 4, it is necessary to reconstruct the data to fill in the missing parts of the point cloud data relative to the image data.

[0060] The point cloud data is preprocessed, and a watertight triangulation network is generated based on the preprocessed point cloud data using the Poisson reconstruction algorithm. The holes in the watertight triangulation network are identified and filled in to obtain the reconstruction model.

[0061] The preprocessing of point cloud data includes voxel downsampling and noise removal. The Poisson reconstruction algorithm, proposed in 2006, is a network-based 3D reconstruction algorithm based on octrees and the Poisson equation. It transforms discrete point clouds into continuous implicit fields and extracts isosurfaces to obtain a closed, watertight triangular network.

[0062] In a closed watertight mesh, each edge is shared by two adjacent triangular faces, while the boundary edge of a hole is used by only one face. Therefore, identifying holes in a watertight triangular mesh involves: extracting all triangular faces of the watertight triangular mesh and extracting each edge; counting the number of times each edge appears, outputting the edge that appears only once, which constitutes the boundary edge of the hole, and connecting these boundary edges into an ordered closed loop.

[0063] The process of filling in the holes involves: extracting the vertices of the boundary edges of all holes, pointing the normal vectors of the boundary edges toward the inside of the holes to guide the reconstructed surface to grow toward the inside of the holes; merging the point cloud of the original watertight triangulation network with the boundary vertices of the holes as input data; and using the Poisson reconstruction algorithm to fill in the holes based on the input data to obtain the reconstructed model.

[0064] A depth map is obtained from the image data using the MiDas model. A mask for marking the target region is obtained from the image data using the YOLOv8 model. An initial point cloud is obtained based on the depth map and the mask. An orbital plane is fitted from the image data. A global scale factor is calculated using the pixel distance between orbits in the image data and the actual orbital spacing. The initial point cloud is calibrated based on the global scale factor to obtain the depth map point cloud.

[0065] Since point cloud data and image data do not have modal mismatch, they cannot be directly fused. Therefore, it is necessary to convert the image data into a depth map point cloud. The MiDaS algorithm is a technique for estimating depth information from monocular images. It uses a deep learning model to predict the depth value of each pixel in a single image, thereby generating a depth map. Therefore, in this embodiment, the MiDaS model is used to obtain the depth map based on the image data.

[0066] The YOLOv8 model is used to obtain a mask for marking the target region based on the image data. Specifically, in step 1, the pre-trained YOLOv8 model has been used to detect the input image data, obtain the target detection result, and output the target bounding box to generate a mask for marking the target region.

[0067] The initial point cloud is obtained based on the depth map and the mask, specifically by: extracting the coordinates of all pixels within the mask; obtaining the depth values ​​corresponding to the coordinates of all pixels within the mask based on the depth map; and calculating the 3D coordinates of all pixels within the mask based on the pixel coordinates and depth values, as shown below, to obtain the initial point cloud.

[0068]

[0069] In the formula, (X,Y,Z) are the 3D coordinates of a pixel, (u,v) are the pixel coordinates, d is the depth value, and (f x ,f y (c) represents the camera's focal length. x ,c y ) represents the coordinates of the camera's principal point.

[0070] However, since the initial point cloud is generated based on image data and lacks a real physical scale, it cannot reflect the true scale of the real target. Therefore, in this embodiment, the initial point cloud needs to be calibrated.

[0071] The orbital plane is fitted based on the image data. Specifically, in this embodiment, the lower part of the image is considered the ground. Three pixels are randomly selected within the lower third of the image data, and the RANSAC algorithm is used to fit the plane to obtain the orbital plane, as shown below:

[0072] aX+bY+cZ+d=0

[0073] In the formula, (X,Y,Z) are 3D coordinates, a is the X component of the orbital plane normal vector, b is the Y component of the orbital plane normal vector, c is the Z component of the orbital plane normal vector, and d is the offset of the orbital plane from the origin.

[0074] The global scale factor is calculated using the pixel distance between tracks in the image data and the actual track spacing. Specifically,

[0075]

[0076] In the formula, L is the actual track spacing, (f x ,f y Z is the focal length of the camera, Δp is the pixel distance between tracks, and Z is the focal length of the camera. plane denoted as the plane depth of the orbit, a as the X component of the orbit plane normal vector, b as the Y component of the orbit plane normal vector, c as the Z component of the orbit plane normal vector, d as the offset of the orbit plane from the origin, w as the width of the image, h as the height of the image, and scale as the global scale factor.

[0077] The depth map point cloud is obtained by calibrating the initial point cloud according to the global scale factor. Specifically,

[0078]

[0079] In the formula, P calibrated The coordinates of the depth map point cloud are given, scale is the global scale factor, and P=(X,Y,Z) are the coordinates of the initial point cloud.

[0080] The nearest neighbor of each point in the reconstruction model in the depth map point cloud is determined to form a point pair. The local deformation of each point is estimated based on the point pair to generate a deformation field. The deformation field is gradually optimized through multiple iterations until the error is less than a preset threshold, and a complete target point cloud model is output.

[0081] Specifically, the nearest neighbor of each point in the reconstruction model in the depth map point cloud is determined to form a point pair. This is achieved by using the KDTree algorithm to find the nearest neighbor of each point in the reconstruction model in the depth map point cloud.

[0082] Based on the local deformation estimation of each point in the point pair, a deformation field is generated to deform the reconstructed model to the local structure of the depth map point cloud. The rotation matrix and translation vector are calculated using the least squares method or other optimization algorithms to move the points of the reconstructed model closer to the points in the depth map point cloud within the point pair, forming the deformation field, as shown below:

[0083]

[0084] In the formula, p i For the points in the reconstructed model, R is the rotation matrix, t is the translation vector, and p' i This is the point after the movement.

[0085] The deformation field is gradually optimized through multiple iterations until the error is less than a preset threshold. A complete target point cloud model is then output. The error is the Euclidean distance between the position of the reconstructed point and the point in the depth map point cloud. If the Euclidean distance is greater than the preset threshold, the rotation matrix and translation vector are calculated using the least squares method to move the moved point to a position closer to the point in the depth map point cloud. This process continues until the error is less than the preset threshold, at which point a complete target point cloud model is then output.

[0086] The target's volume, reflectivity, and spatial density are obtained by extracting features from the target point cloud model. The target's texture and color are obtained by extracting features from the image data. If the volume is less than a preset volume threshold, the reflectivity is less than a preset reflectivity threshold, and the spatial density is less than a preset density threshold, then the target is determined not to be a foreign object. Otherwise, the features of the target point cloud model and the features of the image data are fused.

[0087] The target point cloud model is extracted to obtain its volume, reflectivity, and spatial density. The volume is calculated as follows: the space is divided into a uniform 3D grid based on a set voxel size; each point is mapped to its corresponding voxel grid by traversing the point cloud, generating a binarized voxel mask; the number of occupied voxels is counted; and the physical volume occupied by the point cloud is calculated by multiplying the volume of a single voxel by the total number of effective voxels. The reflectivity of the target is calculated from the intensity values ​​contained in the point cloud data to obtain the average reflectivity. The spatial density is calculated as follows: the space is divided into a uniform 3D grid based on a set voxel size; the number of points contained in each voxel is counted; and the density per unit volume is obtained by normalizing the voxel volume by the inverse, ultimately generating a 3D field reflecting the spatial distribution density of the point cloud.

[0088] The texture and color of the target are obtained by extracting features from the image data. The color extraction process is as follows: first, the RGB image is converted to HSV space, the histogram statistical features of the hue channels are calculated, and then the color distribution characteristics are encoded through two layers of multilayer perceptron. The texture extraction process is as follows: high-order semantic features are extracted through a pre-trained ResNet50 network to capture the target's edge, pattern, and other structured information.

[0089] The criteria for determining if a target is not a foreign object are as follows: its volume is less than a preset volume threshold, its reflectivity is less than a preset reflectivity threshold, and its spatial density is less than a preset density threshold. Specifically, a target that is too small (e.g., debris) is insufficient to pose a significant threat; a target with excessively low reflectivity indicates abnormal reflective properties (e.g., a dark plastic bag has a much lower reflectivity than a metallic object); and a target with excessively low spatial density indicates a sparse point cloud distribution (e.g., tree branches). These types of foreign objects are unlikely to affect train safety. All three criteria must be met simultaneously to determine if a target is not a foreign object. This is because small targets with high reflectivity (e.g., metal screws) and targets with low reflectivity but high density (e.g., dense vegetation) can still pose a safety risk. Therefore, all three conditions must be met simultaneously to ensure the lowest possible false detection rate. If all three criteria cannot be met simultaneously, the target is considered potentially a foreign object, and further processing is initiated.

[0090] The point cloud feature encoder fuses the target's volume, reflectivity, spatial density, and point cloud coordinates of the target's point cloud model to output point cloud fusion features. The image feature encoder fuses the target's texture and color to output image fusion features. The fusion module fuses the point cloud fusion features and the image fusion features to output a fusion feature vector. The fusion feature vector is input to a probability prediction head, which outputs foreign object detection probability, material classification probability, and risk probability.

[0091] First, the target's volume, reflectivity, spatial density, and point cloud coordinates are concatenated into a multidimensional composite feature, which is then normalized to eliminate dimensional differences. The point cloud feature encoder uses a multilayer perceptron to encode the multidimensional composite feature of each point in high dimension, extracting deep point-level representations. A multi-head attention mechanism is used to dynamically model the internal structural relationships of the point cloud. Global average pooling aggregates the features of all points, outputting the point cloud fusion feature. Finally, texture and color features are concatenated to output the image fusion feature.

[0092] The fusion module first projects the point cloud fusion features and the image fusion features into a multi-dimensional space. It then uses a cross-attention mechanism to use the projected point cloud fusion features as a query vector and the image fusion features as key-value pairs. The association between the query vector and the key-value pairs is calculated through a multi-head attention mechanism. Finally, the output of the multi-head attention mechanism is concatenated with the point cloud fusion features and compressed and fused into a fusion feature vector through a multilayer perceptron.

[0093] The probability prediction head employs a parallel branching structure to implement foreign object detection, material classification probability, and risk assessment. The foreign object detection branch outputs a one-dimensional probability through two fully connected layers, which is then activated by a sigmoid function to determine the presence of a foreign object and output the foreign object detection probability. The material classification branch also outputs a one-dimensional probability through two fully connected layers, which is then normalized by Softmax to output the material classification probability, including five probability distributions: metal, plastic, stone, plant, and others. The risk assessment branch compresses the feature dimension and generates risk probabilities through Softmax, including low, medium, and high risk probabilities.

[0094] The foreign object detection probability, the material classification probability, and the risk probability are weighted and summed to obtain the foreign object intrusion probability. The weights are dynamically adjusted according to environmental conditions. The foreign object intrusion probability is:

[0095] P = w1P1 + w2P2 + w3P3

[0096] In the formula, w1 is the foreign object detection weight, w2 is the material weight, w3 is the risk weight, P1 is the foreign object detection probability, P2 is the material classification probability, P3 is the risk probability, and P is the foreign object intrusion probability.

[0097] The weights for foreign object detection, material composition, and risk are dynamically adjusted based on environmental conditions. Default weights are used during sunny daytime conditions, the probability of foreign objects is increased in rainy weather, and the weight for material composition is increased at night. This is because slippery conditions in rain reduce the reliability of material composition assessment, and insufficient light at night affects the confidence level of foreign object detection.

[0098] If the probability of foreign object intrusion is greater than a preset intrusion threshold, a foreign object intrusion is determined to have occurred on the track, and an alarm signal is sent. The preset intrusion threshold can be dynamically adjusted based on feedback; for example, the threshold can be lowered when the false alarm rate is high to reduce missed detections, and the threshold can be raised when the false alarm rate is high.

[0099] Example 2

[0100] This embodiment provides a method and apparatus for detecting foreign object intrusion into a railway track, the apparatus comprising:

[0101] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the above-described method for detecting foreign object intrusion into orbit.

[0102] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for detecting foreign object intrusion into orbit, characterized in that: Acquire image data from a fixed camera beside the track, and detect targets in the image based on the image data; Acquire point cloud data from the onboard lidar of the train, and detect targets in the point cloud data based on the point cloud data; A spatial coordinate system is established, and the position of the target in the point cloud data and the pixel coordinates of the target in the image are converted into coordinate positions in the spatial coordinate system. It is determined whether the two coordinate positions match. If they match, the target is reconstructed based on the point cloud data to obtain a reconstructed model. A depth map point cloud is obtained based on the image data. The reconstructed model and the depth map point cloud are matched to obtain a complete target point cloud model. Features of the target point cloud model and features of the image data are extracted. The features of the target point cloud model and the features of the image data are fused to determine whether there is a foreign object intrusion on the track. The process of obtaining a depth map point cloud based on the image data includes: The MiDas model is used to obtain a depth map from the image data. The YOLOv8 model is used to obtain a mask for marking the target region from the image data. An initial point cloud is obtained from the depth map and the mask. An orbital plane is fitted from the image data. A global scale factor is calculated using the pixel distance between orbits in the image data and the actual orbital spacing. The initial point cloud is calibrated based on the global scale factor to obtain the depth map point cloud. The process of matching the reconstructed model with the depth map point cloud to obtain a complete target point cloud model includes: The nearest neighbor of each point in the reconstruction model in the depth map point cloud is determined to form a point pair. The local deformation of each point is estimated based on the point pair to generate a deformation field. The deformation field is gradually optimized through multiple iterations until the error is less than a preset threshold, and a complete target point cloud model is output.

2. The method according to claim 1, characterized in that, The step of establishing a spatial coordinate system, which converts the position of the target in the point cloud data and the pixel coordinates of the target in the image into coordinate positions in the spatial coordinate system, includes: A spatial coordinate system is established with the position of a fixed camera as the origin of the spatial coordinate system, a straight line parallel to the direction of the track extension as the x-axis, and a straight line perpendicular to the direction of the track extension and located on the horizontal plane as the y-axis. The current train position is obtained, the point cloud data includes the relative position of the target and the train, the target position is determined based on the relative position and the train position, and the target position is converted into the coordinate position of the spatial coordinate system based on the target position and the position of the fixed camera. The position of the fixed camera is obtained, the pixel coordinates of the target are obtained based on the image data, and the pixel coordinates of the target are converted into coordinates in a spatial coordinate system.

3. The method according to claim 1, characterized in that, The determination of whether the two coordinate positions match includes: Calculate the Euclidean distance between the target coordinates of the fixed camera and the target coordinates of the vehicle-mounted LiDAR. If the Euclidean distance is less than a preset distance threshold, the two coordinate positions are considered to match.

4. The method according to claim 1, characterized in that, The process of reconstructing the target based on the point cloud data to obtain a reconstruction model includes: The point cloud data is preprocessed, and a watertight triangular network is generated based on the preprocessed point cloud data using the Poisson reconstruction algorithm. The holes in the watertight triangular network are identified and filled in to obtain the reconstruction model.

5. The method according to claim 1, characterized in that, Extracting features from the target point cloud model and the image data includes: The target's volume, reflectivity, and spatial density are obtained by extracting features from the target point cloud model. The target's texture and color are obtained by extracting features from the image data. If the volume is less than a preset volume threshold, the reflectivity is less than a preset reflectivity threshold, and the spatial density is less than a preset density threshold, then the target is determined not to be a foreign object. Otherwise, the features of the target point cloud model and the features of the image data are fused.

6. The method according to claim 5, characterized in that, The fusion of features from the target point cloud model and features from the image data includes: The point cloud feature encoder fuses the target's volume, reflectivity, spatial density, and point cloud coordinates of the target's point cloud model to output point cloud fusion features. The image feature encoder fuses the target's texture and color to output image fusion features. The fusion module fuses the point cloud fusion features and the image fusion features to output a fusion feature vector. The fusion feature vector is input to a probability prediction head, which outputs foreign object detection probability, material classification probability, and risk probability.

7. The method according to claim 6, characterized in that, The determination of whether a foreign object has intruded into the track includes: The foreign object detection probability, the material classification probability, and the risk probability are weighted and summed to obtain the foreign object intrusion probability. The weights are dynamically adjusted according to environmental conditions. If the foreign object intrusion probability is greater than a preset intrusion threshold, it is determined that a foreign object has intruded into the track, and an alarm signal is sent.

8. A track foreign object intrusion detection device, characterized in that, The device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a foreign object intrusion detection method as described in any one of claims 1-7.

Images