A method for detecting foreign objects on a railway track based on a millimeter wave radar sensor and a camera sensor

Abstract

This invention belongs to the field of foreign object detection technology on railway tracks, specifically disclosing a method for detecting foreign objects on railway tracks based on millimeter-wave radar sensors and camera sensors. The method includes the following steps: a millimeter-wave radar sensor detects and obtains point clouds, which are then preprocessed to obtain a large frame dataset; a camera sensor captures image data, which is then processed using a lightweight YOLOv5s algorithm to obtain the Region of Interest (ROI). C Synchronize the large frame dataset and image data in time, and then project the large frame dataset onto the pixel coordinate system to generate the Region of Interest (ROI). R ROI R and ROI C Form a fusion detection box; calculate each ROI using a decision-level fusion method based on intersection-union ratio. R and its corresponding ROI C The overlap is used to obtain a preliminary fusion result; the EKF algorithm processes the uncertain targets in the preliminary fusion result and outputs the fusion result; the obtained definite targets represent the detected obstacles. This invention uses millimeter-wave radar and camera information fusion technology, which significantly improves detection accuracy and real-time performance, thereby enhancing the safety of railway track traffic.

Description

Technical Field

[0001] This invention belongs to the field of foreign object detection technology on railway tracks, and particularly relates to a method for detecting foreign objects on railway tracks based on millimeter-wave radar sensors and camera sensors. Background Technology

[0002] With the rapid development of China's railway infrastructure in recent years, railway traffic safety has become increasingly important. Despite the fully enclosed management of railways, incidents of foreign objects such as cattle, sheep, and falling rocks still occur from time to time. These intrusions pose a serious threat to locomotive operation safety, disrupting transportation order and causing casualties and economic losses.

[0003] With the development of deep learning, current research on foreign object detection on railway tracks focuses more on image recognition algorithms. One-stage target detection algorithms, such as YOLO, offer fast detection speeds and good real-time performance. However, visual detection algorithms are easily affected by external environments. They are significantly impacted by large variations in ambient light intensity (e.g., in darkness) or adverse weather conditions like rain and fog. Considering this drawback, multi-sensor fusion technology has emerged, which can compensate for data loss when a single sensor cannot collect sufficient external data in certain situations.

[0004] Millimeter-wave radar is small in size and light in weight, capable of long-range detection, and is insensitive to media such as rain, fog, dust, and light, making it widely used in target detection tasks. Combining millimeter-wave radar with cameras can perfectly leverage the advantages of each other to achieve target detection in complex scenarios. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a method for detecting foreign objects on railway tracks based on millimeter-wave radar sensors and camera sensors, thereby solving the current challenges in identifying foreign objects on railway tracks.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for detecting foreign objects on railway tracks based on millimeter-wave radar sensors and camera sensors includes the following steps:

[0008] S1. A millimeter-wave radar sensor detects obstacles on the railway track ahead, obtaining point clouds, which are then preprocessed to obtain a large frame dataset. The specific preprocessing process is as follows:

[0009] To address the sparse point cloud acquisition by millimeter-wave radar sensors, the acquired point cloud (each point in the point cloud includes the (x,y,z) coordinates, reflective area, and velocity of the detected obstacle) is first subjected to multi-frame autonomous fusion processing. The data after multi-frame autonomous fusion processing is then processed based on the DBSCAN clustering algorithm of three-dimensional bounding boxes to obtain a large frame dataset.

[0010] Preferably, in step S1, the specific operation method of the multi-frame autonomous fusion processing is as follows:

[0011] After reading the obtained point cloud, the point cloud of the current frame is first superimposed with the point cloud of the previous three frames to increase the density. That is, the obtained point cloud is superimposed into a large frame every 4 frames. By continuously superimposing 4 frames of point cloud, a large frame set is created. Through multi-frame autonomous fusion processing, while maintaining the original basic shape and spatial structure characteristics of the point cloud, a more refined and denser point cloud is obtained.

[0012] Preferably, in step S1, the DBSCAN clustering algorithm processing includes the following steps:

[0013] The vertical space with the railway track as the base is used as the area to be measured. The millimeter-wave radar scans the area to be measured to obtain point clouds. After multi-frame autonomous fusion processing, a large frame set is formed. After the large frame set is processed by the DBSCAN clustering algorithm, the points in the point cloud obtained by the millimeter-wave radar sensor that belong to the real reflection points of the obstacle are selected to form the large frame dataset.

[0014] S2. The camera sensor captures images of obstacles on the railway track ahead, obtaining image data. The lightweight YOLOv5s algorithm is then used to process the data to obtain the region of interest (ROI). C The specific steps are as follows:

[0015] S2.1 Improve the focus layer and CSP (Cross Stage Partial connections) module in the YOLOv5s algorithm to obtain a lightweight YOLOv5s algorithm;

[0016] Preferably, in step S2.1, the process of obtaining the lightweight YOLOv5s algorithm is as follows:

[0017] S2.1.1, the YOLOv5s algorithm includes a focus layer that transforms a feature map of size H×W×C into... Size, where H and W represent the height and width of the feature map, respectively, and C represents the number of channels in the feature map. To improve the detection speed and accuracy of the YOLOv5s algorithm, the focus layer is improved to transform a feature map of size H×W×C into... The size is adjusted to effectively increase the channel dimension without losing feature map information;

[0018] S2.1.2 Replace the CSP module in the YOLOv5s algorithm with the MobileNetV3 module. The specific process is as follows: In the YOLOv5s algorithm, the MobileNetV3 module is used to replace the original CSP module in the backbone. (1) Perform depthwise convolution on each feature map channel of the input CSP module, and then perform pointwise convolution on all feature map channels of the input CSP module, so as to replace the standard convolution of the input feature map with the depthwise convolution included in MobileNetV3; (2) Replace the residual block connection mechanism in the CSP module with the inverse residual structure included in MobileNetV3. The inverse residual structure expands the number of feature map channels and then performs dilation convolution operation, and then compresses back to the original number of channels; (3) Replace the output layer of the CSP module with the SE module included in MobileNetV3.

[0019] S2.1.3 and S2.2.2.2. An improved YOLOv5s algorithm is obtained, optimized using the Detection On Tracks dataset to achieve a lightweight YOLOv5s algorithm. The Detection On Tracks dataset is divided into training, validation, and test sets in a ratio of 8:1:1. Then, the training, validation, and test sets are used sequentially for training, validation, and testing. The training iterations are performed 300 times until the evaluation metric meets the accuracy requirements. The evaluation criteria are as follows:

[0020]

[0021] Among them, the mean and average precision M AP It is one of the metrics for object detection, and it is the mean of the average accuracy across all categories. A P The average precision for each class is given, where m is the number of classes, and p and r are the precision and recall of the algorithm, respectively. If the above precision evaluation criteria are not met, the Detection On Tracks dataset splitting, training, validation, and testing are repeated, and the parameters are adjusted and updated until the precision meets the requirements.

[0022] S2.2. The lightweight YOLOv5s algorithm obtained in S2.1 is loaded into the visual AI board (Jetson Nano motherboard, B01 version). The image data obtained by the camera sensor from the obstacle on the railway track ahead is processed by the lightweight YOLOv5s algorithm to obtain the visual region of interest (ROI).C ROI C The size is S C S C The pixel area of ​​the obstacle as captured by the camera in pixel coordinates.

[0023] S3. Synchronize the large frame dataset obtained in S1 with the image data in S2 in time. Then, project the large frame dataset obtained in S1 onto the pixel coordinate system to generate the radar region of interest (ROI) in the pixel coordinate system. R Then, the radar region of interest (ROI) R Region of Interest (ROI) C A fusion detection box is formed in the pixel coordinate system;

[0024] Preferably, the specific process of step S3 is as follows:

[0025] S3.1. Synchronize the large frame dataset obtained in S1 with the image data in S2 in time. That is, perform time registration between the large frame dataset obtained in S1 and the image data in S2, using the least common multiple of periods principle for time synchronization. The specific process is as follows:

[0026] In S1, point clouds are stacked in groups of 4 frames to form a large frame, meaning the time interval for forming a large frame is T. Due to time synchronization, the camera also needs to capture images at intervals of T. Therefore, the least common multiple T is chosen as the sampling period for time synchronization. The sampling time interval for the millimeter-wave radar is obtained according to the following formula (3):

[0027]

[0028] Where T is the sensor's operating period and f is the sampling frequency.

[0029] S3.2 Project the large frame dataset obtained in S1 onto the pixel coordinate system. Generate a radar region of interest (ROI) in the pixel coordinate system from the point cloud in each large frame dataset. R ;

[0030] Preferably, the specific process of S3.2 is as follows:

[0031] S3.2.1 Transformation from millimeter-wave radar coordinate system to camera coordinate system:

[0032]

[0033] In equation (4), X C Y C Z C This represents the three-dimensional coordinates of the actual reflection point of the obstacle in the camera coordinate system; X R Y R Z RR is the three-dimensional coordinate of the actual reflection point of the obstacle in the millimeter-wave radar coordinate system, where R is the rotation matrix and t is the displacement matrix. It is the camera extrinsic matrix.

[0034] S3.2.2, Transform from camera coordinate system to pixel coordinate system:

[0035]

[0036] Where u and v are the coordinates of the actual reflection point of the obstacle target in the pixel coordinate system. It is the camera intrinsic parameter matrix, where f x It is the focal length in the horizontal direction, measured in pixels; f y It is the focal length in the vertical direction, measured in pixels; u o It is the pixel coordinate of the image center in the horizontal direction, v o It is the pixel coordinate of the image center in the vertical direction.

[0037] S3.2.3. Transform the coordinates of each point in the point cloud of the large frame dataset obtained in S1 into S3.1.2.1 and S3.1.2.2 in sequence, so that the point cloud of the large frame dataset obtained in S1 in the millimeter-wave radar coordinate system is projected into the pixel coordinate system.

[0038] S3.2.4. In the pixel coordinate system, the pixel coordinates of the points inside the point cloud of the large frame dataset obtained in S1 are used to approximate the size of the 3D bounding box, and the radar region of interest (ROI) is generated. R ROI R Equation (6) represents the ROI corresponding to each large frame in the large frame dataset obtained by S1. R The size is S R S R The pixel area of ​​the point cloud, i.e., the actual reflection point of the obstacle, projected onto the pixel coordinate system in each large frame of data.

[0039] ROI R = (u,v,w,h,t)∈R (6)

[0040] Where u and v represent the coordinates of the actual reflection point of the obstacle detected by the millimeter-wave radar in the pixel coordinate system, w, h, and t are the width, height, and depth of the three-dimensional bounding box in the pixel coordinate system, respectively, and R represents the entire pixel coordinate system.

[0041] S3.3: In the pixel coordinate system, the radar region of interest (ROI) corresponding to each large frame of data was obtained. R And the regions of interest (ROIs) corresponding to the image data acquired by the camera every 200ms in sync with its time. C Radar Region of Interest (ROI) RRegion of Interest (ROI) C The total pixel area of ​​the union is the fusion detection box.

[0042] S4. Using the Intersection over Union (IOU) decision-level fusion method, calculate the radar Region of Interest (ROI) corresponding to each large frame of data. R and the region of interest (ROI) corresponding to the image data acquired by the camera in time. C To determine the overlap and obtain preliminary fusion results, the following steps are included:

[0043] S4.1: Calculate the region of interest (ROI) for each radar to synchronize with time. R and each region of visual interest (ROI) C overlap S IoU The calculation method is as follows:

[0044]

[0045] Among them, S R Region of Interest (ROI) for radar R The size of the pixel area in pixel coordinates, S C Region of Interest (ROI) C The size of a pixel area in pixel coordinates;

[0046] S4.2: Set the detection threshold for evaluation. The evaluation method is as follows:

[0047] S IoU When the value is less than 0.3, the obstacle detected by the millimeter-wave radar and the camera are considered to be different obstacles and are discarded, i.e., false targets;

[0048] 0.3≤S IoU When the value is less than 0.5, the obstacle detected by the millimeter-wave radar and the camera may be the same obstacle, i.e., the target is uncertain.

[0049] 0.5≤S IoU When the value is ≤1, the obstacle detected by the millimeter-wave radar and the camera is considered to be the same obstacle, thus the target is identified.

[0050] Therefore, by judging the degree of overlap S IoU The point cloud and image data reflecting the detected obstacles are initially fused to obtain preliminary fusion results: false targets, definite targets, and uncertain targets.

[0051] S5. The Extended Kalman Filter (EKF) algorithm is used to dynamically track the overlapping boxes corresponding to the uncertain targets in the preliminary fusion result obtained in S4, and the fusion result is output, including the following steps:

[0052] In S5.1 and S4, the overlapping box state vector corresponding to the uncertain target is defined as X = (u, v, v) u ,v v ) T , where u, v, v u v v These represent the x-coordinate, y-coordinate, x-axis velocity, and y-axis velocity of the overlapping bounding box corresponding to the uncertain target in the pixel coordinate system, respectively. u v v This refers to the velocity of the actual reflected point cloud of the obstacle within the overlapping frame.

[0053] Using the extended Kalman filter algorithm, the state equation and prediction equation for the overlapping boxes are as follows:

[0054]

[0055] Where X(k) and X(k-1) represent the state vectors of the overlapping boxes corresponding to the uncertain target at times k and k-1, respectively; Z(k) represents the observation vector of the overlapping boxes corresponding to the uncertain target at time k; f(k) represents the state transition matrix; h(k) represents the observation function; and V(k) and W(k) represent Gaussian white noise.

[0056] S5.2. Based on the state of the uncertain target at time k-1, the predicted state equation of the uncertain target at time k is:

[0057]

[0058] in, P(k|k-1) represents the state prediction vector and prediction error vector of the overlapping box corresponding to the uncertain target at time k, F is the Jacobian matrix of f(k) at X(k-1|k-1), P(k-1|k-1) is the prediction error vector of the overlapping box corresponding to the uncertain target at time k-1, and Q is the covariance matrix of the process noise.

[0059] S5.3. The state update equation for the overlapping box corresponding to the uncertain target is derived from the state equation in Equation 9 as follows:

[0060]

[0061] In equation (10), Here, S(k) represents the state update vector, S(k) represents the innovation covariance, K(k) represents the gain matrix, P(k|k-1) is the prediction error covariance, and H is the state update vector. The Jacobian matrix at point R is the covariance matrix of the observation noise.

[0062] S5.4. The final state equation of the uncertain target is derived from the state update equation of equation (10):

[0063]

[0064] In equation (11), X(k|k) represents the state estimation vector of the uncertain target at time k, P(k|k) represents the covariance at time k, and K(k) represents the gain matrix. P(k|k-1) is the state update vector, P(k|k-1) is the prediction error covariance, and I represents the identity matrix.

[0065] Equation (8) represents the state equation and prediction equation of the overlapping box corresponding to the uncertain target during the entire EKF tracking process. Equation (9) is the state equation for predicting time k based on time k-1. Based on equation (9), the state update equation (10) of the uncertain target is derived. Based on equation (10), the final state equation (11) of the uncertain target is derived.

[0066] S5.5 After the above steps of iterative update, the number of iterations is 10. If the overlapping box corresponding to the tracked uncertain target appears 3 times in a row, then the overlapping box is output as the fusion result, indicating that the overlapping box is the same obstacle that was identified, that is, the uncertain target is actually a definite target.

[0067] S6, the determined target obtained from S4.2, and the determined target obtained from the uncertain target obtained from S4.2 after processing by S5 together represent the detected obstacles on the railway track.

[0068] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0069] This invention provides a method for detecting foreign objects on railway tracks based on millimeter-wave radar and camera sensors. A millimeter-wave radar and a visual AI board are installed on a 3-meter-high utility pole (1.2m above the track) next to the railway track. The camera and the visual AI board are assembled together at a height of 2m, and data is transmitted between the millimeter-wave radar and the visual AI board via a CAN bus. The camera then captures images of obstacles on the track ahead, forming a region of interest (ROI). C Point cloud data obtained by detecting obstacles on the railway track ahead using millimeter-wave radar is projected onto a vision system to form the region of interest (ROI) for the millimeter-wave radar. R Subsequently, the two sensors form a fused detection bounding box for the detected railway obstacle targets, and then the region of interest (ROI) is calculated according to the decision-level fusion method based on the intersection-union ratio (IOU). R ROI CThe overlap is determined to obtain a preliminary fusion result. After obtaining the preliminary fusion result, uncertain targets are dynamically tracked using an extended Kalman filter (EKF) algorithm, and the final fusion result is output. This invention employs information fusion technology combining millimeter-wave radar and cameras, overcoming the shortcomings of traditional track target detection methods, achieving significant improvements in detection accuracy and real-time performance, and enhancing the safety of railway track traffic. Attached Figure Description

[0070] Figure 1 This is a flowchart of a method for detecting foreign objects on railway tracks based on a millimeter-wave radar sensor and a camera sensor, according to the present invention.

[0071] Figure 2 A schematic diagram of the extended Kalman filter algorithm;

[0072] Figure 3 The railway track selected for the experiment of the railway track foreign object detection method of the present invention is the railway track experimental base in the North Campus of East China Jiaotong University. Data was collected and tested using a real railway track scene that was restored one-to-one.

[0073] Figure 4 The changes in point cloud density before and after multi-frame overlay and DBSCAN clustering processing;

[0074] Figure 5 The left image shows the visual region of interest (ROI) obtained using the YOLOv5 algorithm. C The right figure shows the region of interest (ROI) obtained by the lightweight YOLOv5 algorithm in this invention, along with its confidence level. C and its confidence level;

[0075] Figure 6 Region of Interest (ROI) for radar R Region of Interest (ROI) C The fusion detection frame is composed of these components. Detailed Implementation

[0076] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0077] like Figure 1 As shown, a method for detecting foreign objects on railway tracks based on millimeter-wave radar sensors and camera sensors includes the following steps:

[0078] S1. A millimeter-wave radar sensor detects obstacles on the railway track ahead, obtaining point clouds. After preprocessing, a point cloud dataset is obtained. The specific process is as follows:

[0079] To address the sparse point cloud acquired by millimeter-wave radar sensors, the acquired point cloud (each point in the point cloud includes the (x,y,z) coordinates, reflective area, and velocity of the detected obstacle) is first subjected to multi-frame autonomous fusion processing. The data after multi-frame autonomous fusion processing is then processed based on the DBSCAN clustering algorithm of three-dimensional bounding boxes.

[0080] The specific operation method of the multi-frame autonomous fusion processing is as follows:

[0081] After reading the obtained point cloud, the point cloud of the current frame is first superimposed with the point cloud of the previous three frames to increase the density. That is, the obtained point cloud is superimposed into a large frame every 4 frames. By continuously superimposing 4 frames of point cloud, a large frame set is created. Through multi-frame autonomous fusion processing, while maintaining the original basic shape and spatial structure characteristics of the point cloud, a more refined and denser point cloud is obtained.

[0082] The DBSCAN clustering algorithm process includes the following steps:

[0083] The area to be measured is a vertical space with a straight railway track, 50 meters long and 1.43 meters wide, as its base. Figure 3 The millimeter-wave radar scans the area under test to obtain point clouds, which are then processed through multi-frame autonomous fusion to form a large frame set. For each large frame in the large frame set, a preset neighborhood radius of ε is defined, and the minimum number of points within the ε neighborhood is set to P. min For a given large frame, a point P is randomly selected from the point cloud within that frame. If point P is identified as a core point, the density reachability principle is used to find all points in the point cloud within that frame that are directly and indirectly density reachable from point P, and these points are grouped into the same cluster. If point P is a boundary point, it is grouped into the cluster containing the core points in its neighborhood. If point P is neither a core point nor a boundary point, it is classified as noise and not grouped into any cluster. Thus, after filtering out outliers in the point cloud using the DBSCAN clustering algorithm, the average value of all remaining points in the point cloud is used to estimate the center position. By searching the outermost points, the spatial range of all remaining points in the point cloud is determined.

[0084] After each frame in the large frame set is processed by the DBSCAN clustering algorithm described above, the points in the point cloud obtained by the millimeter-wave radar sensor that belong to the actual reflection points of the obstacle are selected to form the large frame dataset.

[0085] S2. The camera sensor captures images of obstacles on the railway track ahead, obtaining image data. The lightweight YOLOv5s algorithm is then used to process the data to obtain the region of interest (ROI). C The specific steps are as follows:

[0086] S2.1. Considering engineering practicality, the focus layer and CSP (Cross Stage Partial connections) module in the YOLOv5s algorithm are improved to lightweight the algorithm, resulting in a lightweight YOLOv5s algorithm that improves detection speed and accuracy. The process of obtaining the lightweight YOLOv5s algorithm is as follows:

[0087] S2.1.1, the YOLOv5s algorithm includes a focus layer that transforms a feature map of size H×W×C into... Size, where H and W represent the height and width of the feature map, respectively, and C represents the number of channels in the feature map. To improve the detection speed and accuracy of the YOLOv5s algorithm, the focus layer is improved to transform a feature map of size H×W×C into... The size is adjusted to effectively increase the channel dimension without losing feature map information;

[0088] S2.1.2 Replace the CSP module in the YOLOv5s algorithm with the MobileNetV3 module. The specific process is as follows: In the YOLOv5s algorithm, the MobileNetV3 module replaces the original CSP module in the backbone. (1) Perform depthwise convolution on each feature map channel of the input CSP module, and then perform pointwise convolution on all feature map channels of the input CSP module. (1) Using the depthwise separable convolution included in MobileNetV3 to replace the CSP module for standard convolution of the input feature map; (2) Using the inverse residual structure included in the MobileNetV3 module to replace the residual block connection mechanism in the CSP module. The inverse residual structure expands the number of feature map channels before performing dilation convolution operation, and then compresses back to the original number of channels. This changes the characteristic of the residual block connection mechanism in the CSP module that performs multiple convolutions on the feature map channels and connects them together in a skip manner; (3) Using the SE module included in the MobileNetV3 module to replace the output layer of the CSP module. The SE module is an attention mechanism module that can adaptively adjust the channel output weights of the feature map by adjusting the weights of each channel of the feature map. Through the above processing, the number of model parameters and the computational complexity of the feature map channels during the input and output process are significantly reduced.

[0089] The improved YOLOv5s algorithm, derived from S2.1.3 and S2.2.1.2, is optimized using the Detection On Tracks dataset (a public dataset containing 3766 images of human behavior on railway tracks, with a resolution of 1080×1080, annotated in YOLO format (txt) and detailed class labels) to obtain a lightweight YOLOv5s algorithm. The training conditions are: Windows system, Intel(R) Core(TM) i5-12400f CPU, and NVIDIA GTX 4060 (8GB) GPU. The Detection On Tracks dataset is divided into training, validation, and test sets in a ratio of 8:1:1 (i.e., the training set includes 3012 images, and the test and validation sets each include 374 images). Training, validation, and testing are then performed sequentially using the training, validation, and test sets. The training iterations are repeated 300 times until the evaluation metric meets the accuracy requirements. The evaluation criteria are as follows:

[0090]

[0091] Among them, the mean and average precision M AP It is one of the metrics for object detection, and it is the mean of the average accuracy across all categories. A P The average precision for each class is given, where m is the number of classes, and p and r are the precision and recall of the algorithm, respectively. If the above precision evaluation criteria are not met, the Detection On Tracks dataset splitting, training, validation, and testing are repeated, and the parameters are adjusted and updated until the precision meets the requirements.

[0092] S2.2. The lightweight YOLOv5s algorithm obtained in S2.1 is loaded into the visual AI board (Jetson Nano motherboard, B01 version). The image data obtained by the camera sensor from the obstacle on the railway track ahead is processed by the lightweight YOLOv5s algorithm to obtain the visual region of interest (ROI). C ROI C The size is S C S C The pixel area of ​​the obstacle as captured by the camera in pixel coordinates.

[0093] S3. Synchronize the large frame dataset obtained in S1 with the image data in S2 in time. Then, project the large frame dataset obtained in S1 onto the pixel coordinate system to generate the radar region of interest (ROI) in the pixel coordinate system. R Then, the radar region of interest (ROI) R Region of Interest (ROI) C exist Pixel coordinate systemThe following steps are included in forming the fusion detection frame:

[0094] S3.1. Synchronize the large frame dataset obtained in S1 with the image data in S2 in time. That is, perform time registration between the large frame dataset obtained in S1 and the image data in S2, using the least common multiple of periods principle for time synchronization. The specific process is as follows:

[0095] Because different sensors have different sampling frequencies, there is a time difference in the collected data. The sampling frequency of the millimeter-wave radar is 20 Hz, that is, 20 samples per second, and the sampling frequency of the camera is 30 Hz, that is, 30 samples per second. According to equation (3), the sampling time interval of the millimeter-wave radar is 50 ms, and the sampling time interval of the camera is 33.3 ms. In S1, the point cloud is superimposed into a large frame every 4 frames, that is, the time interval for forming a large frame is 200 ms. Due to time synchronization, the camera also needs to take pictures at 200 ms intervals to obtain an image. Therefore, the least common multiple of 200 ms is selected as the sampling period for time synchronization (specifically, to make the fast camera compatible with the slow radar). The essence of time synchronization is to achieve consistency of the data obtained by the two sensors in time, and to prevent large deviations in the information after fusion.

[0096]

[0097] Where T is the sensor's operating period and f is the sampling frequency.

[0098] S3.2 Project the large frame dataset obtained in S1 onto the pixel coordinate system. Generate a radar region of interest (ROI) in the pixel coordinate system from the point cloud in each large frame dataset. R The specific process is as follows:

[0099] S3.2.1 Transformation from millimeter-wave radar coordinate system to camera coordinate system:

[0100]

[0101] In equation (4), X C Y C Z C This represents the three-dimensional coordinates of the actual reflection point of the obstacle in the camera coordinate system; X R Y R Z R R is the three-dimensional coordinate of the actual reflection point of the obstacle in the millimeter-wave radar coordinate system, where R is the rotation matrix and t is the displacement matrix. It is the camera extrinsic matrix.

[0102] S3.2.2, Transform from camera coordinate system to pixel coordinate system:

[0103]

[0104] Where u and v are the coordinates of the actual reflection point of the obstacle target in the pixel coordinate system. It is the camera intrinsic parameter matrix, where f x It is the focal length in the horizontal direction, measured in pixels; f y It is the focal length in the vertical direction, measured in pixels; u o It is the pixel coordinate of the image center in the horizontal direction, v o It is the pixel coordinate of the image center in the vertical direction.

[0105] S3.2.3, Transform the coordinates of each point in the point cloud of the large frame dataset obtained in S1 sequentially using S3.1.2.1) and S3.1.2.2) to project the point cloud of the large frame dataset obtained in S1 from the millimeter-wave radar coordinate system to the pixel coordinate system.

[0106] S3.2.4. In the pixel coordinate system, the pixel coordinates of the points inside the point cloud of the large frame dataset obtained in S1 are used to approximate the size of the 3D bounding box, and the radar region of interest (ROI) is generated. R ROI R Equation (6) represents the ROI corresponding to each large frame in the large frame dataset obtained by S1. R The size is S R S R The pixel area of ​​the point cloud, i.e., the actual reflection point of the obstacle, projected onto the pixel coordinate system in each large frame of data.

[0107] ROI R = (u,v,w,h,t)∈R (6)

[0108] Where u and v represent the coordinates of the actual reflection point of the obstacle detected by the millimeter-wave radar in the pixel coordinate system, w, h, and t are the width, height, and depth of the three-dimensional bounding box in the pixel coordinate system, respectively, and R represents the entire pixel coordinate system.

[0109] S3.3: In the pixel coordinate system, the radar region of interest (ROI) corresponding to each large frame of data was obtained. R And the regions of interest (ROIs) corresponding to the image data acquired by the camera every 200ms in sync with its time. C Radar Region of Interest (ROI) R Region of Interest (ROI) C The total pixel area of ​​the union is the fusion detection box.

[0110] S4. Using the Intersection over Union (IOU) decision-level fusion method, calculate the radar Region of Interest (ROI) corresponding to each large frame of data. RAnd the regions of interest (ROIs) corresponding to the image data acquired by the camera every 200ms in sync with its time. C To determine the overlap and obtain preliminary fusion results, the following steps are included:

[0111] S4.1: Calculate the region of interest (ROI) for each radar to synchronize with time. R and each region of visual interest (ROI) C overlap S IoU The calculation method is as follows:

[0112]

[0113] Among them, S R Region of Interest (ROI) for radar R The size of the pixel area in pixel coordinates, S C Region of Interest (ROI) C The size of a pixel area in pixel coordinates;

[0114] S4.2: Set the detection threshold for evaluation. The evaluation method is as follows:

[0115] S IoU When the value is less than 0.3, the obstacle detected by the millimeter-wave radar and the camera are considered to be different obstacles and are discarded, i.e., false targets;

[0116] 0.3≤S IoU When the value is less than 0.5, the obstacle detected by the millimeter-wave radar and the camera may be the same obstacle, i.e., the target is uncertain.

[0117] 0.5≤S IoU When the value is ≤1, the obstacle detected by the millimeter-wave radar and the camera is considered to be the same obstacle, thus the target is identified.

[0118] Therefore, by judging the degree of overlap S IoU The point cloud and image data reflecting the detected obstacles are initially fused to obtain preliminary fusion results: false targets, definite targets, and uncertain targets.

[0119] S5, such as Figure 2 As shown, the Extended Kalman Filter (EKF) algorithm is used to dynamically track the overlapping boxes corresponding to uncertain targets in the preliminary fusion result and output the fusion result, including the following steps:

[0120] In S5.1 and S4, the overlapping box state vector corresponding to the uncertain target is defined as X = (u, v, v) u ,v v ) T , where u, v, v u v vThese represent the x-coordinate, y-coordinate, x-axis velocity, and y-axis velocity of the overlapping bounding box corresponding to the uncertain target in the pixel coordinate system, respectively. u v v This refers to the velocity of the actual reflected point cloud of the obstacle within the overlapping frame.

[0121] Using the extended Kalman filter algorithm, the state equation and prediction equation for the overlapping boxes are as follows:

[0122]

[0123] Where X(k) and X(k-1) represent the state vectors of the overlapping boxes corresponding to the uncertain target at times k and k-1, respectively; Z(k) represents the observation vector of the overlapping boxes corresponding to the uncertain target at time k; f(k) represents the state transition matrix; h(k) represents the observation function; and V(k) and W(k) represent Gaussian white noise.

[0124] S5.2. Based on the state of the uncertain target at time k-1, the predicted state equation of the uncertain target at time k is:

[0125]

[0126] in, Let F represent the state prediction vector and prediction error vector of the overlapping box corresponding to the uncertain target at time k, F be the Jacobian matrix of f(k) at X(k-1|k-1), P(k-1|k-1) be the prediction error vector of the overlapping box corresponding to the uncertain target at time k-1, and Q be the covariance matrix of the process noise.

[0127] S5.3. The state update equation for the overlapping box corresponding to the uncertain target is derived from the state equation in Equation 9 as follows:

[0128]

[0129] In equation (10), Here, S(k) represents the state update vector, S(k) represents the innovation covariance, K(k) represents the gain matrix, P(k|k-1) is the prediction error covariance, and H is the state update vector. The Jacobian matrix at point R is the covariance matrix of the observation noise.

[0130] S5.4. The final state equation of the uncertain target is derived from the state update equation of equation (10):

[0131]

[0132] In equation (11), X(k|k) represents the state estimation vector of the uncertain target at time k, P(k|k) represents the covariance at time k, and K(k) represents the gain matrix. P(k|k-1) is the state update vector, P(k|k-1) is the prediction error covariance, and I represents the identity matrix.

[0133] Equation (8) represents the state equation and prediction equation of the overlapping box corresponding to the uncertain target during the entire EKF tracking process. Equation (9) is the state equation for predicting time k based on time k-1. Based on equation (9), the state update equation (10) of the uncertain target is derived. Based on equation (10), the final state equation (11) of the uncertain target is derived.

[0134] S5.5 After the above steps of iterative update, the number of iterations is 10. If the overlapping box corresponding to the tracked uncertain target appears 3 times in a row, then the overlapping box is output as the fusion result, indicating that the overlapping box is the same obstacle that was identified, that is, the uncertain target is actually a definite target.

[0135] S6, the determined target obtained from S4.2, and the determined target obtained from the uncertain target obtained from S4.2 after processing by S5 together represent the detected obstacles on the railway track.

[0136] In this embodiment, the millimeter-wave radar used is Continental's ARS408-21, which operates at a sampling frequency of 20Hz and can achieve medium and long-range detection capabilities; the camera used is a traffic camera module manufactured by Yutong Optics, with a sampling frequency of approximately 30Hz.

[0137] In actual obstacle detection scenarios on railway tracks, a detection system including millimeter-wave radar sensors and camera sensors is first installed on the side of the track. When an obstacle appears on the track, the radar scans the obstacle to obtain a point cloud. After multi-frame overlay and DBSCAN clustering, the number of points in the actual reflected point cloud of the obstacle increases from the usual 11 to 44. Figure 4 ).

[0138] After improvements to the YOLOv5s algorithm and further optimization on the training platform, A P The indicator is ≥88%, M AP With a confidence level of ≥81%, it becomes a lightweight YOLOv5s algorithm. After being loaded into the visual AI board, the actual detection confidence level is 0.82, which is 0.08 higher than the unmodified YOLOv5s confidence level of 0.74. Figure 5 The camera sensor captures image data of obstacles, which is then processed by the lightweight YOLOv5s algorithm to obtain the region of interest (ROI). C ROI C The size is 110px × 280px.

[0139] The information from the two sensors is synchronized in time. After multi-frame overlay and DBSCAN clustering, the point cloud is projected onto the pixel coordinate system through coordinate transformation, i.e., onto the image captured by the camera. The actual reflection point cloud of the obstacle forms the region of interest (ROI) on the image. R ROI R The size is 103px × 275px. Figure 6 Region of Interest (ROI) for radar R Region of Interest (ROI) C The fusion detection box is composed of [components]; after calculation, S R ∩S C 24980px, S R ∪S C It is 32025px, according to S was calculated IoU A value of 0.78 indicates that the target has been identified, meaning an obstacle on the rails has been detected.

[0140] The core innovation of this invention lies in its unique combination of sensor fusion strategy and advanced data processing algorithms. Firstly, at the data acquisition and processing level, this invention employs multi-frame autonomous fusion technology to deeply integrate point clouds continuously acquired by millimeter-wave radar, effectively solving the problem of information loss due to noise or occlusion in single-frame data. Simultaneously, it introduces the DBSCAN clustering algorithm based on 3D bounding boxes, which can accurately extract potential target regions from complex backgrounds, laying a solid foundation for subsequent target recognition and tracking. Regarding sensor fusion, this invention abandons the traditional simple data overlay method and innovatively proposes a decision-level fusion strategy based on Intersection over Union (IOU). This strategy not only considers the spatial relationship between radar and camera detection results but also evaluates their consistency by calculating the overlap of the region of interest, thereby achieving more accurate target screening and confirmation. This fusion method not only improves detection accuracy but also reduces false alarms and missed alarms. Furthermore, in the target tracking stage, this invention introduces the extended Kalman filter algorithm. This algorithm combines the target's motion model with observation data to smooth and dynamically predict the fused target trajectory, effectively suppressing noise interference and improving the stability and continuity of target tracking. The application of this technology enables the invention to maintain efficient and accurate detection performance even in complex and ever-changing orbital environments. In summary, the technological innovations of this invention cover multiple key aspects such as data acquisition, processing, fusion, and tracking, bringing a completely new solution to the field of foreign object detection in orbital systems.

[0141] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or variations that can be easily conceived by those skilled in the art within the technical framework disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the claims of the present invention should be used as the standard for determining the scope of protection of this application.

Claims

1. A method for detecting foreign objects on railway tracks based on millimeter-wave radar sensors and camera sensors, comprising the following steps: S1. A millimeter-wave radar sensor detects obstacles on the railway track ahead, obtaining point clouds, which are then preprocessed to obtain a large frame dataset. The specific preprocessing process is as follows: To address the sparse point cloud acquisition by millimeter-wave radar sensors, the acquired point cloud is first subjected to multi-frame autonomous fusion processing. The data after multi-frame autonomous fusion processing is then processed based on the DBSCAN clustering algorithm of 3D bounding boxes to obtain a large frame dataset. The specific operation method of the multi-frame autonomous fusion processing is as follows: After reading the obtained point cloud, the point cloud of the current frame is superimposed with the point cloud of the previous three frames to increase the density. That is, the obtained point cloud is superimposed into a large frame every 4 frames. By continuously superimposing 4 frames of point cloud, a large frame set is created. The DBSCAN clustering algorithm processing includes the following steps: the vertical space with the railway track as the bottom surface is taken as the area to be measured, the millimeter-wave radar scans the area to be measured to obtain point cloud, and after multi-frame autonomous fusion processing, a large frame set is formed; after the large frame set is processed by the DBSCAN clustering algorithm in turn, the points in the point cloud obtained by the millimeter-wave radar sensor that belong to the real reflection points of the obstacle are filtered out to form a large frame dataset. S2. The camera sensor captures images of obstacles on the railway track ahead, obtaining image data. The lightweight YOLOv5s algorithm is then used to process the data to obtain the region of interest (ROI). C The specific steps are as follows: S2.1 Improve the focus layer and CSP module in the YOLOv5s algorithm to obtain a lightweight YOLOv5s algorithm; S2.

2. The lightweight YOLOv5s algorithm obtained in S2.1 is loaded into the visual AI board. The image data obtained by the camera sensor from the obstacles on the railway track ahead is processed by the lightweight YOLOv5s algorithm to obtain the visual region of interest (ROI). C ROI C The size is S C S C The pixel area of ​​the obstacle as captured by the camera in pixel coordinates; S3. Synchronize the large frame dataset obtained in S1 with the image data in S2 in time. Then, project the large frame dataset obtained in S1 onto the pixel coordinate system to generate the radar region of interest (ROI) in the pixel coordinate system. R Then, the radar region of interest (ROI) R Region of Interest (ROI) C A fusion detection bounding box is formed in the pixel coordinate system; the specific process of step S3 is as follows: S3.

1. Synchronize the large frame dataset obtained in S1 with the image data in S2 in time. That is, perform time registration between the large frame dataset obtained in S1 and the image data in S2, using the least common multiple of periods principle for time synchronization. The specific process is as follows: In S1, point clouds are stacked into a large frame every 4 frames, meaning the time interval for forming a large frame is T. Due to time synchronization, the camera also needs to take pictures at intervals of T to obtain an image. Therefore, the least common multiple T is chosen as the sampling period for time synchronization. The sampling time interval of the millimeter-wave radar is obtained according to the following formula (3): Where T is the sensor's operating period and f is the sampling frequency; S3.2 Project the large frame dataset obtained in S1 onto the pixel coordinate system. Generate a radar region of interest (ROI) in the pixel coordinate system from the point cloud in each large frame dataset. R ; S3.3: In the pixel coordinate system, the radar region of interest (ROI) corresponding to each large frame of data was obtained. R and the region of interest (ROI) corresponding to the image data acquired by the camera in time. C Radar Region of Interest (ROI) R Region of Interest (ROI) C The total pixel area of ​​the union is the fusion detection box; S4. Using the cross-union ratio (CUIR) decision-level fusion method, calculate the radar region of interest (ROI) corresponding to each large frame of data. R and the region of interest (ROI) corresponding to the image data acquired by the camera in time. C To determine the overlap and obtain preliminary fusion results, the following steps are included: S4.1: Calculate the region of interest (ROI) for each radar to synchronize with time. R and each region of visual interest (ROI) C overlap S IoU The calculation method is as follows: Among them, S R Region of Interest (ROI) for radar R The size of the pixel area in pixel coordinates, S C Region of Interest (ROI) C The size of a pixel area in pixel coordinates; S4.2: Set the detection threshold for evaluation. The evaluation method is as follows: S IoU When the value is less than 0.3, the obstacle detected by the millimeter-wave radar and the camera are considered to be different obstacles and are discarded, i.e., false targets; 0.3≤S IoU When the value is less than 0.5, the obstacle detected by the millimeter-wave radar and the camera may be the same obstacle, i.e., the target is uncertain. 0.5≤S IoU When the value is ≤1, the obstacle detected by the millimeter-wave radar and the camera is considered to be the same obstacle, thus the target is identified. Therefore, by judging the degree of overlap S IoU The point cloud and image data reflecting the detected obstacles are initially fused to obtain preliminary fusion results: false targets, confirmed targets, and uncertain targets. S5. The extended Kalman filter algorithm is used to dynamically track the overlapping boxes corresponding to the uncertain targets in the preliminary fusion result obtained in S4, and the fusion result is output; including the following steps: In S5.1 and S4, the overlapping box state vector corresponding to the uncertain target is defined as X = (u, v, v) u ,v v ) T , where u, v, v u v v These represent the x-coordinate, y-coordinate, x-axis velocity, and y-axis velocity of the overlapping bounding box corresponding to the uncertain target in the pixel coordinate system, respectively. u v v That is, the velocity of the actual reflected point cloud of the obstacle within the overlapping frame; Using the extended Kalman filter algorithm, the state equation and prediction equation for the overlapping boxes are as follows: Where X(k) and X(k-1) represent the state vectors of the overlapping boxes corresponding to the uncertain target at times k and k-1, respectively; Z(k) represents the observation vector of the overlapping boxes corresponding to the uncertain target at time k; f(·) represents the state transition matrix; h(·) represents the observation function; V(k) and W(k) represent Gaussian white noise. S5.

2. Based on the state of the uncertain target at time k-1, the predicted state equation of the uncertain target at time k is: in, P(k|k-1) represents the state prediction vector and prediction error covariance matrix of the overlapping box corresponding to the uncertain target at time k, F is the Jacobian matrix of f(·) at X(k-1|k-1), P(k-1|k-1) is the prediction error covariance matrix of the overlapping box corresponding to the uncertain target at time k-1, and Q is the covariance matrix of process noise. S5.

3. The state update equation for the overlapping box corresponding to the uncertain target is derived from the state equation in Equation 9 as follows: In equation (10), Here, S(k) represents the state update vector, S(k) represents the innovation covariance, K(k) represents the gain matrix, P(k|k-1) is the prediction error covariance matrix, and H is the state update vector. The Jacobian matrix at point R is the covariance matrix of the observation noise; S5.

4. The final state equation of the uncertain target is derived from the state update equation of equation (10): In equation (11), X(k|k) represents the state estimation vector of the uncertain target at time k, P(k|k) represents the error covariance matrix at time k, and K(k) represents the gain matrix. P(k|k-1) is the state update vector, P(k|k-1) is the prediction error covariance matrix, and I represents the identity matrix. S5.5 After the above steps of iterative update, the number of iterations is 10. If the overlapping box corresponding to the tracked uncertain target appears continuously for no less than 3 times, then the overlapping box is output as the fusion result, indicating that the overlapping box is the same obstacle that was identified, that is, the uncertain target is actually a definite target. S6, the determined target obtained from S4.2, and the determined target obtained from the uncertain target obtained from S4.2 after processing by S5 together represent the detected obstacles on the railway track.

2. The method for detecting foreign objects on railway tracks according to claim 1, characterized in that, In step S2.1, the process of obtaining the lightweight YOLOv5s algorithm is as follows: S2.1.1, the YOLOv5s algorithm includes a focus layer that transforms a feature map of size H×W×C into... Size, where H and W represent the height and width of the feature map, respectively, and C represents the number of channels in the feature map; to improve the detection speed and accuracy of the YOLOv5s algorithm, the focus layer is improved to transform a feature map of size H×W×C into... The size is adjusted to effectively increase the channel dimension without losing feature map information; S2.1.2 Replace the CSP module in the YOLOv5s algorithm with the MobileNetV3 module. The specific process is as follows: In the YOLOv5s algorithm, the MobileNetV3 module is used to replace the original CSP module in the backbone. (1) Perform depthwise separable convolutions included in the MobileNetV3 module on each feature map channel of the input CSP module, and then perform pointwise convolutions included in the MobileNetV3 module on all feature map channels of the input CSP module, so as to replace the standard convolution of the input feature map with the depthwise separable convolutions included in the MobileNetV3 module; (2) Replace the residual block connection mechanism in the CSP module with the inverse residual structure included in the MobileNetV3 module. The inverse residual structure expands the number of feature map channels and then performs dilation convolution operation, and then compresses back to the original number of channels; (3) Replace the output layer of the CSP module with the SE module included in the MobileNetV3 module. S2.1.

3. The improved YOLOv5s algorithm obtained from S2.1.1 and S2.1.2 is optimized using the Detection On Tracks dataset to obtain a lightweight YOLOv5s algorithm. The Detection On Tracks dataset is divided into training, validation, and test sets in a ratio of 8:1:

1. Then, the training, validation, and test sets are used sequentially for training, validation, and testing. The training iterations are 300 times until the evaluation metric meets the accuracy requirements. The evaluation criteria are as follows: Among them, the mean and average precision M AP It is one of the metrics for object detection, and it is the mean of the average accuracy across all categories. A P The average precision for each class is given, where m is the number of classes, and p and r are the precision and recall of the algorithm, respectively. If the above precision evaluation criteria are not met, the Detection On Tracks dataset splitting, training, validation, and testing are repeated, and the parameters are adjusted and updated until the precision meets the requirements.

3. The method for detecting foreign objects on railway tracks according to claim 1, characterized in that, The specific process of S3.2 is as follows: S3.2.1 Transformation from millimeter-wave radar coordinate system to camera coordinate system: In equation (4), X C Y C Z C This represents the three-dimensional coordinates of the actual reflection point of the obstacle in the camera coordinate system; X R Y R Z R R is the three-dimensional coordinate of the actual reflection point of the obstacle in the millimeter-wave radar coordinate system, where R is the rotation matrix and t is the displacement matrix. It is the camera extrinsic parameter matrix; S3.2.2, Transform from camera coordinate system to pixel coordinate system: Where u and v are the coordinates of the actual reflection point of the obstacle target in the pixel coordinate system. It is the camera intrinsic parameter matrix, where f x It is the focal length in the horizontal direction, measured in pixels; f y It is the focal length in the vertical direction, measured in pixels; u o It is the pixel coordinate of the image center in the horizontal direction, v o These are the pixel coordinates of the image center in the vertical direction; S3.2.3, Transform the coordinates of each point in the point cloud of the large frame dataset obtained in S1 into S3.2.1 and S3.2.2 in sequence, so that the point cloud of the large frame dataset obtained in S1 in the millimeter-wave radar coordinate system is projected into the pixel coordinate system. S3.2.

4. In the pixel coordinate system, the pixel coordinates of the points inside the point cloud of the large frame dataset obtained in S1 are used to approximate the size of the 3D bounding box, and the radar region of interest (ROI) is generated. R ROI R Equation (6) represents the ROI corresponding to each large frame in the large frame dataset obtained by S1. R The size is S R S R The size of the pixel area projected onto the pixel coordinate system from the point cloud, i.e., the actual reflection point of the obstacle, in each large frame of data; ROI R =(u,v,w,h,t)∈R (6) Where u and v represent the coordinates of the actual reflection point of the obstacle detected by the millimeter-wave radar in the pixel coordinate system, w, h, and t are the width, height, and depth of the three-dimensional bounding box in the pixel coordinate system, respectively, and R represents the entire pixel coordinate system.

Images