A method for detecting intruders on a rail using a millimeter wave radar sensor and a camera sensor

Abstract

The application belongs to the technical field of railway track safety, and specifically discloses a rail on-intrusion object detection method based on heterogeneous fusion of a millimeter wave radar sensor and a camera sensor. The detection method comprises the following steps: S1, synchronizing the time stamps of the millimeter wave radar sensor and the camera sensor and aligning the spatial coordinates; S2, data acquisition and preprocessing; S3, processing the point set obtained in S2 by using a PCA algorithm and a K-means algorithm to obtain speed v r , position c k and shape; processing the image data set obtained in S2 by using a YOLOv8 model to obtain two-dimensional coordinates s i =(x i ,y i ), speed v c and shape features; S4, fusing the speed v r , position c k and two-dimensional coordinates s i =(x i ,y i ), speed v c , and fusing the shape and shape features; S5, inputting the fused features into the YOLOv8 model again to obtain a detection result. The millimeter wave radar sensor and the camera sensor are fused to detect the intrusion of foreign objects, the advantages of the millimeter wave radar and the camera can be complementary, the robustness and accuracy of detection are improved, and the real-time safety of the railway is ensured.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of railway track safety, and particularly relates to a method for detecting intruding objects on a rail by fusing a millimeter wave radar sensor and a camera sensor. BACKGROUND

[0002] As a symbol of modernity, the railway has been playing a core role in the process of national unity and economic and social development in China since the late Qing Dynasty. As the main artery of the national economy, the railway undertakes a large number of personnel and cargo transportation tasks. Once a safety accident occurs, not only will it cause huge economic losses, but also may cause serious casualties and social impact. Therefore, ensuring railway safety is the primary task of railway work.

[0003] CN118072233A discloses a computer vision and radar fusion single-track foreign object detection method, which mainly emphasizes vision as the main detection. Only when the vision detects the intrusion of foreign objects, the radar fusion detection will be performed. The model does not consider the situation that the vision detection is easily affected by the environment and has a certain misjudgment rate.

[0004] In order to more accurately detect the intrusion of foreign objects on the railway and improve the safety factor of the railway, a more accurate detection method needs to be introduced, specifically using millimeter wave radar and vision for fusion, and detecting them at the same time. No matter which sensor detects the intrusion of foreign objects, fusion detection needs to be performed to improve the accuracy and reduce the misidentification rate, thereby ensuring the real-time safety of the railway. SUMMARY

[0005] In view of the defects in the prior art, the purpose of the present application is to provide a method for detecting intruding objects on a rail by fusing a millimeter wave radar sensor and a camera sensor. The advantages of the millimeter wave radar sensor and the camera sensor are utilized for fusion detection to detect the intrusion of foreign objects on the rail, so as to achieve real-time detection of the intrusion of foreign objects on the railway and ensure the real-time safety of the railway.

[0006] To achieve the above-mentioned purpose, the technical scheme adopted by the present application is a method for detecting intruding objects on a rail by fusing a millimeter wave radar sensor and a camera sensor, comprising the following steps:

[0007] S1, sensor synchronization: synchronize the time stamps of the millimeter wave radar sensor and the camera sensor, and align the spatial coordinates to ensure that the data obtained by the two sensors correspond to the same target;

[0008] First, the NTP network time protocol is used for time stamp synchronization; then, the spatial coordinate alignment is performed. The spatial coordinate alignment is to convert the point cloud obtained by the millimeter wave radar sensor from the radar coordinate system to the camera coordinate system, so as to realize the unification of the point cloud and the image data in the camera coordinate system, thereby effectively performing data fusion.

[0009] Preferably, in the step S1, the specific process of the timestamp synchronization is:

[0010] The clock bias of the millimeter wave radar sensor is:

[0011]

[0012] The clock bias of the camera sensor is:

[0013]

[0014] The synchronized millimeter wave radar sensor timestamp is:

[0015] T r sync = T r recv + θ r (3)

[0016] The synchronized camera sensor timestamp is:

[0017] T c sync = T c recv + θ c (4)

[0018] Wherein, T r req is the time when the millimeter wave radar sensor sends the NTP request, T r recv is the time when the millimeter wave radar sensor receives the response, T c req is the time when the camera sensor sends the NTP request, T c recv is the time when the camera sensor receives the response, T s recv is the time when the time server receives the request, T s send is the time when the time server sends the response.

[0019] The specific process of the spatial coordinate alignment is:

[0020]

[0021] Wherein, is the coordinate of the reflection point of the intruder in the camera coordinate system, is the coordinate of the reflection point of the intruder in the radar coordinate system, R is the rotation matrix of the radar coordinate system converted to the camera coordinate system, and T is the translation vector of the radar coordinate system converted to the camera coordinate system.

[0022] S2, data acquisition and preprocessing: adjust the millimeter wave radar sensor and the camera sensor to the best state, detect the intruder on the track at the same time, obtain point cloud and image data respectively, and then preprocess the collected data; the specific process of the preprocessing is as follows:

[0023] The point cloud collected by the millimeter wave radar sensor includes three-dimensional coordinates, intruder speed and intruder reflection intensity. Remove the data with the intruder reflection intensity less than 5dB, save the processed point cloud, and obtain the point set; crop and scale the image data collected by the camera sensor to the size of 640px*640px, and then perform normalization processing to normalize the pixel value from [0, 255] to the interval [0, 1] to meet the input requirements of the YOLOv8 model, and then use the labelImg tool to label the image data in the YOLO label format, the labeled label includes people, animals, rockfall and mud flow blockage, and the image data set is output.

[0024] S3, data processing: using PCA algorithm and K-means algorithm to process the point set obtained in S2, the obtained features include speed v r , position c k and shape, to detect the intruder; at the same time, using YOLOv8 model to process the image data set obtained in S2, obtaining two-dimensional coordinates s i =(x i ,y i ), speed v c and shape features to detect the intruder.

[0025] Preferably, in step S3, the processing mode of the point set specifically includes the following steps:

[0026] S3.1, the coordinates of the points in the point set are three-dimensional, but the coordinates of the image data are two-dimensional, and PCA algorithm is used for data dimension reduction:

[0027] First, before executing the PCA algorithm, standardize the points so that each feature of the points (i.e. three-dimensional coordinates, intruder speed) has a mean of zero and a standard deviation of 1. This is because the PCA algorithm relies on the covariance between features, and standardization can eliminate the influence between different scales. The formula used in standardization processing is:

[0028]

[0029] Where X is the given data matrix, μ is the mean of each feature, and σ is the standard deviation of each feature;

[0030] Then, the core idea of ​​the PCA algorithm is to find the direction with the largest variance in the data. In order to find the direction with the largest variance in the point concentration, it is necessary to calculate the covariance matrix Σ of the data matrix, which is calculated as follows:

[0031]

[0032] Among them, X scaled is the normalized data matrix.

[0033] Next, we use eigenvalue decomposition to find the direction with the largest variance in the point concentration: perform eigenvalue decomposition on the covariance matrix Σ to obtain the eigenvalues ​​and eigenvectors: Σ = λ i v i , where v i is the i-th eigenvector of the covariance matrix Σ, λ i is the i-th eigenvalue of the covariance matrix Σ.

[0034] The eigenvector represents the direction of data projection, and the eigenvalue represents the variance in that direction. The purpose of the PCA algorithm is to retain the direction with the largest variance, thereby reducing the dimensionality of the data. Usually, the first k eigenvectors with the largest eigenvalues ​​(i.e., principal components) are selected to perform data dimensionality reduction. Select the first k eigenvectors with the largest eigenvalues ​​v1, v2, ..., v k , forming a matrix V k , where each column is a feature vector.

[0035] Finally, by projecting the point set onto the matrix V k , get the point set X after dimensionality reduction pca , according to the formula:

[0036] X pca =X scaled V k (8)

[0037] S3.2, the dimensionality-reduced point set X obtained in S3.1 pca Project the image to the pixel coordinate system synchronized with the timestamp for visualization.

[0038] Since the obtained dimensionality reduction point set X pca The midpoint is mainly a two-dimensional coordinate (x ipca ,y ipca ), which is two-dimensional data. The two-dimensional data can be mapped to the pixel coordinate system through the camera intrinsic parameter matrix L. The intrinsic parameter matrix is:

[0039]

[0040] Among them, f x and f y are the focal lengths of the camera sensor along the x and y axes, respectively, cx and c y are the coordinates of the principal point of the image (usually the center of the image).

[0041] Based on the internal parameter matrix L, through the formula The point set X after dimensionality reduction can be directly pca Projected onto the pixel coordinate system to obtain the pixel coordinate set {(u1,v1),(u2,v2),...,(u n ,v n )}, where (u i ,v i ) is the pixel coordinate of the i-th point in the point set after dimensionality reduction projected in the pixel coordinate system.

[0042] S3.3. Use the K-means clustering algorithm to cluster the pixel coordinate set obtained in S3.2. The K-means clustering algorithm divides the data set into several clusters (a cluster is a cluster or set) to extract key features. The specific process is as follows:

[0043] S3.3.1. Initialize the cluster centers and select m initial cluster centers {c1, c2, ..., c m}, c m =(u m ,v m ), where each c m It is a point in the pixel coordinate system;

[0044] S3.3.2, calculate each point (u i ,v i ) to the distance of m initial cluster centers, and each point (u i ,v i ) is assigned to the initial cluster center with the smallest distance, thus obtaining m clusters. The distance is the Euclidean distance:

[0045] S3.3.3. Update: Recalculate the center of each of the m clusters. The new cluster center is the mean of all points in each cluster: Among them, S m is the set of points assigned to the mth cluster, |S m | is the number of points in the mth cluster.

[0046] S3.3.4. Iteration: Repeat steps S3.3.2 and S3.3.3. When the cluster center no longer changes or the maximum number of iterations is reached, the K-means algorithm stops and several clusters are obtained.

[0047] S3.4, S3.3 obtains several clusters, the position, shape, speed characteristics of the cluster are extracted to detect the intruder. The specific extraction method is as follows:

[0048] First, the position characteristics of the cluster are obtained by calculating the centroid coordinates of the cluster (the centroid coordinates are taken as the position characteristics), and the calculation formula of the centroid coordinates is: And the width and height of the bounding box of each cluster are calculated: W k = u max - u min , H k = v max - v min ;

[0049] Where |S k | is the number of points in the kth cluster, c k is the centroid coordinates of the kth cluster, W k and H k are the width and height of the bounding box of the kth cluster, and the minimum coordinates (u min , v min ) and the maximum coordinates (u max , v max ) of the bounding box are obtained from and .

[0050] Then, the shape characteristics of the cluster are obtained, and the specific process is: according to the width and height of the bounding box of each cluster, the area A k of the cluster is calculated: k = W k × H k , according to the area A k , the aspect ratio AR k and the circularity C k are calculated: According to the values of the two, the shape of the cluster is judged: for rectangle or ellipse, the value of the aspect ratio is usually large (greater than 1), and for objects close to a circle, the aspect ratio will be close to 1; the value of the circularity is between [0, 1], the closer the value is to 1, the closer the shape of the object is to a circle, and the smaller the value, the more irregular the shape of the object; wherein P k is the boundary perimeter of the kth cluster, which is obtained as follows: the points P = {p1, p2,..., p n} of the cluster are all on a two-dimensional plane, the GrahamScan convex hull algorithm is used to obtain the convex hull of the cluster, that is, the convex boundary point H = {h1, h2,..., h m} of the cluster, and then the Euclidean distance between adjacent points in the convex hull is calculated: And the Euclidean distance between the mth point and the first point: The final boundary perimeter P can be obtained by the following formula k :

[0051]

[0052] where h i = (x i , y i ) is the i-th boundary point.

[0053] S3.5, after extracting the position feature and shape feature of the cluster in S3.4, the speed feature is extracted according to the displacement and time difference method, and the extraction method is: the position features of the intruder in the continuous time frames t1 and t2, i.e. the centroids c1 and c2, are extracted, and the speed feature is obtained by the following formula:

[0054]

[0055] Finally, the feature extraction of the point cloud collected by the millimeter wave radar is completed, and the extracted features include the speed v r , the position c k and the shape.

[0056] Preferably, in step S3, the processing method of the image data set specifically includes the following steps:

[0057] S3.5, obtaining the CIoU loss function model

[0058] S3.5.1, the image data set obtained in S2 is divided into a training set, a validation set and a test set according to the ratio of 7:1:2, which is used to optimize the improved CIoU loss function; wherein the training set is used for training the improved CIoU loss function, and the optimized CIoU loss function is obtained; the validation set is used to verify the effect of the optimized CIoU loss function offline in order to adjust the parameters; the test set is used to test the generalization ability of the CIoU loss function model obtained after the verification is completed, and to judge whether it maintains the same performance on other data.

[0059] S3.5.2, construction of the improved CIoU loss function, the specific process is as follows:

[0060] The CIoU calculation formula is: where b = (x b , y b , w b , h b ) is the center coordinates (x b , y b ) of the real boundary box, the width w b and the height h b , is the center coordinates width and height is the intersection over union of the real bounding box and the predicted bounding box, is the Euclidean distance between the center points: c is the diagonal length of the minimum closed box: where x max , y max and x min , y min are the diagonal coordinates of the minimum rectangle containing the real bounding box and the predicted bounding box, and v is the aspect ratio consistency measure: α is an adjustment factor, 0 < α < 1, used to balance the various parts in the loss, usually set as a hyperparameter.

[0061] Since the original CIoU relies too much on the distance between the center points, the weight of the distance between the center points is too large, especially when the scale difference of the object is large, which may lead to inaccurate optimization of the frame of the smaller object and imperfect aspect ratio adjustment, therefore a scale adaptive loss function is introduced, which is designed to adjust the weight of each index (such as the distance between the center points, the aspect ratio adjustment) under different scales and improve the aspect ratio loss, and the Chamfer distance is introduced to replace the original aspect ratio loss, which can capture the shape difference of the bounding box more finely and improve the detection accuracy. The calculation formula of the scale adaptive loss function is:

[0062] Loss function Under different scales, the weight of each index (such as the distance between the center points, the aspect ratio adjustment) is adjusted and the aspect ratio loss is improved, and the Chamfer distance is introduced to replace the original aspect ratio loss, which can capture the shape difference of the bounding box more finely and improve the detection accuracy. The calculation formula of the scale adaptive loss function is:

[0063]

[0064] where A b = w b x h b is the area of the real bounding box b, A avg is the average area of all real bounding boxes, and λ is a tunable hyperparameter used to control the sensitivity of the scale difference; P = {(x1, y1), (x2, y2), (x3, y3), (x4, y4)} and P' = {(x1', y1'), (x2', y2'), (x3', y3'), (x4', y4')} are the four vertex sets of the real bounding box and the predicted bounding box, respectively.

[0065] According to equations (15), (16) and (17), the improved CIoU loss function is:

[0066]

[0067] S3.5.3, optimization of the improved CIoU loss function

[0068] ​The training set, the validation set and the test set are used to train, validate and test the improved CIoU loss function respectively, so as to optimize the improved CIoU loss function, until the evaluation index meets the accuracy requirement, and the evaluation standard is that the average precision mean (mAP) is improved by more than 0.02, the recall rate (Recall) is improved by more than 0.03, the precision (Precision) is improved by more than 0.02, and the positioning error is reduced; if the evaluation standard is not met, steps S3.5.1 and S3.5.2 are repeated, the parameters are adjusted and updated, until the error meets the requirement or the maximum iteration number is reached; after optimization, the CIoU loss function model is obtained, the optimized parameters in the CIoU loss function model are saved, and are used for subsequent intrusion detection.

[0069] S3.6, the training set, the validation set and the test set described in S3.5.1 are used to train, validate and test the YOLOv8 algorithm (including MobileNetV3 module and YOLO detection head module) respectively, so as to optimize the YOLOv8 algorithm, until the evaluation index meets the requirement, and the evaluation standard is that the average precision mean (mAP) is improved by more than 0.02, the recall rate (Recall) is improved by more than 0.03, the precision (Precision) is improved by more than 0.02, and the positioning error is reduced; if the evaluation standard is not met, the scale adaptive loss function λ is adjusted and the training is continued until the evaluation standard is met. The optimized YOLOv8 algorithm has faster training speed and better training result, so the optimized YOLOv8 algorithm is named as improved YOLOv8 model, and the optimized parameters obtained in S3.5.3 are loaded into the improved YOLOv8 model; then, the image data normalized in S2 is input into the MobileNetV3 module of the improved YOLOv8 model, and the features from low level to high level are extracted from the image data normalized in S2, including two-dimensional coordinates s i =(x i ,y i ), speed v c , and shape features, so as to convert the image data into multi-level feature maps, and save the feature maps for subsequent intrusion detection. The speed can be obtained according to two time frames t1 and t2 and combined with the two-dimensional coordinates s1 and s2 of the intrusion, that is:

[0070]

[0071] S3.7, the feature maps obtained in S3.6 are input into the YOLO detection head module included in the improved YOLOv8 model, so as to generate a plurality of prediction boxes in the feature maps obtained in S3.6.

[0072] S3.8, processing the feature map containing multiple prediction boxes obtained in S3.7 by using a non-maximum suppression (NMS) algorithm, removing redundant boxes among the multiple prediction boxes contained in the feature map, and retaining the optimal box, thereby obtaining an intruder feature map.

[0073] S4, data fusion: fusing the speed v r , position c k obtained in S3 with the two-dimensional coordinates s i = (x i , y i ), respectively, and the speed v c by using a weighted average method.

[0074] Preferably, the fusion of the speed v r , position c k obtained in S3 with the two-dimensional coordinates s i = (x i , y i ), respectively, and the speed v c by using a weighted average method in step S4 is as follows:

[0075] First, different weights are assigned to the features of the point cloud obtained by the millimeter wave radar sensor scanning and the image data obtained by the camera sensor scanning, respectively The weights satisfy and Then, the features corresponding to each intruder i, including the radar features {F Radar,1 , F Radar,2 ,...} and the camera features {F Camera,1 , F Camera,2 ,...}, are fused by using a weighted average method to obtain the fused features

[0076]

[0077] wherein F Radar = (c k , v r ), F Camera = (s i , v c ), and F fusion = {(x, y, z), Speed: _m / s}

[0078] Preferably, the fusion of the shape and shape features obtained in S3 by using a cosine similarity judgment and then by using a weighted average method in step S4 is as follows:

[0079]

[0080] Among them, f radar and f camera These are the shape extracted by S3.4 and the shape features extracted by S3.8. The extracted shapes and shape features are both vector data. radar || and ||f camera || represents the modulus of the vector, also called the norm, and the calculation formula is: Among them, a i are the components of their respective vectors.

[0081] The cosine similarity value range is: [-1,1], 1 means completely similar, -1 means completely dissimilar, and 0 means they are irrelevant. Therefore, when sim(f radar ,f camera )>0.5, weighted average fusion is performed; the fusion formula is:

[0082]

[0083] S5. Detect intruders: The fused features obtained in S4 are input into the YOLOv8 model in S3 again, and intruder detection is performed again to obtain the intruder detection results.

[0084] Preferably, the specific process of S5 is:

[0085] First, the fused features obtained by S4 and f fused Input again into the improved YOLOv8 model in S3.6, the improved YOLOv8 model includes a MobileNetV3 module from the input and f fused Extract features from low-level to high-level, and combine the fused features and f fused Converted into a multi-level feature map. Then, the feature map is input into the YOLO detection head module included in the improved YOLOv8 model to generate a prediction box. It is then processed by the non-maximum suppression (NMS) algorithm to remove redundant boxes in the prediction box and retain the optimal box in the prediction box. The optimal box is the detection box, and the detection result is displayed in the detection box, and the confidence is obtained. The detection result is an intruder detection map containing all features, including the fused position (x, y, z) (z comes from the three-dimensional coordinates obtained by millimeter-wave radar detection), speed Speed: _m / s, and the shape extracted based on the point cloud collected by the millimeter-wave radar sensor, the confidence obtained from the fused information, and other information. Finally, the detection result is output and saved.

[0086] Compared with the prior art, the application has the following advantages and beneficial effects:

[0087] The application provides a rail on-intrusion object detection method based on heterogeneous fusion of a millimeter wave radar sensor and a camera sensor, real-time detection is performed through the millimeter wave radar sensor and the camera sensor, after an object is detected, the millimeter wave radar sensor and the camera sensor are associated by using a Euclidean distance, real-time tracking is then performed by using a YOLOv8 model respectively, finally, tracking information is fused into a video result by using a weighted average method to perform judgment, and the safety on a railway is ensured in real time. The application fully gives play to the advantages of the millimeter wave radar and the camera to perform real-time detection, can detect all-weather and is not affected by weather, the accuracy and robustness of detection are improved by fusing the information of the two sensors, and the false alarm rate is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0088] Figure 1 A millimeter wave radar and camera heterogeneous fusion railway foreign object detection method flowchart;

[0089] Figure 2 A millimeter wave radar and camera fusion detection framework diagram;

[0090] Figure 3 An improved YOLOv8 model architecture diagram;

[0091] Figure 4 A point cloud actual processing diagram collected by a millimeter wave radar sensor;

[0092] Figure 5 A multi-level feature map after feature extraction of image data at the same time by the improved YOLOv8 model;

[0093] Figure 6 A detection result obtained by the application in an experiment. DETAILED DESCRIPTION

[0094] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present application.

[0095] The embodiments of the application will be further described in detail below in connection with the drawings.

[0096] As Figure 1 , 2As shown, a rail on invasion object detection method of millimeter wave radar sensor and camera sensor heterogeneous fusion, comprising the following steps:

[0097] S1, sensor synchronization: synchronize the timestamps of millimeter wave radar sensor and camera sensor, align the space coordinates, and ensure that the data obtained by the two sensors correspond to the same target;

[0098] First, use NTP network time protocol for timestamp synchronization; then align the space coordinates, the space coordinate alignment is to convert the point cloud obtained by the millimeter wave radar sensor from the radar coordinate system to the camera coordinate system, thereby realizing the unification of point cloud and image data in the camera coordinate system, and effectively performing data fusion.

[0099] The specific process of timestamp synchronization is:

[0100] The clock deviation of the millimeter wave radar sensor is:

[0101]

[0102] The clock deviation of the camera sensor is:

[0103]

[0104] The synchronized millimeter wave radar sensor timestamp is:

[0105] T r sync = T r recv + θ r (3)

[0106] The synchronized camera sensor timestamp is:

[0107] T c sync = T c recv + θ c (4)

[0108] Where, T r req is the time when the millimeter wave radar sensor sends an NTP request, T r recv is the time when the millimeter wave radar sensor receives a response, T c req is the time when the camera sensor sends an NTP request, T c recv is the time when the camera sensor receives a response, T s recv is the time when the time server receives a request, T ssend Send a response for the time to the time server.

[0109] The specific process of the spatial coordinate alignment is:

[0110]

[0111] Wherein, is the coordinate of the reflection point of the intruder in the camera coordinate system, is the coordinate of the reflection point of the intruder in the radar coordinate system, R is the rotation matrix of the radar coordinate system converted to the camera coordinate system, and T is the translation vector of the radar coordinate system converted to the camera coordinate system.

[0112] S2, data acquisition and pretreatment: adjust the millimeter wave radar sensor and the camera sensor to the best state, detect the intruder on the track at the same time, obtain point cloud and image data respectively, and then pretreat the collected data; the specific process of the pretreatment is as follows:

[0113] The point cloud collected by the millimeter wave radar sensor includes three-dimensional coordinates, intruder speed and intruder reflection intensity (each point includes three-dimensional coordinates, intruder speed and intruder reflection intensity). Since the collected point cloud is very sparse, in order to obtain denser point cloud, according to the actual detection requirement and combining the best detection distance of the millimeter wave radar sensor, the detection distance is set to 50 meters, and the millimeter wave radar sensor can only detect the intruder when the intruder reflection intensity is greater than or equal to 5dB. Remove the data with reflection intensity less than 5dB, save the processed point cloud, and obtain the point set; crop and scale the image data collected by the camera sensor (image data includes two-dimensional coordinates, image texture and appearance shape features) to the size of 640px*640px, and then perform normalization processing to normalize the pixel value from [0, 255] to the interval [0, 1] to meet the input requirements of the YOLOv8 model, and then use the labelImg tool to label the image data in the YOLO label format. The label of the label includes people, animals, falling rocks and mud flow blockage, and the image data set is output.

[0114] S3, data processing: using PCA algorithm and K-means algorithm to process the point set obtained in S2 to detect the intruder; at the same time, using YOLOv8 model to process the image data set obtained in S2 to detect the intruder.

[0115] The processing mode of the point set specifically includes the following steps:

[0116] S3.1, the coordinates of the points in the point set are three-dimensional, but the coordinates of the image data are two-dimensional, and PCA algorithm is used for data dimension reduction:

[0117] First, before performing the PCA algorithm, the points are standardized so that each feature of the points (i.e. three-dimensional coordinates, intruder velocity) has a mean of zero and a standard deviation of 1. This is because the PCA algorithm relies on the covariance between features, and standardization can eliminate the influence between different scales. The formula used for standardization is:

[0118]

[0119] where X is the given data matrix, μ is the mean of each feature, and σ is the standard deviation of each feature;

[0120] Then, the core idea of the PCA algorithm is to find the direction with the largest variance in the data. In order to find the direction with the largest variance in the point set, the covariance matrix Σ of the data matrix X needs to be calculated, and the calculation formula is:

[0121]

[0122] where X scaled is the standardized data matrix.

[0123] Next, the direction with the largest variance in the point set is found by eigenvalue decomposition: the covariance matrix Σ is subjected to eigenvalue decomposition to obtain the eigenvalues and eigenvectors: Σ = λ i v i , where v i is the i-th eigenvector of the covariance matrix Σ, and λ i is the i-th eigenvalue of the covariance matrix Σ.

[0124] The eigenvector represents the direction of data projection, and the eigenvalue represents the variance of the direction. The purpose of the PCA algorithm is to retain the direction with the largest variance, thereby reducing the dimensionality of the data. Usually, the first k eigenvectors with the largest eigenvalues (i.e. principal components) are selected to perform data dimensionality reduction. The first k eigenvectors v1, v2,..., vk k are selected, which form a matrix V k , where each column is an eigenvector.

[0125] Finally, by projecting the point set onto the matrix V k , the reduced point set X pca is obtained, according to the formula:

[0126] X pca = X scaled V k (8)

[0127] S3.2, project the reduced point set X pca obtained in S3.1 to the timestamp-synchronized pixel coordinate system for visualization processing.

[0128] Since the obtained point set X pca after dimension reduction is in two-dimensional coordinate (x ipca ,y ipca ), which belongs to two-dimensional data, the two-dimensional data can be mapped to a pixel coordinate system through an intrinsic matrix L of a camera, and the intrinsic matrix is:

[0129]

[0130] where f x and f y are the focal lengths of the camera sensor along the x and y axes, respectively, and c x and c y are the coordinates of the principal point (usually the center point of the image) of the image.

[0131] Based on the intrinsic matrix L, the point set X pca after dimension reduction can be directly projected onto the pixel coordinate system to obtain a pixel coordinate set {(u1,v1),(u2,v2),...,(u n ,v n )} through the formula , where (u i ,v i ) is the pixel coordinate of the i-th point in the point set after dimension reduction projected on the pixel coordinate system.

[0132] S3.3, the pixel coordinate set obtained in S3.2 is clustered by using a K-means clustering algorithm, and the K-means clustering algorithm is used to extract key features by dividing the data set into several clusters (one cluster is one cluster or set). The specific process is as follows:

[0133] S3.3.1, initialize the cluster center, select m initial cluster centers {c1,c2,...,c m}, c m =(u m ,v m ), where each c m is a point in the pixel coordinate system;

[0134] S3.3.2, calculate the distance from each point (u i ,v i ) to the m initial cluster centers, and assign each point (u i ,v i ) to the initial cluster center with the smallest distance, thereby obtaining m clusters. The distance is the Euclidean distance:

[0135] S3.3.3, update: recalculate the center of each cluster in the m clusters, and the new cluster center is the mean of all points in each cluster: where S m is the set of points assigned to the m-th cluster, |S m | is the number of points in the m-th cluster.

[0136] S3.3.4, iteration: repeat steps S3.3.2, S3.3.3, when the cluster center no longer changes or reaches the maximum number of iterations, the K-means algorithm stops, and a number of clusters are obtained.

[0137] S3.4, after obtaining a number of clusters by S3.3, the position, shape, and speed features of the clusters are extracted for intrusion object detection. The specific extraction method is as follows:

[0138] First, the position feature of the cluster is obtained by calculating the centroid coordinates of the cluster (the centroid coordinates are taken as the position feature), and the calculation formula of the centroid coordinates is: and the width and height of the bounding box of each cluster are calculated: W k = u max - u min , H k = v max - v min ;

[0139] where |S k | is the number of points in the k-th cluster, c k is the centroid coordinates of the k-th cluster, W k and H k are the width and height of the bounding box of the k-th cluster, and the minimum coordinates (u min , v min ) and the maximum coordinates (u max , v max ) of the bounding box are obtained from and .

[0140] Then, the shape feature of the cluster is obtained, and the specific process is: according to the width and height of the bounding box of each cluster, the area A k of the cluster is calculated: k A k = W k × H k , according to the area A k , the aspect ratio AR k and the circularity C k are calculated: According to the values of the two, the shape of the cluster is judged: for a rectangle or an ellipse, the value of the aspect ratio is usually large (greater than 1), and for an object close to a circle, the aspect ratio will be close to 1; the value of the circularity is between [0, 1], the closer the value is to 1, the closer the shape of the object is to a circle, and the smaller the value, the more irregular the shape of the object;is the perimeter of the k-th cluster, which is obtained as follows: the points P = {p1, p2,..., pk} of the cluster are in a two-dimensional plane, and the convex hull of the cluster is obtained using the GrahamScan convex hull algorithm, i.e., the convex boundary points H = {h1, h2,..., hn} of the cluster, and then the Euclidean distances between adjacent points in the convex hull are calculated: n m and the Euclidean distance between the m-th point and the first point: The perimeter P of the boundary can finally be obtained as follows: k

[0141]

[0142] where hi = (xi, yi) is the i-th boundary point. i i i

[0143] S3.5, after extracting the position and shape features of the cluster in S3.4, the speed feature is extracted according to the displacement and time difference method, and the extraction method is as follows: the position features of the intruder in the consecutive time frames t1 and t2, i.e., the centroids c1 and c2, are extracted, and the speed feature is obtained by the following formula:

[0144]

[0145] Finally, the feature extraction of the point cloud collected by the millimeter wave radar is completed, and the extracted features include the speed v r , the position c k and the shape.

[0146] The processing method of the image data set specifically includes the following steps:

[0147] S3.5, obtaining the CIoU loss function model

[0148] S3.5.1, the image data set obtained in S2 is divided into a training set, a validation set and a test set according to a ratio of 7:1:2, which is used to optimize the improved CIoU loss function; wherein the training set is used for training the improved CIoU loss function to obtain the optimized CIoU loss function; the validation set is used to verify the effect of the optimized CIoU loss function offline to facilitate parameter adjustment; the test set is used to test the generalization ability of the CIoU loss function model obtained after verification is completed, to determine whether it maintains the same performance on other data.

[0149] S3.5.2, construction of the improved CIoU loss function, the specific process is as follows:

[0150] The CIoU calculation formula is: where b = (x​​​​​​b ,y b ,w b ,h b ) is the center coordinate (x b ,y b ), width w b and height h b of the ground truth bounding box, is the center coordinate width and height of the predicted bounding box, is the intersection over union of the ground truth bounding box and the predicted bounding box, c is the diagonal length of the minimum enclosing box: where x max , y max and x min , y min are the diagonal coordinates of the minimum rectangle containing the ground truth bounding box and the predicted bounding box, v is the aspect ratio consistency measure: α is an adjustment factor, 0 < α < 1, used to balance the various parts in the loss, usually set as a hyperparameter.

[0151] Since the original CIoU relies too much on the distance between the center points, the weight of the distance between the center points is too large, especially when the scale difference of the object is large, which may lead to inaccurate optimization of the frame of small objects and imperfect aspect ratio adjustment. Therefore, a stronger scale adaptive mechanism is introduced, and a scale adaptive loss function is designed The weight of each indicator (such as the distance between the center points and the aspect ratio adjustment) is adjusted under different scales, and the aspect ratio loss is improved. The introduction of Chamfer distance to replace the original aspect ratio loss can capture the shape difference of the bounding box more finely and improve the detection accuracy. The calculation formula of the scale adaptive loss function is as follows:

[0152]

[0153]

[0154] where A b = w b × h b is the area of the ground truth bounding box b, A avg is the average area of all ground truth bounding boxes, λ is a tunable hyperparameter used to control the sensitivity of the scale difference; P = {(x1, y1), (x2, y2), (x3, y3), (x4, y4)} and are the four vertex sets of the ground truth bounding box and the predicted bounding box, respectively.

[0155] According to formulas (15), (16) and (17), the improved CIoU loss function is:

[0156]

[0157] S3.5.3, optimizing the improved CIoU loss function

[0158] The training set, the validation set and the test set are used to train, validate and test the improved CIoU loss function respectively, so as to optimize the improved CIoU loss function until the evaluation index meets the accuracy requirement. The evaluation standard is that the average precision mean (mAP) is improved by more than 0.02, the recall rate is improved by more than 0.03, the precision is improved by more than 0.02, and the positioning error is reduced. If the evaluation standard is not met, repeat steps S3.5.1 and S3.5.2 to adjust and update the parameters until the error meets the requirement or the maximum iteration number is reached. After optimization, the CIoU loss function model is obtained, and the optimized parameters in the CIoU loss function model are saved for subsequent intrusion detection.

[0159] S3.6, the training set, the validation set and the test set described in S3.5.1 are used to train, validate and test the YOLOv8 algorithm (including MobileNetV3 module and YOLO detection head module) respectively, so as to optimize the YOLOv8 algorithm until the evaluation index meets the requirement. The evaluation standard is that the average precision mean (mAP) is improved by more than 0.02, the recall rate is improved by more than 0.03, the precision is improved by more than 0.02, and the positioning error is reduced. If the evaluation standard is not met, adjust the scale adaptive loss function λ and continue training until the evaluation standard is met. The optimized YOLOv8 algorithm has faster training speed and better training result, so the optimized YOLOv8 algorithm is named as improved YOLOv8 model. The optimized parameters obtained in S3.5.3 are loaded into the improved YOLOv8 model. Then, the image data normalized in S2 is input into the MobileNetV3 module of the improved YOLOv8 model, and the features from low level to high level are extracted from the image data normalized in S2, including two-dimensional coordinates s i =(x i ,y i ), speed v c , and shape features, so as to convert the image data into multi-level feature maps, and save the feature maps for subsequent intrusion detection. The speed can be obtained according to two time frames t1 and t2 and combined with the two-dimensional coordinates s1 and s2 of the intrusion, that is:

[0160]

[0161] S3.7, passing the feature map obtained in S3.6 through the YOLO detection head module included in the improved YOLOv8 model, thereby generating a plurality of prediction boxes in the feature map obtained in S3.6.

[0162] S3.8, processing the feature map containing a plurality of prediction boxes obtained in S3.7 using a non-maximum suppression (NMS) algorithm, removing redundant boxes among the plurality of prediction boxes contained in the feature map, and retaining the optimal box, thereby obtaining an intruder feature map.

[0163] S4: Data fusion: the features obtained in S3.4 include velocity v r , position c k , and shape, and the intruder feature map obtained in S3.6 includes two-dimensional coordinates s i =(x i ,y i ), velocity v c , and shape features, and the velocity v r , position c k obtained in S3.4 and the two-dimensional coordinates s i =(x i ,y i ), velocity v c obtained in S3.6 are fused by weighted averaging. The fusion method is as follows:

[0164] First, different weights are assigned to the features of the point cloud obtained by scanning the millimeter wave radar sensor and the image data obtained by scanning the camera sensor, respectively and Then, the features corresponding to each intruder i, including radar features {F Radar,1 ,F Radar,2 ,...} and camera features {F Camera,1 ,F Camera,2 ,...}, are fused by weighted averaging to obtain the fused features

[0165]

[0166] where F Radar =(c k ,v r ), F Camera =(s i ,v c ), and F fusion{(x, y, z), Speed: m / s}, z is obtained by processing the three-dimensional coordinates obtained by millimeter wave radar detection.

[0167] The shape obtained in S3.4 and the shape feature obtained in S3.6 are fused by cosine similarity judgment and then by weighted average:

[0168]

[0169] wherein f radar and f camera are the shape extracted in S3.4 and the shape feature extracted in S3.8, respectively, and the extracted shape and shape feature are vector data. ||f radar || and ||f camera || represent the length of the vector, also called norm, and the calculation formula is: wherein a i is each component of the respective vector.

[0170] wherein the cosine similarity value range is: [-1, 1], 1 represents complete similarity, -1 represents complete dissimilarity, and 0 represents no relation between the two, therefore, when sim(f radar ,f camera )>0.5, weighted average fusion is performed; the fusion formula is:

[0171]

[0172] S5, detecting intruders: the fused features obtained in S4 are input into the YOLOv8 model in S3.6 again, and intruder detection is performed again to obtain the intruder detection result. The specific process is:

[0173] First, the fused features obtained in S4 and f fused are input into the improved YOLOv8 model in S3.6 again, wherein the MobileNetV3 module included in the improved YOLOv8 model extracts features from low to high from the input and f fused , and the fused features and f fusedThe feature map is converted into a multi-level feature map. Then, the feature map is input into a YOLO detection head module included in the improved YOLOv8 model to generate a prediction box. After processing by a non-maximum suppression (NMS) algorithm, redundant boxes in the prediction box are removed, and the optimal box in the prediction box is retained, which is a detection box. The detection result is displayed in the detection box, and a confidence is obtained. The detection result is an intrusion object detection image containing all features, including the fused position (x, y, z), speed Speed:_m / s, shape extracted from the point cloud collected by the millimeter wave radar sensor, confidence obtained from the fused information, and other information. Finally, the detection result is output and saved.

[0174] Figure 3 The improved YOLOv8 model is shown in the schematic diagram. In addition to the improved CIoU loss function, the MobileNetV3 module and the YOLO detection head module are also included. The MobileNetV3 module uses deep separable convolution to reduce the amount of calculation and the number of parameters, and combines the reverse residual structure to improve the feature extraction capability and information flow. The introduced SE module (Squeeze-and-Excitation) enhances the network performance through channel attention mechanism, and the Hard Swish activation function replaces the traditional ReLU activation function to improve the calculation efficiency and accuracy. The linear bottleneck structure avoids information loss and enhances the low-dimensional feature expression capability. Through neural architecture search (NAS), MobileNetV3 combines the EfficientNet idea to automatically design an efficient network structure, and finally classifies the target through a simple classifier layer to further improve the efficiency and accuracy. The YOLO detection head module uses an upsampling layer to improve the resolution of the feature map and enhance the detail information. Then, the feature map is spliced and fused from different scales to combine the information from low and high layers to enrich the target representation. Through convolution operation, the feature map is further optimized to extract more distinctive features. The attention mechanism, such as channel attention, is introduced to enhance the representation ability of important features. The model generates the coordinates and class labels of the prediction box through boundary box regression and classification, and finally uses non-maximum suppression (NMS) to remove redundant boxes and optimize the final detection result.

[0175] For the millimeter wave radar sensor, the first collected data is a 3D point cloud Figure 4 (a). After data standardization and PCA algorithm dimensionality reduction, a 2D point cloud of Figure 4 (b) can be obtained. Then, it is projected into the pixel coordinate system to obtain the point cloud graph of Figure 4 (c). Then, the K-means algorithm is used to cluster the point cloud. Since there is only one person as an intrusion object in the graph, the point cloud of Figure 4(d) the midpoint cloud map, only one point cloud cluster. Namely, the main features can be extracted through the main feature extraction, and the fusion is prepared.

[0176] Figure 5 The improved YOLOv8 model is used for feature extraction of the image at the same time, and the multi-level feature map is obtained, which includes the detection frame, the two-dimensional coordinates and the confidence. Figure 6 The final detection map extracted by the improved YOLOv8 model after feature fusion is shown in the figure, which includes position (x, y, z), speed Speed:_m / s, shape extracted based on the point cloud collected by the millimeter wave radar sensor, and confidence extracted based on the image data collected by the camera sensor: as shown in the figure, x:2.76, y:9.13, z:3.38, speed:1.08 m / s and confidence person:0.9 real-time data, wherein z is obtained by processing the three-dimensional coordinates obtained by millimeter wave radar detection. Figure 6

[0177] The present application detects the intruder through sensor synchronization, data acquisition, data processing, data fusion and detection of intruder. The sensor synchronization mainly synchronizes the time stamp of the millimeter wave radar sensor and the camera sensor, aligns the space coordinates, and ensures that the two sensors describe the same target; the data acquisition is the data acquisition of the millimeter wave radar sensor and the camera sensor at the same time, the millimeter wave radar sensor collects point cloud data, the camera sensor collects image data, and the collected data is made into a data set through preprocessing. The data processing is to process the point cloud data set collected by the millimeter wave radar sensor through PAC algorithm and K-means algorithm, and obtain the main features. The improved CIoU loss function is used to train the YOLOv8 model, and the optimized network weight after training is obtained, and then the image data set collected by the camera sensor is processed according to the network weight, and the multi-level feature map is obtained. The data fusion is to fuse the features obtained by the two sensors to obtain the fused features. Finally, the obtained fused features are put into the improved YOLOv8 model which has been trained to detect again, and the final output result is obtained. The present application detects the intruder by fusing the features of the two sensors, which combines the advantages of the two sensors, can detect the intruder in complex environment, improves the redundancy and reliability, and enhances the target recognition and positioning accuracy.

[0178] ​The foregoing detailed description of the application has been presented for purposes of illustration and description. Various modifications and changes can be made to these embodiments without departing from the spirit and scope of the application. It is intended that the scope of the application should not be limited by the particular representative embodiments described above.

Claims

1. A method for detecting intruders on rails using a heterogeneous fusion of a millimeter-wave radar sensor and a camera sensor, comprising the following steps: S1. Sensor synchronization: Synchronize the timestamps and align the spatial coordinates of the millimeter-wave radar sensor and the camera sensor to ensure that the data obtained by the two sensors correspond to the same target. First, use the NTP network time protocol to synchronize the timestamps. Then, spatial coordinate alignment is performed, which converts the point cloud obtained by the millimeter-wave radar sensor from the radar coordinate system to the camera coordinate system, thereby unifying the point cloud and image data in the camera coordinate system; S2. Data Collection and Preprocessing: The millimeter-wave radar sensor and camera sensor are adjusted to the optimal state, and intruders on the rails are detected simultaneously to obtain point cloud and image data, respectively. The collected data is then preprocessed. The specific process of the preprocessing is as follows: The point cloud collected by the millimeter-wave radar sensor includes three-dimensional coordinates, the speed of the intruder, and the reflection intensity of the intruder. Data with reflection intensity less than 5dB are removed, and the processed point cloud is saved to obtain a point set. The image data collected by the camera sensor was cropped and scaled to 640px×640px, and then normalized from [0, 255] to [0, 1] to meet the input requirements of the YOLOv8 model. The image data was then annotated with the labelImg tool in the YOLO label format, with labels including people, animals, fallen rocks, and mudflow blockages, to obtain the output image dataset. S3, data processing: PCA algorithm and K-means algorithm are used to process the point set obtained in S2, and the features obtained include speed v r 、Position c k and shape to detect intruders; at the same time, the image dataset obtained by S2 is processed using the YOLOv8 model to obtain the two-dimensional coordinates s i =(x i ,y i ), speed v c and shape features to detect intruders; S4, data fusion: the speed v obtained in S3 r 、Position c k and the two-dimensional coordinates s i =(x i ,y i ), speed v c The shapes and shape features obtained in S3 are fused by weighted average after being judged by cosine similarity and then fused by weighted average; S5. Detect intruders: The fused features obtained in S4 are input into the YOLOv8 model in S3 again, and intruder detection is performed again to obtain the intruder detection results.

2. The method for detecting intruders on rails according to claim 1, characterized in that: In step S1, the specific process of timestamp synchronization is as follows: The clock deviation of the millimeter wave radar sensor is: The clock bias of the camera sensor is: The timestamp of the synchronized millimeter-wave radar sensor is: T rsync =T rrecv +θ r (3) The synchronized camera sensor timestamp is: T csync =T crecv +θ c (4) Among them, T rreq The time when the millimeter wave radar sensor sends the NTP request, T rrecv is the time it takes for the millimeter-wave radar sensor to receive a response, T creq The time when the camera sensor sends the NTP request, T crecv is the time when the camera sensor receives the response, T srecv The time when the time server receives the request, T ssend The time when the time server sends the response; The specific process of the spatial coordinate alignment is: in, is the coordinate of the reflection point of the intruder in the camera coordinate system, is the coordinate of the reflection point of the intruder in the radar coordinate system, R is the rotation matrix converted from the radar coordinate system to the camera coordinate system, and T is the translation vector converted from the radar coordinate system to the camera coordinate system.

3. The method for detecting intruders on rails according to claim 1, wherein: In step S3, the point set processing method specifically includes the following steps: S3.

1. The coordinates of the points in the point set are three-dimensional, but the coordinates of the image data are two-dimensional. The PCA algorithm is used to reduce the dimensionality of the data: First, before executing the PCA algorithm, the points are standardized so that each feature of the point has a mean of zero and a standard deviation of 1; the formula used for standardization is: Where X is the given data matrix, μ is the mean of each feature, and σ is the standard deviation of each feature; Then, in order to find the direction with the largest variance in the point set, the covariance matrix Σ of the data matrix is ​​calculated as follows: Among them, X scaled is the normalized data matrix; Next, we use eigenvalue decomposition to find the direction with the largest variance in the point concentration: perform eigenvalue decomposition on the covariance matrix Σ to obtain the eigenvalues ​​and eigenvectors: Σ = λ i v i , where v i is the i-th eigenvector of the covariance matrix Σ, λ i is the i-th eigenvalue of the covariance matrix Σ; The eigenvector represents the direction of data projection, and the eigenvalue represents the variance in that direction; the eigenvectors with the largest eigenvalues ​​are selected for data dimensionality reduction; the eigenvectors with the largest eigenvalues ​​v1, v2, ..., v k , forming a matrix V k , where each column is a feature vector; Finally, by projecting the point set onto the matrix V k , get the point set X after dimensionality reduction pca , according to the formula: X pca =X scaled V k (8) S3.2, the dimensionality-reduced point set X obtained in S3.1 pca Projecting to the pixel coordinate system synchronized with the timestamp for visualization; Point set X after dimensionality reduction pca The midpoint is the two-dimensional coordinate (x ipca ,y ipca ), which is two-dimensional data. The two-dimensional data is mapped to the pixel coordinate system through the camera intrinsic parameter matrix L. The intrinsic parameter matrix is: Among them, f x and f y are the focal lengths of the camera sensor along the x and y axes, respectively, c x and c y are the principal point coordinates of the image; Based on the internal parameter matrix L, through the formula Directly reduce the dimension of the point set X pca Projected onto the pixel coordinate system to obtain the pixel coordinate set {(u1,v1),(u2,v2),...,(u n ,v n )}, where (u i ,v i ) is the pixel coordinate of the i-th point in the point set after dimensionality reduction projected in the pixel coordinate system; S3.

3. Cluster the pixel coordinate set obtained in S3.2 using the K-means clustering algorithm. The K-means clustering algorithm extracts key features by dividing the data set into several clusters. The specific process is as follows: S3.3.

1. Initialize the cluster centers and select m initial cluster centers {c1, c2, ..., c m }, c m =(u m ,v m ), where each c m It is a point in the pixel coordinate system; S3.3.2, calculate each point (u i ,v i ) to the distance of m initial cluster centers, and each point (u i ,v i ) is assigned to the initial cluster center with the smallest distance, thereby obtaining m clusters; the distance is the Euclidean distance: S3.3.

3. Update: Recalculate the center of each cluster in the m clusters, and the new cluster center is Among them, S m is the set of points assigned to the mth cluster, |S m | is the number of points in the mth cluster; S3.3.4, Iteration: Repeat steps S3.3.2 and S3.3.

3. When the cluster center no longer changes or the maximum number of iterations is reached, the K-means algorithm stops and several clusters are obtained; After S3.4 and S3.3 obtain several clusters, extract the cluster features including position, shape, and speed. The specific extraction method is as follows: First, the location features of the cluster are obtained by calculating the centroid coordinates of the cluster. The calculation formula of the centroid coordinates is: And, calculate the width and height of the bounding box of each cluster: W k =u max -u min ,H k =v max -v min ; Among them, |S k | is the number of points in the kth cluster, c k is the centroid coordinate of the kth cluster, W k and H k are the width and height of the bounding box of the kth cluster, and the minimum coordinate of the bounding box (u min ,v min ) and the maximum coordinate (u max ,v max )Depend on to obtain; Then, the shape features of the cluster are obtained. The specific process is as follows: Based on the width and height of the bounding box of each cluster, the area A of the cluster is calculated. k :A k =W k ×H k , according to the area A k , calculate the aspect ratio AR k and circularity C k : The shape of the cluster is determined by the values ​​of the two: for rectangles or ellipses, the value of the aspect ratio is usually larger, while for objects close to circles, the aspect ratio is close to 1; the circularity value is between [0, 1], the closer the value is to 1, the closer the shape of the object is to a circle, and the smaller the value is, the more irregular the shape of the object is; k is the boundary perimeter of the kth cluster, which is obtained as follows: the cluster point P = {p1, p2, ..., p n } are all on a two-dimensional plane, and the Graham Scan convex hull algorithm is used to obtain the cluster to obtain the convex hull of the cluster, that is, the convex boundary points of the cluster H = {h1,h2,...,h m }, and then calculate the Euclidean distance between adjacent points in the convex hull: And the Euclidean distance between the mth point and the first point: Finally, the boundary perimeter P can be obtained by the following formula k : where h i =(x i ,y i ) is the i-th boundary point; S3.

5. After the position and shape features of the clusters are extracted in S3.4, velocity features are extracted using the displacement and time difference method. The extraction method is as follows: the position features of the intruder, i.e., the centroids c1 and c2, are extracted at consecutive time frames t1 and t2. The velocity features are obtained using the following formula: Finally, the feature extraction of the point cloud collected by the millimeter wave radar is completed. The extracted features include speed v r 、Position c k And shape.

4. The method for detecting intruders on rails according to claim 1, wherein: In step S3, the image data set is processed in the following steps: S3.

5. Obtain CIoU loss function model S3.5.

1. Divide the image dataset obtained in S2 into a training set, a validation set, and a test set in a ratio of 7:1:2 to optimize the improved CIoU loss function. S3.5.

2. Construction of the improved CIoU loss function. The specific process is as follows: Where b=(x b ,y b ,w b ,h b ) is the center coordinate of the ground-truth bounding box (x b ,y b ), width w b and height h b , is the center coordinate of the predicted bounding box width and height is the intersection-over-union ratio of the true bounding box and the predicted bounding box, is the Euclidean distance between the center points: c is the diagonal length of the minimum enclosing box: Among them, x max 、y max and x min 、y min are the diagonal coordinates of the smallest rectangle that contains the true bounding box and the predicted bounding box, and v is the aspect ratio consistency measure: α is an adjustment factor, 0<α<1, used to balance the various parts of the loss and set as a hyperparameter; Introducing a scale-adaptive mechanism, a scale-adaptive loss function Adjust the weights of various indicators at different scales and improve the aspect ratio loss. Introduce Chamfer distance to replace the original aspect ratio loss to capture the shape differences of the bounding box; scale-adaptive loss function The calculation formula is: Among them, A b =w b ×h b is the area of ​​the ground-truth bounding box b, A avg is the average area of ​​all ground-truth bounding boxes, λ is an adjustable hyperparameter that controls the sensitivity to scale differences; P = {(x1,y1),(x2,y2),(x3,y3),(x4,y4)} and are the four vertex sets of the true bounding box and the predicted bounding box respectively; According to equations (15), (16) and (17), the improved CIoU loss function is: S3.5.

3. Optimized and improved CIoU loss function The improved CIoU loss function is trained, verified, and tested using the training set, validation set, and test set, respectively, to optimize the improved CIoU loss function until the evaluation indicators meet the accuracy requirements. The evaluation criteria are: the average precision is improved by more than 0.02, the recall rate is improved by more than 0.03, the precision is improved by more than 0.02, and the positioning error is reduced. If the evaluation criteria are not met, steps S3.5.1 and S3.5.2 are repeated to adjust and update the parameters until the error meets the requirements or the maximum number of iterations is reached. After the optimization is completed, the CIoU loss function model is obtained, and the optimized parameters in the CIoU loss function model are saved for subsequent intruder detection. S3.

6. Use the training set, validation set, and test set described in S3.5.1 to train, validate, and test the YOLOv8 algorithm, which includes the MobileNetV3 module and the YOLO detection head module, to optimize the YOLOv8 algorithm until the evaluation indicators meet the requirements. The evaluation criteria are: the average precision is improved by more than 0.02, the recall rate is improved by more than 0.03, the precision is improved by more than 0.02, and the positioning error is reduced. If the evaluation criteria are not met, adjust the scale-adaptive loss function. Continue training with λ in until the evaluation criteria are met; name the optimized YOLOv8 algorithm as the improved YOLOv8 model, and then load the optimized parameters obtained in S3.5.3 into the improved YOLOv8 model; then, input the image data normalized in S2 into the MobileNetV3 module of the improved YOLOv8 model, and extract features from low-level to high-level from the image data normalized in S2, including the two-dimensional coordinates s i =(x i ,y i ), speed v c , and shape features, thereby converting the image data into a multi-level feature map, and saving the feature map for subsequent intruder detection; wherein the speed is obtained based on the two time frames t1 and t2 and the two-dimensional coordinates s1 and s2 of the intruder, that is: S3.

7. Pass the feature map obtained in S3.6 through the YOLO detection head module included in the improved YOLOv8 model, thereby generating multiple prediction boxes in the feature map obtained in S3.6; S3.

8. Use a non-maximum suppression algorithm to process the feature map containing multiple prediction frames obtained in S3.7, remove redundant frames from the multiple prediction frames contained in the feature map, and retain the optimal frame, thereby obtaining an intruder feature map.

5. The method for detecting intruders on rails according to claim 1, wherein: In step S4, the speed v obtained in S3 is r 、Position c k and the two-dimensional coordinates s i =(x i ,y i ), speed v c The fusion method of weighted averaging is as follows: First, different weights are assigned to the features of the point cloud scanned by the millimeter-wave radar sensor and the image data scanned by the camera sensor. Weight Satisfaction and Then, the features corresponding to each intrusion object i, including radar features {F Radar,1 ,F Radar,2 ,...} and camera features {F Camera,1 ,F Camera,2 ,...}, perform weighted average fusion to obtain the fused features where, F Radar = (c k , v r ), F Camera = (s i , v c ), F fusion = {(x, y, z), Speed: _m / s}.

6. The method for detecting intruders on rails according to claim 1, wherein: In step S4, the shapes and shape features obtained in step S3 are judged by cosine similarity and then fused by weighted average. The specific steps are as follows: Among them, f radar and f camera are the shape and shape features extracted by S3, and the extracted shape and shape features are both vector data; || f radar || and ||f camera || represents the modulus of the vector, and the calculation formula is: Among them, a i are the components of their respective vectors; Among them, the cosine similarity value range is: [-1,1], 1 means complete similarity, -1 means complete dissimilarity, and 0 means that the two are irrelevant. Therefore, when sim(f radar ,f camera )>0.5, weighted average fusion is performed; the fusion formula is:

7. The method for detecting intruders on rails according to claim 1, wherein: The specific process of S5 is as follows: First, the fused features obtained by S4 and f fused Input again into the improved YOLOv8 model in S3, the improved YOLOv8 model includes the MobileNetV3 module from the input and f fused Extract features from low-level to high-level, and combine the fused features and f fused The method converts the feature map into a multi-level feature map; then, the feature map is input into the YOLO detection head module included in the improved YOLOv8 model to generate a prediction box; the feature map is then processed by the non-maximum suppression algorithm to remove redundant boxes in the prediction box and retain the optimal box in the prediction box, which is the detection box. The detection result is displayed in the detection box, and the confidence score is obtained. The detection result is an intruder detection map containing all features; finally, the detection result is output and saved.

Images