A train forward obstacle detection method based on 4D millimeter wave radar

Abstract

The present invention relates to a train forward obstacle detection method based on 4D millimeter wave radar, and relates to the technical field of intelligent train assisted driving. Firstly, the collected 4D millimeter wave radar point cloud data is preprocessed by using statistical filtering and through filtering to remove the interference of outliers, noise points and high-altitude obstacles. Then, the point cloud segmentation range on the left and right sides of the train track is adjusted based on the DBSCAN clustering algorithm, which reduces the model calculation consumption and retains the point cloud information of the obstacle as much as possible. Finally, the obstacle target detection is performed through multi-scale and multi-dimensional feature extraction and a deep learning network model with enhanced attention in spatial dimension and channel dimension, so that the computing resources are saved while the detection accuracy is guaranteed, and the intelligent assisted driving of the train is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent train assisted driving, and in particular to a method for detecting forward obstacles of a train based on a 4D millimeter wave radar. Background Art

[0002] With the expansion of urban scale and the rapid development of science and technology, rail transit has become one of the indispensable public transportation infrastructures in cities. Realizing intelligent assisted driving in the field of rail transit can reduce accidents caused by human errors, monitor the surrounding environment in real time, make faster and more accurate responses, avoid collisions and reduce traffic accidents. Environmental perception can be regarded as the first threshold of intelligent assisted driving. Accurate perception of the surrounding environment and obtaining the precise location and category information of the target object are one of the keys to realizing intelligent assisted driving of trains.

[0003] At present, the main sensors used for environmental perception are cameras, laser radars, and millimeter-wave radars. As a common environmental perception sensor, cameras can obtain high-resolution environmental information, which is conducive to obstacle detection. However, the image information obtained by the camera only provides semantic features of the 2D plane, and it is difficult to obtain accurate depth information of the target. In addition, when the light is insufficient and there is bad weather such as rain, snow, and fog, false detection and missed detection are inevitable. The point cloud information output by the laser radar has a high resolution and contains the 3D semantic features of the target. However, the high price of the laser radar and the rapid decline in its detection accuracy in bad weather such as rain, snow, and fog limit the scope of application of the laser radar. The working principle of the traditional 3D millimeter-wave radar is to use radio waves to detect targets. Since the wavelength of radio waves is relatively long and the size of raindrops, snowflakes, and fog is small, the millimeter-wave radar can have good robustness in the face of extreme weather. However, the traditional 3D millimeter-wave radar can only measure the radial distance, azimuth, and radial velocity of the object, lacking important height information, and the low resolution leads to sparse output point clouds, which cannot be used as the main sensor for single-mode target detection. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the deficiencies in the prior art and provide a method for detecting obstacles ahead of the train based on a 4D millimeter-wave radar, which preprocesses point cloud data according to the driving environment of the train and improves the deep learning network model to achieve target detection of obstacles ahead of the train.

[0005] The present invention is achieved through the following technical solutions:

[0006] A method for detecting obstacles ahead of a train based on a 4D millimeter wave radar comprises the following steps:

[0007] S1. Collect the original point cloud data of 4D millimeter wave radar and obtain obstacle point cloud data after preprocessing;

[0008] S2. Clustering the obstacle point cloud data to determine the detection range on both sides of the current frame point cloud track;

[0009] S3. Divide the point cloud after clustering into columns, randomly select multiple points for each column for feature extraction, perform maximum pooling on the point cloud dimension of the column, find the most representative feature of the current column, and then restore the most representative feature of the current column into a pseudo image according to the area represented by the column;

[0010] S4. Input the pseudo image into the 2D convolution backbone network to perform multi-dimensional and multi-scale feature extraction to obtain a feature map of the 2D convolution extracted features;

[0011] S5. Strengthen the channel dimension and spatial dimension attention of the feature map of the 2D convolution feature extraction;

[0012] S6. Input the feature map after channel dimension and spatial dimension attention enhancement into the SSD detection head to perform obstacle target detection.

[0013] According to the above technical solution, preferably, step S1 includes: collecting original point cloud data of 4D millimeter wave radar; statistically filtering the original point cloud data to remove noise points; using pass-through filtering to filter out point clouds that are 1.5 times higher than the train height; and obtaining obstacle point cloud data after preprocessing.

[0014] According to the above technical solution, preferably, in step S2, a DBSCAN clustering algorithm is used to perform clustering processing on the obstacle point cloud data.

[0015] According to the above technical solution, preferably, in step S3, 32 points are randomly selected from each column for feature extraction; if the number of points in the column is less than 32, the average value of all points in the current column is used for filling.

[0016] According to the above technical solution, preferably, in step S3, 11-dimensional features (x, y, z, r, v, x) are extracted from each point. c ,y c ,z c ,x p ,y p ,z p ), where x, y, z represent the position information of each point in the Cartesian coordinate system, r represents the reflection intensity of the object, v is the Doppler velocity of the object, and x c ,y c ,z c The difference between the point and the average value of all points in the current column, xp ,y p , z p is the relative position of the point and the geometric center of the column.

[0017] According to the above technical solution, preferably, in step S3, the most representative feature of the current column is restored into a (C, H, W) pseudo-image according to the area represented by the column, wherein C represents the feature dimension of each column, and H and W represent the resolution of the pseudo-image in the X-axis direction and the Y-axis direction, respectively.

[0018] According to the above technical solution, preferably, step S4 includes: downsampling the input pseudo image three times to obtain feature maps of three scales, which respectively represent the features of different dimensions of the current frame point cloud; passing the feature map after the third downsampling in parallel through two hole convolutions with different expansion factors, and splicing the two outputs after the hole convolution with the feature map after the third downsampling as a high-dimensional global feature; upsampling the outputs of multiple dimensions to the same size through deconvolution and splicing them to obtain the feature map of the 2D convolution extraction feature.

[0019] According to the above technical solution, preferably, in step S5, the feature map of the 2D convolution extraction feature is subjected to channel dimension attention enhancement, including: the input feature map is subjected to maximum pooling in the spatial dimension and average pooling in the spatial dimension, respectively, to obtain two sizes of (C in , 1, 1) feature block F channel_max 、F channel_avg ; The two feature blocks are extracted by multi-layer perceptron and activation function; the results of feature extraction are added element by element and then passed through a sigmoid activation function to obtain the channel dimension attention weight F channel_output , F channel_output With F in Multiply them to strengthen the information of channel dimension; among them,

[0020] F channel_output =sigmoid(MLP(AvgPool(F in ))+MLP(MaxPool(F in )))

[0021] F′ in =F channel_output *F in .

[0022] According to the above technical solution, preferably, in step S5, the feature map of the 2D convolution extraction feature is subjected to spatial dimension attention enhancement, including: the input feature map is subjected to channel dimension maximum pooling and channel dimension average pooling respectively, and two sizes of (1, H in, W in )’s feature map F spatial_max 、F spatial_avg ; After the two feature maps are concatenated, they are converted into (1, H in , W in ) feature map; the feature map is passed through a sigmoid activation function to obtain the spatial dimension attention weight F spatial_output , F spatial_output and F′ in Multiply to strengthen the information of spatial dimension; among them,

[0023] F spatial_output =conv([AvgPool(F′ in );MaxPool(F′ in )])

[0024] F output =F spatial_output *F in .

[0025] The beneficial effects of the present invention are:

[0026] The present invention adopts 4D millimeter-wave radar as the main sensor, which can not only ensure the accuracy of detection in bad weather, but also make up for the problems of lack of height information and low resolution of traditional 3D millimeter-wave radar. In the train running environment, the detection range is dynamically set according to the clustering results of the current frame point cloud, and the deep learning network model enhanced by the attention mechanism is used to perform target detection of obstacles, so as to save computing resources while ensuring the detection accuracy and realize intelligent assisted driving of the train. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the process of the present invention.

[0028] Figure 2 It is a schematic diagram of the train detection range based on the DBSCAN clustering algorithm of the present invention.

[0029] Figure 3 It is a schematic diagram of the structure of the three-dimensional target detection model of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and the best embodiment. Based on the embodiments in the invention, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the invention.

[0031] Combination Figure 1The method of the present invention mainly comprises the following steps:

[0032] S1. Collect the original point cloud data of 4D millimeter wave radar and obtain the obstacle point cloud data after preprocessing.

[0033] Specifically, taking into account the characteristics of 4D millimeter-wave radar point clouds that are sparse at a distance and dense near, statistical filtering is used to pre-process the output point cloud to eliminate noise points. Pass-through filtering is used to filter out point clouds that are higher than 1.5 times the height of the train to avoid interference from other high-altitude obstacles such as viaducts, and to give full play to the role of the 4D millimeter-wave radar output pitch angle information.

[0034] S2. Clustering the obstacle point cloud data to determine the detection range on the left and right sides of the current frame point cloud track.

[0035] Since the train's running track is a fixed track line, the obstacle detection range can be limited to a certain area on the left and right sides of the track. However, in order to avoid the point cloud of some obstacles being cut due to setting a rigid detection area, the DBSCAN clustering algorithm is used in this example to cluster the obstacle point cloud data, and then the detection range on the left and right sides of the track is determined.

[0036] Combination Figure 2 The train detection range method based on DBSCAN clustering algorithm of the present invention mainly includes the following steps:

[0037] S21: First set the minimum detection range Y on the left and right sides of the track left_min , Y right_min During initialization, the left and right segmentation boundaries of the point cloud are initialized to the corresponding minimum detection ranges on both sides:

[0038] Y left =Y left_min

[0039] Y right =Y right_min

[0040] S22: Use the DBSCAN clustering algorithm to cluster the point cloud of the current frame and obtain the corresponding clustered 3D box, which can roughly represent the position estimate of the obstacle in the current frame.

[0041] S23: traverse the 3D box corresponding to each cluster after the current frame is clustered, and determine whether the currently traversed 3D box is within the range of (Y left_min , Y right_min )Inside.

[0042] S24: If the currently viewed 3Dbox frame is in (Y left_min , Yright_min ), the point cloud division range is modified as follows according to the Y-axis range of the current 3Dbox box, so that all points contained in the 3Dbox box are retained.

[0043] Y left =max(Y left , Y box_left )

[0044] Y right =min(Y right , Y box_right )

[0045] S25: If the currently viewed 3Dbox is not in (Y left_min , Y right_min ), then go to step S23 to continue traversing the clustered 3Dbox frames until all 3Dbox frames have been traversed.

[0046] S3. Divide the point cloud after clustering processing into columns, randomly select multiple points for each column for feature extraction, perform maximum pooling on the information of each dimension in the column point cloud, find the most representative feature of the current column, and then restore the most representative feature of the current column into a pseudo image according to the area represented by the column.

[0047] S4. Input the pseudo image into a 2D convolution backbone network to perform multi-dimensional and multi-scale feature extraction to obtain a feature map of the 2D convolution extraction features.

[0048] S5. Strengthen the attention of the channel dimension and spatial dimension of the feature map of the 2D convolutional feature extraction.

[0049] Among them, combined Figure 3 , the target detection network model of the present invention, the data processing process in steps S3-S5 mainly includes the following steps:

[0050] S31: Columnize and extract features from the preprocessed point cloud. Columnize the point cloud in the XY plane with a step size of [0.16m, 0.16m]. Randomly select 32 points in each column for feature extraction. If the number of points in the column is less than 32, fill it with the average value of all points in the current column. Extract 11-dimensional features (x, y, z, r, v, x) from each point. c ,y c , z c , x p ,y p , z p ); where: x, y, z represent the position information of each point in the Cartesian coordinate system, r represents the reflection intensity of the object, v is the Doppler velocity of the object, x c ,yc , z c The difference between the point and the average value of all points in the current column, that is:

[0051]

[0052]

[0053]

[0054] x p ,y p , z p is the relative position of the point and the geometric center of the column, that is:

[0055] x p =x i -x pillar_center

[0056] y p =y i -y pillar_center

[0057] z p =z i -z pillar_center

[0058] The features of the points selected in the column are extracted through multiple layers of fully connected layers, and then the maximum pooling is used for each dimension in the column, that is, the feature that best represents the current column is found. Finally, the feature that best represents the current column is restored to a pseudo image of (C, H, W) according to the area represented by the column, where: C represents the feature dimension representing the current column after the fully connected layer and the maximum pooling, H and W represent the number of columns after the range in the X-axis direction and the Y-axis direction is divided by the column step length, that is:

[0059] c=F max (Linear(C in ))

[0060] H=X frame / 0.16

[0061] W=Y frame / 0.16

[0062] S32: After feature extraction of each column, a pseudo image (C, H, W) is obtained, which is input into the 2D convolution backbone network to perform multi-dimensional and multi-scale feature extraction on the input pseudo image. The input pseudo image is downsampled three times to obtain features of three scales, which represent the features of different dimensions of the current point cloud frame. Since the resolution of the 4D millimeter-wave radar point cloud is low and the local features are few, the feature map after the third downsampling is passed through two hole convolutions with different expansion factors in parallel to expand the receptive field of the network, and the two outputs of the hole convolution are spliced ​​with the feature map after the third downsampling as high-dimensional global features. Finally, the outputs of multiple dimensions are upsampled to the same size through deconvolution and spliced ​​to obtain the feature map of 2D convolution extraction features.

[0063] S33: The feature map F obtained in step S32 is in , to strengthen the attention of the channel dimension, firstly, the input feature map is subjected to the maximum pooling of the spatial dimension and the average pooling of the spatial dimension, and two sizes of (C in , 1, 1) feature block F channel_max 、F channel_avg , and these two feature blocks are extracted by multi-layer perceptron and activation function, and then the feature extraction results are added element by element and then passed through a sigmoid activation function to obtain the feature output F of channel dimension attention enhancement channel_output , F channel_output With F in Multiplying them strengthens the information in the channel dimension, making the recognizable obstacle information more prominent.

[0064] F channel_output =sigmoid(MLP(AvgPool(F in ))+MLP(MaxPool(F in )))

[0065] F′ in =F channel_output *F in

[0066] S34: The feature map F′ obtained in step S33 is in , to strengthen the attention in the spatial dimension, firstly, the input feature map is subjected to the maximum pooling of the channel dimension and the average pooling of the channel dimension, and two sizes of (1, H in , W in )’s feature map F spatial_max 、F spatial_avg , and after the two feature maps are spliced, they are converted into (1, H in , Win ), and then the feature map is passed through a sigmoid activation function to obtain the feature output F of spatial dimension attention enhancement spatial_output , F spatial_output and F′ in Multiplying them strengthens the information of the spatial dimension, making the area where the obstacles are located more noticeable.

[0067] F spatial_output =conv([AvgPool(F′ in );MaxPool(F′ in )])

[0068] F output =F spatial_output *F in

[0069] S6. Input the feature map after channel dimension and spatial dimension attention enhancement into the SSD detection head to perform obstacle target detection.

[0070] The present invention adopts 4D millimeter-wave radar as the main sensor, which can not only ensure the accuracy of detection in bad weather, but also make up for the problems of lack of height information and low resolution of traditional 3D millimeter-wave radar. The 4D millimeter-wave radar output point cloud has radial distance, azimuth, pitch angle, and radial velocity information. The output point cloud is similar to the laser radar, contains 3D geometric information of the target, and has good robustness under extreme weather conditions; at the same time, the present invention only uses single-mode 4D millimeter-wave radar point cloud information for obstacle target detection. Compared with the multimodal method of millimeter-wave radar point cloud and camera image, the required equipment cost is lower and the environmental perception information relied on is less. In the train driving environment, the detection range is dynamically set according to the clustering results of the current frame point cloud, and the deep learning network model enhanced by the attention mechanism is used to detect the target of the obstacle, so that computing resources can be saved while ensuring the detection accuracy, thereby realizing intelligent assisted driving of the train.

[0071] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A train forward obstacle detection method based on 4D millimeter wave radar, characterized in that: The steps include: S1. Collect the original point cloud data of 4D millimeter wave radar and obtain obstacle point cloud data after preprocessing; S2. Clustering the obstacle point cloud data to determine the detection range on both sides of the current frame point cloud track; S3. Divide the point cloud after clustering into columns, randomly select multiple points for each column for feature extraction, perform maximum pooling on the information of each dimension in the column point cloud, find the most representative feature of the current column, and then restore the most representative feature of the current column into a pseudo image according to the area represented by the column; S4. Input the pseudo image into the 2D convolution backbone network to perform multi-dimensional and multi-scale feature extraction to obtain a feature map of the 2D convolution extracted features; S5. Strengthen the channel dimension and spatial dimension attention of the feature map of the 2D convolution feature extraction; S6. Input the feature map after channel dimension and spatial dimension attention enhancement into the SSD detection head to perform obstacle target detection; Step S1 includes: collecting original point cloud data of 4D millimeter wave radar; performing statistical filtering on the original point cloud data to remove noise points; using straight-through filtering to filter out point clouds that are 1.5 times higher than the height of the train; and obtaining obstacle point cloud data after preprocessing; In step S5, the feature map of the 2D convolutional feature extraction is subjected to channel dimension attention enhancement, including: the input feature map is subjected to maximum pooling in the spatial dimension and average pooling in the spatial dimension, respectively, to obtain two sizes of (C in ,1,1) feature block F chanel_max 、F channel_avg ; The two feature blocks are extracted by multi-layer perceptron and activation function; the results of feature extraction are added element by element and then passed through a sigmoid activation function to obtain the channel dimension attention weight F channel_output , F channel_output With F in Multiply them to strengthen the information of channel dimension; among them, F channel_output =sigmoid(MLP(AvgPool(F in ))+MLP(MaxPool(F in ))) F′ in =F channel_output *F in ; In step S5, the feature map of the 2D convolutional feature extraction is subjected to spatial dimension attention enhancement, including: the input feature map is subjected to maximum pooling in the channel dimension and average pooling in the channel dimension, and two sizes of (1, H in ,W in )’s feature map F spatial_max 、F spatial_avg ; After the two feature maps are concatenated, they are converted into (1,H in ,W in ) feature map; the feature map is passed through a sigmoid activation function to obtain the spatial dimension attention weight F spatial_output , F spatial_output and F′ in Multiply to strengthen the information of spatial dimension; among them, F spatial_output =conv([AvgPool(F′ in );MaxPool(F′ in )]) F output =F spatial_output *F′ in 。 2. According to claim 1, a method for detecting obstacles ahead of a train based on a 4D millimeter wave radar, characterized in that: In step S2, the obstacle point cloud data is clustered using the DBSCAN clustering algorithm.

3. According to claim 1, a method for detecting obstacles ahead of a train based on 4D millimeter wave radar, characterized in that: In step S3, 32 points are randomly selected from each column for feature extraction; if the number of points in the column is less than 32, the average value of all points in the current column is used for filling.

4. According to claim 3, a method for detecting obstacles ahead of a train based on a 4D millimeter wave radar is characterized in that: In step S3, 11-dimensional features (x, y, z, r, v, x) are extracted for each point. c ,y c ,z c ,x p ,y p ,z p ), where x, y, z represent the position information of each point in the Cartesian coordinate system, r represents the reflection intensity of the object, v is the Doppler velocity of the object, and x c ,y c ,z c The difference between the point and the average value of all points in the current column, x p ,y p ,z p is the relative position of the point and the geometric center of the column.

5. According to claim 4, a method for detecting obstacles ahead of a train based on 4D millimeter wave radar, characterized in that: In step S3, the most representative feature of the current column is restored into a (C, H, W) pseudo image according to the area represented by the column, where C represents the feature dimension of each column, and H and W represent the resolution of the pseudo image in the X-axis direction and the Y-axis direction, respectively.

6. The method for detecting obstacles ahead of a train based on 4D millimeter wave radar according to claim 1, characterized in that: Step S4 includes: downsampling the input pseudo image three times to obtain feature maps of three scales, which respectively represent the features of different dimensions of the current frame point cloud; passing the feature map after the third downsampling in parallel through two dilation factors of different hole convolutions, and splicing the two outputs after the hole convolution with the feature map after the third downsampling as a high-dimensional global feature; upsampling the outputs of multiple dimensions to the same size through deconvolution and splicing them to obtain the feature map of the 2D convolution extraction feature.

Images