Deep learning network based on vehicle-mounted 4d radar point cloud

Abstract

The application provides a deep learning network based on a vehicle-mounted 4D radar point cloud, comprising the following steps: S1, collecting scene data to form a 4D radar point cloud dataset, and dividing the dataset into a training set, a verification set and a test set; S2, dividing scene point cloud data in the training set into multiple spatial regions, and learning point cloud distribution features in different spatial regions through a density-aware PointRCNN network; S3, learning global semantic features of points point by point, searching and traversing spatial regions to segment foreground points from the input point cloud, and generating a 3D proposal box to estimate the position and size of a vehicle target; S4, learning local features through local coordinate conversion, optimizing the position and direction of the 3D proposal box, and thus generating a 3D bounding box for detecting the vehicle target; and S5, setting a network loss function, and training the density-aware PointRCNN network model in combination with the verification set until the network model converges. The application can obtain a more robust network model and stronger target detection capability.

Description

TECHNICAL FIELD

[0001] The present application relates to 4D radar point cloud vehicle target detection technology, and particularly relates to a deep learning network based on vehicle-mounted 4D radar point cloud. BACKGROUND

[0002] In the environment of rapid development of advanced driving assistance system (ADAS), the 4D millimeter wave radar system is widely concerned by researchers due to its low cost and advantage of not being affected by bad environment, and has important research significance. Considering the advantages of the 4D millimeter wave radar, the vehicle target detection technology of the system is imminent, and there are few related researches at present. It is also an important attempt to rely on the 4D radar alone to perceive the environment.

[0003] In the past, the millimeter wave radar mainly utilizes the scattering field characteristics generated at the vehicle target, such as echo amplitude, phase, polarization, etc., to realize vehicle target detection through echo modeling and other schemes. However, in such detection schemes, the information representation is complex. For example, the vehicle target posture changes complex in the driving process, and the engine vibration of the vehicle and the limb swing of the pedestrian will cause nonlinear changes in the Doppler feature, resulting in blurred imaging. Moreover, the detection resolution and generalization ability of the previous schemes need to be improved, and it is difficult to meet the real-time requirements of ADAS technology.

[0004] Compared with the traditional echo features of the radar, the point cloud data can provide more accurate vehicle target geometry, and is less affected by the vehicle target motion. Moreover, compared with Mesh, voxel and other types, the point cloud data has simple expression, which can meet the end-to-end application requirements of the ADAS system and improve the reliability and stability of the vehicle perception system. Therefore, the 4D radar echo signal processing is output in the form of point cloud, which simply and powerfully represents the spatial vehicle target information, and is combined with the existing point cloud vehicle target detection technology for research. At present, the vehicle target detection scheme based on the 3D point cloud of the laser radar has made great progress, but the 4D millimeter wave radar point cloud data is much sparser than the laser radar, and the system often works in a complex road traffic environment. How to capture real-time and accurate vehicle target information by using the 4D radar point cloud still needs further research. SUMMARY

[0005] The present application proposes a deep learning network based on vehicle-mounted 4D radar point cloud to solve the problems of lack of 4D millimeter wave radar vehicle target detection scheme and low detection accuracy, and has the potential to explore the independent environmental perception work of vehicle-mounted 4D millimeter wave radar.

[0006] In order to achieve the above purpose, the present application proposes a deep learning network based on vehicle-mounted 4D radar point cloud, which specifically comprises the following steps:

[0007] S1, collect scene data to form a 4D radar point cloud dataset, and divide it into a training set, a validation set and a test set;

[0008] S2, based on the distribution characteristics of the radar point cloud, divide the scene point cloud data in the training set into multiple spatial regions, and learn the point cloud distribution characteristics in different spatial regions through the density-aware PointRCNN network;

[0009] S3, learn the global semantic features of the point cloud point by point, search and traverse the spatial region, and segment the foreground points from the input point cloud, and generate a 3D proposal box corresponding to the foreground points according to the distribution characteristics of the foreground points, to estimate the position and size of the vehicle target;

[0010] S4, learn the local features of the foreground point cloud through local coordinate transformation, so as to optimize the position and direction of the 3D proposal box, thereby generating a 3D bounding box for detecting the vehicle target;

[0011] S5, set the network loss function, and train the density-aware PointRCNN network model combined with the validation set until it converges;

[0012] S6, test the density-aware PointRCNN network model trained in step S5 on the test set to realize the detection of the vehicle target.

[0013] Further, a data acquisition system is used to collect scene data to form a 4D radar point cloud dataset; the data acquisition system comprises a 4D millimeter wave radar, a laser radar and a camera, and the 4D millimeter wave radar, the laser radar and the camera simultaneously collect scene data.

[0014] Further, the method for generating the 4D radar point cloud dataset specifically comprises the following steps:

[0015] S1.1, screen the 4D radar point cloud scene collected by the 4D millimeter wave radar, and sample the data frames in the point cloud scene;

[0016] S1.2, align the time stamps of the data collected by the 4D millimeter wave radar, the laser radar and the camera to realize time synchronization;

[0017] S1.3, convert the key frame data collected by the 4D millimeter wave radar and the laser radar to the same coordinate system to realize spatial synchronization;

[0018] S1.4, refer to the scene shot by the laser radar and the camera, and carry out 4D millimeter wave radar point cloud labeling work, and finally generate a 4D radar point cloud dataset.

[0019] Further, in the step S2, based on the distribution characteristics of the 4D radar point cloud data in the training set, the point cloud data in each scene is divided into three spatial regions according to the distance range, which are near region, middle region and far region, and there is allowed to have overlap between adjacent regions.

[0020] Further, in the step S2, based on the distribution characteristics of the 4D radar point cloud data in the training set, the point cloud data in each scene is divided into three spatial regions according to the distance range, which are near region, middle region and far region, and there is allowed to have overlap between adjacent regions.

[0021] Further, in the step S2, the backbone network of the density-aware PointRCNN network comprises three branch networks, and the point cloud data in the three spatial regions are respectively input into the three branch networks; an uncertainty sampling method is used to reasonably set the input point number of each branch network, so as to balance the detection effect of the vehicle target in each region.

[0022] Further, the uncertainty sampling method is as follows:

[0023] The average values of the point cloud data in the near region, the middle region and the far region are m1, m2 and m3 respectively.

[0024] The standard deviations of the point cloud data in the near region, the middle region and the far region are σ1, σ2 and σ3 respectively.

[0025] The division rule is: m1+α·σ1, m2+β·σ2, m3+γ·σ3 are respectively sampled from the near region, the middle region and the far region.

[0026] Further, in the step S3, the method for estimating the position and size of the vehicle target according to the 3D proposal box comprises the following steps:

[0027] S3.1, limiting the search distance in the X and Z directions of the foreground points, and dividing the search distance into uniform region segments according to a certain length;

[0028] S3.2, using a cross-entropy loss function based on region segments to optimize the positioning of the vehicle target;

[0029] S3.3, dividing the direction 2π into n uniform intervals, and calculating the residual regression angle along the X and Z directions respectively and the region classification angle to estimate the orientation θ of the vehicle target;

[0030] S3.4, calculating the average size residual of the vehicle target to regress the size (h, w, l) of the vehicle target;

[0031] The formula for positioning the vehicle target is as follows:

[0032]

[0033]

[0034]

[0035] where (x (p) ,y (p) ,z (p) ) denotes the foreground point position of interest, (x p ,y p ,z p ) is the vehicle target center, is the position of the point in the X and Z directions, is the residual in the region segment, used to optimize positioning accuracy, is the estimated error in the y direction.

[0036] Further, the step S4 specifically comprises the following steps:

[0037] S4.1, enlarge the 3D proposal box to perform a pooling operation on the foreground points and the corresponding local features;

[0038] S4.2, for each foreground point, determine whether the foreground point is contained in the new 3D proposal box after enlargement; if yes, retain the foreground point and its features to fine-tune the 3D proposal box;

[0039] S4.3, perform local coordinate system conversion on the pooled points in the new 3D proposal box;

[0040] S4.4, learn the local spatial features of the 3D proposal box based on the local coordinate system;

[0041] S4.5, make the local spatial feature dimension of the proposal box the same as the global semantic feature by encoding, connect the local and global features and send them into a point cloud encoder for learning, for fine-tuning the bounding box and optimizing the confidence classification;

[0042] S4.6, use non-maximum suppression to remove the bounding boxes, to obtain the accurate 3D bounding box of the vehicle target.

[0043] Further, the network loss function of the step S5 comprises a minimum bounding box regression loss under different training loss items, and a loss in the proposal box optimization stage.

[0044] The minimum bounding box regression loss L reg under different training loss items can be expressed as:

[0045]

[0046]

[0047]

[0048] where N is the number of foreground points pos , is the predicted segmentation residual, is the position residual of foreground point p, and is the corresponding value of the real vehicle target, F cls is the cross-entropy loss, F reg is the smooth L1 loss

[0049] where, C is the total number of classifications;

[0050] The loss of the proposal box optimization stage can be represented as:

[0051]

[0052] where B pos represents a positive regression proposal box, prob i is the estimated confidence of label i is the corresponding vehicle target class, F cls is the cross-entropy loss of the estimated confidence, calculated using the pooled bounding box and The calculation method is similar to and in the above formula.

[0053] Further, the step S5 specifically comprises the following steps:

[0054] S5.1, taking the 4D millimeter wave radar point cloud data set as input, observing the network detection performance, and verifying the effectiveness of the application;

[0055] S5.2, learning the global point cloud features of the training set and the labeled foreground point features, and generating a large number of preliminary vehicle target 3D proposal boxes;

[0056] S5.3, learning the local features of the point cloud in the local coordinate system, and combining the global features to fine-tune and optimize the proposal boxes, realize the center positioning and size estimation of the vehicle target, and evaluate the model training state through the loss function curve, and constantly adjust the density perception PointRCNN network model parameters;

[0057] S5.4, using the trained model to verify the detection effect on the verification set, and further optimizing the density perception PointRCNN network model parameters according to the detection effect;

[0058] S5.5, repeatedly performing steps S5.2-S5.4, constantly adjusting the density-aware PointRCNN network model parameters, so that the network loss function converges, and finally obtaining an effective and reliable density-aware PointRCNN network model.

[0059] The present application has the following advantages:

[0060] The present application takes the 4D radar point cloud dataset as the input of the density-aware PointRCNN network, and divides multiple spatial regions based on the distribution characteristics of the radar point cloud, thereby balancing the effectiveness and reliability of network sampling, learning target features based on the multi-branch density-aware PointRCNN network, generating preliminary 3D proposal boxes, and learning point cloud local features through local coordinate conversion, learning the local detail features of each proposal box and removing redundant information, thereby optimizing the position and direction of the 3D proposal box, and finally generating a 3D bounding box of the detected target, obtaining a more robust network model and stronger target detection capability according to the point cloud distribution characteristics. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 It is a flowchart of a deep learning network based on vehicle-mounted 4D radar point cloud of the present application;

[0062] Figure 2 It is a configuration diagram of a data acquisition system;

[0063] Figure 3 It is a 4D millimeter wave radar point cloud annotation scene diagram;

[0064] Figure 4 It is a point cloud distribution with distance in the 4D radar data set;

[0065] Figure 5 It is the first stage structure of the density-aware PointRCNN network;

[0066] Figure 6 It is the backbone network structure of the density-aware PointRCNN network;

[0067] Figure 7 It is a schematic diagram of 3D proposal box local coordinate conversion;

[0068] Figure 8 It is a comparison diagram of two network loss function curves in the training process;

[0069] Figure 9 It is a precision-recall curve comparison of the method of the present application and the PointRCNN network;

[0070] Figure 10 It is a vehicle target detection result and effect comparison based on the method of the present application. DETAILED DESCRIPTION

[0071] The application will be described in further detail in connection with the attached drawings and specific embodiments. The advantages and features of the present application will become more fully apparent from the following description and appended claims. It is to be understood that the drawings are simplified and are not drawn to scale, and are merely intended to aid in the description of the embodiments of the present application.

[0072] As shown in Figure 1 The present application proposes a deep learning network based on vehicle-mounted 4D radar point cloud, comprising the following steps:

[0073] S1, collecting scene data to form a 4D radar point cloud dataset, and dividing it into a training set, a validation set and a test set;

[0074] Specifically, the scene data is collected by a data acquisition system, as shown in Figure 2 The data acquisition system comprises a 4D millimeter wave radar, a laser radar, a camera and the like, and the 4D millimeter wave radar and the laser radar and the camera simultaneously collect scene data. Then, the data collected by the 4D millimeter wave radar is preprocessed such as screening, cropping and labeling for vehicle targets to form a 4D radar point cloud dataset.

[0075] The method for generating the 4D radar point cloud dataset specifically comprises the following steps:

[0076] S1.1, screening the 4D radar point cloud scene collected by the 4D millimeter wave radar, and sampling the data frames in the point cloud scene.

[0077] Screening the 4D radar point cloud scene can ensure the diversity of buildings, vehicles and left and right traffic in the scene, and sampling the data frames can improve the reliability of the dataset; in this embodiment, the data frames are sampled at 2Hz.

[0078] S1.2, aligning the timestamps of the data collected by the 4D millimeter wave radar, the laser radar and the camera; achieving time synchronization.

[0079] S1.3, converting the key frame data collected by the 4D millimeter wave radar and the laser radar to the same coordinate system; achieving spatial synchronization.

[0080] The key frame data collected by the 4D millimeter wave radar and the laser radar is converted to the same coordinate system by using the fixed relationship of the sensor coordinates.

[0081] S1.4, referring to the scene photographed by the laser radar and the camera, carrying out 4D millimeter wave radar point cloud labeling work, and finally generating a 4D radar point cloud dataset.

[0082] In this embodiment, the PCAT tool is used to carry out 4D millimeter wave radar point cloud labeling work, as shown inFigure 3

[0083] The 4D radar point cloud dataset contains a label file generated by a labeling tool and 4D millimeter wave radar point cloud, as well as pictures and laser radar data of the same scene for labeling and detection effect reference.

[0084] The 4D radar point cloud dataset is proportionally divided into a training set, a validation set and a test set. In this embodiment, the 4D radar point cloud dataset data is 3726 frames, which is divided into a training set, a validation set and a test set according to the leave-out method, and the division ratio is 3:1:1.

[0085] S2, based on the distribution characteristics of the 4D radar point cloud data in the training set, the point cloud data in each scene is divided into multiple spatial regions, and the density-aware PointRCNN network model is used to learn the point cloud distribution characteristics in different spatial regions.

[0086] Specifically, the distribution characteristics of the 4D radar point cloud data are analyzed and the distribution curve is fitted, as shown in Figure 4 It can be seen that the distribution curve tends to be positively skewed, and two points (s1, n1) and (s2, n2) are obtained on the curve, where (s1, n1) is the peak point of the curve, (s1 < s2). In this embodiment, the point cloud data in the scene spatial region is divided into three spatial regions according to 0-s1, s1-s2 and s2-s max , respectively, corresponding to the near, middle and far regions, where s max is the maximum distance of the distribution, as shown in Figure 5 To avoid the vehicle target being cut off at the boundary, the adjacent regions are allowed to overlap by ω, that is, the distribution interval is 0-s1+ω, s1-s2+ω and s2-s max , so as to obtain a more robust branch.

[0087] As shown in Figure 5 and Figure 6 (1), the spatial point cloud is input into the first stage of the density-aware PointRCNN network model backbone network to adaptively learn the point cloud distribution characteristics in different spatial regions through the density-aware PointRCNN network model. The backbone network of the density-aware PointRCNN network contains multiple branches, and by reasonably setting the input point number of each branch, the influence of the balanced sampling strategy on the distribution of each spatial region can be balanced, so as to balance the vehicle target detection effect of each spatial region, and make the vehicle target 3D proposal box more accurate and stable.

[0088] ​In this embodiment, the sampling strategy with uncertainty is adopted: when processing a single frame, the density-aware network PointRCNN samples 16384 points from the spatial point cloud as input, and for the density-aware network PointRCNN, the input needs to be reasonably divided. Statistics show that the average values of the point cloud data in the near, middle and far regions are m1, m2 and m3. Further analysis of the distribution of points shows that the standard deviations of the point cloud data in the near, middle and far regions are σ1, σ2 and σ3. In order to balance the influence of the sampling strategy on the learning of features in each region of the model and enhance the vehicle target detection effect in the middle and far regions, the division rule is adopted: m1+α·σ1, m2+β·σ2, m3+γ·σ3, wherein the coefficients (α, β, γ) are selected according to the detection effect.

[0089] S3, learn the global semantic features of the point cloud point by point, segment the foreground points from the input point cloud after searching and traversing the spatial region, and generate a 3D proposal box corresponding to the foreground points according to the distribution characteristics of the foreground points to estimate the position and size of the vehicle target.

[0090] Specifically, according to the method of separating foreground points from input point cloud according to artificial annotation 3D box and estimating the position and size of vehicle target according to 3D proposal box, the following steps are included:

[0091] S3.1, the point cloud covered by the artificial annotation 3D box is regarded as the foreground point, the search range S in the X and Z directions of the foreground point is limited, and the search range is divided into uniform region segments according to the length δ;

[0092] S3.2, using the cross-entropy loss function based on the region segment to optimize the vehicle target positioning; so that the result is more accurate and robust.

[0093] The vehicle target positioning formula is as follows:

[0094]

[0095]

[0096]

[0097] In the formula, (x (p) ,y (p) ,z (p) ) represents the position of the foreground point of interest, (x p ,y p ,z p ) is the center of the vehicle target, is the position of the vehicle target in the X and Z directions, is the residual error in the region segment, which is used to optimize the positioning accuracy, and since the y direction is perpendicular to the ground and downward, the variation is small, only the estimation error in the y direction is considered

[0098] S3.3, divide the direction 2π into n uniform intervals, and calculate the residual regression angle along the x, z direction respectively and the region classification angle to estimate the vehicle target orientation θ;

[0099] Specifically, the residual regression angle and the region classification angle are calculated based on the vehicle target positioning formula and the region classification angle

[0100] S3.4, calculate the vehicle target average size residual to estimate the vehicle target size length, width, height;

[0101] Specifically, considering that the relative change of the size of each category in the training set is small, the vehicle target average size residual is calculated based on the target positioning formula to estimate the vehicle target size.

[0102] S4, learn the local features of the foreground point cloud through local coordinate conversion, thereby optimizing the position and direction of the 3D proposal box, and generating a 3D bounding box for detecting the vehicle target; obtain a more robust network model and stronger vehicle target detection capability.

[0103] Further, step S4 can be implemented through the following sub-steps:

[0104] S4.1, enlarge the 3D proposal box to perform pooling operation on the foreground points and the corresponding local features.

[0105] Specifically, for the 3D proposal box b i = (x i , y i , z i , h i , w i , l i , θ i ), enlarge it outward to generate a new 3D proposal box wherein h, w, and l are the length, width, and height of the vehicle target respectively, and η is the enlargement scale.

[0106] S4.2, for each foreground point p = (x (p) , y (p) , z (p) ), determine whether the foreground point p is contained in the new 3D proposal box after enlargement. If yes, retain the foreground point and its features to fine-tune the 3D proposal box b i .

[0107] S4.3, perform local coordinate system conversion on the pooled points in the new 3D proposal box.

[0108] Specifically, the foreground points after the pooling processing in step S4.1 are fed into the second stage in the backbone network of the density-aware PointRCNN network model, as shown in FIG. 2, to refine the bounding box information and improve the confidence of the vehicle target. Figure 6

[0109] In combination with the 3D proposal box and local feature information obtained in the first stage in the backbone network of the density-aware PointRCNN network, the foreground points in the 3D proposal box after the pooling are converted into a local coordinate system. As shown in FIG. 3, the proposal box features in the local coordinate system are: Figure 7

[0110] (1) The origin is located at the center of the 3D proposal box in the first stage;

[0111] (2) The local X' and Z' axes are parallel to the ground plane, wherein the X' axis is parallel to the orientation of the proposal box, the Y' axis is perpendicular to the ground downward, and the Z' axis is perpendicular to the X' and points to the left side of the proposal box.

[0112] If the intersection over union IoU of a real labeled bounding box and a 3D proposal box is greater than 0.55, the real labeled bounding box is assigned to the 3D proposal box for fine-tuning of the 3D proposal box. Both the 3D proposal box and the corresponding real bounding box are converted into the local coordinate system, i.e., the 3D proposal box b i i i i i i i i and the real bounding box b i gt i gt i gt i gt i gt i gt i gt i gt will be converted into

[0113] b i

[0114]

[0115] The center position of the i-th proposal box ​​​​​​​​​​​​​​​​​The setting method is the same as the above formula, but the search range S is smaller, and the 3D proposal box is fine-tuned. Regarding the average size of each class in the data set, the sparse point cloud that is pooled usually cannot provide sufficient size information (h i ,w i ,l i ), and the size residual is still directly regressed.

[0116] S4.4, learning the local spatial features of the 3D proposal box based on the local coordinate system.

[0117] Convert the coordinates p of all foreground points in the proposal box to the local coordinate system Convert the local point features as input, and use a multilayer perception machine to learn the local point features.

[0118] S4.5, by encoding, the dimensions of the local spatial features of the proposal box are the same as the global semantic features, the local and global features are connected and sent into the point cloud encoder for learning, which is used for fine-tuning the bounding box and optimizing the confidence classification.

[0119] S4.6, use Non-Maximum Suppression (NMS) to remove the bounding box, and get the accurate 3D bounding box of the vehicle target.

[0120] S5, set the network loss function, and train the density-aware PointRCNN network model combined with the validation set until it converges.

[0121] Specifically, the network loss function is used to reflect the training of the model, and is used as the basis for model optimization. The network loss function includes the minimum bounding box regression loss under different training loss terms, and the loss in the proposal box optimization stage.

[0122] Combined with the vehicle target positioning formula in step S3.2, the minimum bounding box regression loss L reg under different training loss terms can be represented as:

[0123]

[0124]

[0125]

[0126] Wherein, the number of foreground points is denoted as N pos , is the predicted segment to which the target belongs, is the predicted residual of the foreground point p, and are the true corresponding values of the target, and is the estimated error and true error value of the fine-tuning amount (y,w,h,l), F cls is the cross entropy loss, F reg is the smooth L1 loss, where C is the total number of categories,

[0127] The loss of the proposal optimization stage can be expressed as:

[0128]

[0129] Among them, B pos Indicates positive regression proposal box, prob i for The estimated confidence of label i is the corresponding vehicle target category, F cls To estimate the confidence cross entropy loss, the bounding box after pooling is used to calculate and The calculation method is the same as in the above formula and resemblance.

[0130] The density-aware PointRCNN network model is continuously trained and optimized by combining the vehicle target loss function with the validation and test sets until the network loss function converges, resulting in an effective and reliable density-aware PointRCNN network model. This includes the following steps:

[0131] S5.1. Using the 4D millimeter-wave radar point cloud dataset as input, observe the network detection performance and verify the effectiveness of the present invention.

[0132] Specifically, in this embodiment, the experiments are all carried out in the virtual environment Car_Detection created by conda, and the data are input into the PointRCNN network and the improved Three-branch PointRCNN network respectively.

[0133] S5.2. Learn the global point cloud features of the training set and the features of the annotated foreground points to generate a large number of preliminary 3D proposal boxes of vehicle targets.

[0134] S5.3. Learn the local features of the point cloud in the local coordinate system and fine-tune and optimize the proposal box in combination with the global features to achieve vehicle target center positioning and size estimation. The model training status is evaluated through the loss function curve and the model parameters are continuously adjusted.

[0135] S5.4. Use the trained model to verify the detection effect on the validation set, and further optimize the density-aware PointRCNN network model parameters based on the detection effect.

[0136] S5.5. Repeat steps S5.2 to S5.4, and continuously adjust the density-aware PointRCNN network model parameters so that the network loss function converges, and finally obtain an effective and reliable density-aware PointRCNN network model.

[0137] S6. Use the test set to test the density-aware PointRCNN network model trained in S5 to detect vehicle targets.

[0138] Comparison of loss function Loss curve during density-aware PointRCNN network model training Figure 8 The density-aware PointRCNN network model indicators are shown in Table 1. Combining the curves and network indicators, it can be seen that the convergence of the improved density-aware network PointRCNN model is less affected by the sample distribution, and the effect is better than PointRCNN.

[0139] Table 1 Comparison of network training

[0140]

[0141] In order to analyze the vehicle target detection performance of the density-aware PointRCNN network model, the precision-recall curve is drawn, as shown in Figure 9 Comparing the PR curves of the density-aware PointRCNN network processing and the PointRCNN network detection output, we find that the area under the PR curve of the density-aware PointRCNN network processing is significantly larger than that of the latter, significantly improving the algorithm's vehicle target detection performance. The calculated AP value during the test phase was 65.51%, 2.88% higher than the AP value of direct PointRCNN network detection, demonstrating the strong generalization ability of the detection network and good detection performance.

[0142] The final detection effect is visualized as follows Figure 10 As shown, Figure 10 (a) is a camera image of the scene, (b) is a bird's-eye view of the scene point cloud collected by the lidar, (c) is the detection result of the PointRCNN network, and (d) is the detection result of the density-aware PointRCNN network. The rectangular box in the figure is the vehicle target detection bounding box, and the vertical line segment outward from the bounding box indicates the direction of the vehicle head. It can be seen that both networks can basically correctly detect nearby vehicles. The PointRCNN network fails to detect some distant vehicles and vehicle targets with sparse point clouds, or the detection position deviation is large. The density-aware PointRCNN network improves this situation and can accurately detect distant vehicles. The detection effect of distant vehicle targets is greatly improved, basically meeting the requirements. The network improvement plan is effective and feasible.

[0143] While the application has been described in detail by reference to preferred embodiments thereof, it is to be understood that the description is not to be construed as limiting the scope of the application. Modifications and equivalents of what has been described are to be considered as falling within the scope of the application. Accordingly, the scope of the application is defined by the appended claims.

Claims

1. A deep learning network based on vehicular 4D radar point clouds, characterized in that, Comprising the following steps: S1, collecting scene data to form a 4D radar point cloud dataset, and dividing into a training set, a validation set and a test set; S2, based on the distribution characteristics of the radar point cloud, dividing the scene point cloud data in the training set into multiple spatial regions, and learning the point cloud distribution characteristics in different spatial regions through the density-aware PointRCNN network; S3, learning the global semantic features of the point cloud point by point, searching and traversing the spatial region to segment the foreground points from the input point cloud, and generating a 3D proposal box corresponding to the foreground points according to the distribution characteristics of the foreground points to estimate the position and size of the vehicle target; S4, learning the local features of the foreground point cloud through local coordinate transformation to optimize the position and direction of the 3D proposal box, thereby generating a 3D bounding box for detecting the vehicle target; S5, setting a network loss function, and training the density-aware PointRCNN network model in combination with the validation set until it converges; The method for estimating the position and size of the vehicle target according to the 3D proposal box in step S3 comprises the following steps: S3.1, limiting the search distance in the X and Z directions of the foreground points, and dividing the search distance into uniform region segments according to a certain length; S3.2, using a cross-entropy loss function based on the region segment to optimize the positioning of the vehicle target; S3.3, divide the direction 2π into n uniform intervals, and calculate the residual regression angle along the X, Z direction respectively and regional classification angle Estimate the vehicle target orientation θ in this way; S3.4, compute vehicle target average size residual Regression to obtain vehicle target size; The formula for the positioning of the vehicle target is as follows: where (x (p) ,y (p) ,z (p) ) represents the position of the foreground point of interest, (x p ,y p ,z p ) is the center of the vehicle target, is the position of the point in the X and Z directions, is the residual in the region segment, used to optimize positioning accuracy, is the estimated error in the y direction; The network loss function in step S5 comprises a minimum bounding box regression loss under different training loss items, and a loss in the proposal box optimization stage; The minimum bounding box regression loss L under the different training loss terms reg may be represented as: where the number of foreground points is denoted as N pos , is the predicted segment residual, is the position residual of foreground point p, and is the corresponding value of the real vehicle target, F cls is the cross-entropy loss, F reg is the smooth L1 loss; wherein, C is the total number of classifications; The loss in the proposal box optimization stage can be expressed as: where B pos represents the positive regression proposal box, prob i is the estimated confidence, label F i is the corresponding vehicle target category, F cls is the cross-entropy loss of the estimated confidence, calculated using the pooled bounding box and The calculation method is similar to the above formula and .

2. The deep learning network based on vehicular 4D radar point clouds of claim 1, wherein, Further comprising the following steps: S6, testing the density-aware PointRCNN network model trained in step S5 on the test set to realize the detection of the vehicle target.

3. The deep learning network based on vehicular 4D radar point clouds of claim 1, wherein, A data acquisition system is used to collect scene data to form a 4D radar point cloud dataset; the data acquisition system comprises a 4D millimeter wave radar, a laser radar and a camera, and the 4D millimeter wave radar, the laser radar and the camera simultaneously collect scene data.

4. The deep learning network based on vehicular 4D radar point clouds of claim 3, wherein, The generation method of the 4D radar point cloud dataset specifically comprises the following steps: S1.1, screening the 4D radar point cloud scene collected by the 4D millimeter wave radar, and sampling the data frames in the point cloud scene; S1.2, aligning the time stamps of the data collected by the 4D millimeter wave radar, the laser radar and the camera to realize time synchronization; S1.3, converting the key frame data collected by the 4D millimeter wave radar and the laser radar to the same coordinate system to realize spatial synchronization; S1.4, referring to the scene photographed by the laser radar and the camera, carrying out 4D millimeter wave radar point cloud labeling work, and finally generating a 4D radar point cloud dataset.

5. The vehicle-mounted 4D radar point cloud-based deep learning network of claim 1, wherein, In step S2, based on the distribution characteristics of the 4D radar point cloud data, the point cloud data in each scene is divided into three spatial regions according to the distance range, namely the near region, the middle region and the far region, and there is allowed overlap between adjacent regions.

6. The deep learning network based on vehicular 4D radar point clouds of claim 5, wherein, In the step S2, the backbone network of the density-aware PointRCNN network comprises three branch networks, and the point cloud data in the three spatial regions are respectively input into the three branch networks; an uncertainty sampling method is used to reasonably set the input point number of each branch network, so as to balance the detection effect of the vehicle target in each spatial region.

7. The deep learning network based on vehicular 4D radar point clouds of claim 6, wherein, The uncertainty sampling method is as follows: The average values of the point cloud data in the near region, the middle region and the far region are m1, m2 and m3 respectively. The standard deviations of the point cloud data in the near region, the middle region and the far region are σ1, σ2 and σ3 respectively. The division rules are m1+α·σ1, m2+β·σ2 and m3+γ·σ3 respectively for sampling from the near region, the middle region and the far region; wherein α, β and γ are coefficients.

8. The vehicle-mounted 4D radar point cloud-based deep learning network of claim 1, wherein, The step S4 specifically comprises the following steps: S4.1, amplifying the 3D proposal box to perform a pooling operation on the foreground points and the corresponding local features; S4.2, for each foreground point, judging whether the foreground point is contained in the new 3D proposal box after amplification; if yes, retaining the foreground point and its feature to fine-tune the 3D proposal box; S4.3, performing local coordinate system conversion on the pooled points in the new 3D proposal box; S4.4, learning the local spatial features of the 3D proposal box based on the local coordinate system; S4.5, making the local spatial feature dimension of the proposal box the same as the global semantic feature by encoding, connecting the local and global features and sending them into the point cloud encoder for learning, so as to fine-tune the bounding box and optimize the confidence classification; S4.6, using non-maximum suppression to remove the bounding box, to obtain the accurate 3D bounding box of the vehicle target.

9. The vehicle-mounted 4D radar point cloud-based deep learning network of claim 1, wherein, The step S5 specifically comprises the following steps: S5.1, taking the 4D millimeter wave radar point cloud data set as input, observing the network detection performance, and verifying the effectiveness of the present application; S5.2, learning the global point cloud features of the training set and the labeled foreground point features, to generate a large number of preliminary vehicle target 3D proposal boxes; S5.3, learning the local point cloud features in the local coordinate system, and combining the global features to fine-tune and optimize the proposal box, to realize the center positioning and size estimation of the vehicle target, and to evaluate the model training state through the loss function curve, and to continuously adjust the density-aware PointRCNN network model parameters; S5.4, using the trained model to verify the detection effect on the verification set, and further optimizing the density-aware PointRCNN network model parameters according to the detection effect; S5.5, repeatedly performing steps S5.2 to S5.4, continuously adjusting the density-aware PointRCNN network model parameters, so that the network loss function converges, and finally obtaining an effective and reliable density-aware PointRCNN network model.

Images