Millimeter-wave radar target detection model training method, detection method and device

Abstract

This invention provides a millimeter-wave radar target detection model training method, detection method, and apparatus. The method involves pre-training a millimeter-wave radar target detection model based on multiple consecutive frames of millimeter-wave radar point cloud data and 3D bounding box annotation information to obtain a pre-trained millimeter-wave radar target detection model. Ground truth (GT) bounding box velocity information is generated based on the multiple consecutive frames of millimeter-wave radar point cloud data. The pre-trained millimeter-wave radar target detection model is then subjected to velocity regression training based on the target bounding box predicted position information and GT bounding box velocity information output by the pre-trained model, resulting in a millimeter-wave radar target detection model with velocity estimation capabilities. Through pre-training and velocity regression training, accurate velocity prediction information can be obtained, meeting user needs. Furthermore, by automatically generating GT bounding box velocity information, the slow convergence problem caused by relying solely on self-supervised learning can be avoided, further improving velocity estimation accuracy.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a millimeter-wave radar target detection model training method, detection method, and device. Background Technology

[0002] Millimeter-wave radar utilizes radiated electromagnetic energy to measure targets within the sensor's field of view, exhibiting strong long-range target detection capabilities. With advancements in radar and chip technology, millimeter-wave radar is becoming increasingly smaller, enabling it to perform well in complex multi-target environments and provide better early warning for drivers. As millimeter-wave radar expands into higher levels of autonomous driving applications, it demands more accurate and detailed target information. Furthermore, with the rapid development of deep learning in computer vision, autonomous driving is beginning to adopt deep learning methods to replace traditional algorithms. Existing deep learning-based target detection methods rely on manually labeled data; deep neural networks undergo supervised learning using this data to adjust model parameters and complete pre-defined detection tasks. Predicting vehicle speed is crucial for target detection in autonomous driving scenarios. However, traditional millimeter-wave radar target detection methods rely on the spatiotemporal relationship of 3D bounding boxes labeled with consecutive frames to infer the vehicle's speed. Since these methods depend on consecutive frame annotation, which increases annotation costs, and autonomous driving is an open environment where manual annotation of consecutive frames is difficult, traditional LiDAR target detection methods lack speed inference capabilities. Furthermore, millimeter-wave radar point clouds only contain radial velocity information, and their positional errors are large, resulting in slow convergence speed and low accuracy in speed prediction, which fails to meet user needs. Summary of the Invention

[0003] This invention provides a millimeter-wave radar target detection model training method, detection method, and device to solve the shortcomings of traditional millimeter-wave radar target detection models that rely on labeled data for training, are not suitable for autonomous driving speed prediction that requires massive amounts of data, and have slow speed prediction convergence speed.

[0004] This invention provides a method for training a millimeter-wave radar target detection model, comprising:

[0005] Based on continuous multi-frame millimeter-wave radar point cloud data and 3D bounding box annotation information, the millimeter-wave radar target detection model is pre-trained to obtain the pre-trained millimeter-wave radar target detection model.

[0006] Generate GT frame velocity information based on the continuous multi-frame millimeter-wave radar point cloud data;

[0007] Based on the target bounding box predicted position information and the ground truth bounding box velocity information output by the pre-trained millimeter-wave radar target detection model, velocity regression training is performed on the pre-trained millimeter-wave radar target detection model to obtain the target predicted position of the millimeter-wave radar target detection model with velocity estimation capability.

[0008] According to the present invention, a method for training a millimeter-wave radar target detection model includes pre-training the millimeter-wave radar target detection model, comprising:

[0009] Coarse feature extraction is performed on the continuous multi-frame millimeter-wave radar point cloud data to obtain multi-frame coarse feature information from a bird's-eye view.

[0010] Multi-dimensional feature information is obtained by superimposing the coarse feature information of the multiple frames and then performing feature learning.

[0011] The predicted target bounding box information is obtained based on the multidimensional feature information;

[0012] Calculate the pre-training loss value based on the predicted target box information and the preset standard target box information;

[0013] The model parameters are adjusted based on the pre-training loss value to obtain a pre-trained millimeter-wave radar target detection model.

[0014] According to the millimeter-wave radar target detection model training method provided by the present invention, the step of obtaining the predicted target box information based on the multi-dimensional feature information includes:

[0015] Create a Doppler velocity map from a bird's-eye view, the Doppler velocity map comprising multiple grids;

[0016] Projecting multiple frames of point cloud data from the continuous multi-frame millimeter-wave radar point cloud data onto the Doppler velocity map;

[0017] The maximum Doppler velocity value in each of the multiple grids is extracted as the Doppler velocity feature information of the corresponding grid.

[0018] The multidimensional feature information is concatenated with the Doppler velocity feature information of each grid to obtain the concatenation result;

[0019] The predicted target box information is obtained based on the concatenation result.

[0020] According to the present invention, a millimeter-wave radar target detection model training method is provided, wherein generating ground truth bounding box velocity information based on the continuous multi-frame millimeter-wave radar point cloud data includes:

[0021] Foreground point segmentation is performed on the continuous multi-frame millimeter-wave radar point cloud data to obtain point cloud clusters belonging to each ground truth (GT) box;

[0022] The Doppler velocity of each point cloud is calculated based on the point cloud azimuth angle and the GT frame orientation angle in the point cloud cluster.

[0023] Outlier point clouds are removed based on the difference between the Doppler velocity of each point cloud and the median Doppler velocity of all point clouds in the point cloud cluster.

[0024] The GT box velocity information is obtained by considering the relationship between the Doppler velocity and the target box position in the point cloud cluster after removing outliers.

[0025] According to the present invention, a millimeter-wave radar target detection model training method is provided, wherein the step of performing velocity regression training on the pre-trained millimeter-wave radar target detection model to predict the target position based on the target bounding box prediction position information and the ground truth bounding box velocity information output by the pre-trained millimeter-wave radar target detection model includes:

[0026] Calculate the first loss value based on the predicted target location and the second loss value based on the GT box velocity information, respectively.

[0027] The weighted sum of the first loss value and the second loss value is used as the velocity regression training loss value;

[0028] The model parameters are adjusted based on the velocity regression training loss value to obtain a trained millimeter-wave radar target detection model.

[0029] According to the millimeter-wave radar target detection model training method provided by the present invention, the calculation of the first loss value based on the predicted target position includes:

[0030] Input the point cloud data at time t-1 from multiple consecutive frames of millimeter-wave radar point cloud data into the pre-trained millimeter-wave radar target detection model to obtain the predicted position of the target box at time t-1.

[0031] Based on the vehicle's pose change from time t-1 to time t, motion compensation is performed on the predicted position of the target box at time t-1.

[0032] The target position at time t is calculated based on the predicted position and velocity of the target box at time t-1 after motion compensation, thus obtaining the predicted position of the target box at time t.

[0033] The predicted location of the target box at time t is matched with the 3D bounding box annotation information at time t to obtain multiple matching pairs;

[0034] The average distance between the center points of multiple matching pairs is used as the first loss value.

[0035] The present invention also provides a millimeter-wave radar target detection method, comprising:

[0036] Acquire real-time millimeter-wave radar point cloud data;

[0037] The real-time millimeter-wave radar point cloud data is input into a millimeter-wave radar target detection model with velocity estimation capability to obtain target detection results. The target detection results include the target's position and velocity. The millimeter-wave radar target detection model is trained based on any of the millimeter-wave radar target detection model training methods described above.

[0038] The present invention also provides a millimeter-wave radar target detection model training device, comprising:

[0039] The pre-training module is used to pre-train the millimeter-wave radar target detection model based on multiple consecutive frames of millimeter-wave radar point cloud data and 3D bounding box annotation information, so as to obtain the pre-trained millimeter-wave radar target detection model.

[0040] The generation module is used to generate GT frame velocity information based on the continuous multi-frame millimeter-wave radar point cloud data;

[0041] The velocity regression training module is used to perform velocity regression training on the pre-trained millimeter-wave radar target detection model based on the target bounding box prediction position information and the ground truth (GT) bounding box velocity information output by the pre-trained millimeter-wave radar target detection model, so as to obtain the target prediction position of the millimeter-wave radar target detection model with velocity estimation capability.

[0042] The present invention also provides a millimeter-wave radar target detection device, comprising:

[0043] The acquisition module is used to acquire real-time millimeter-wave radar point cloud data;

[0044] The detection module is used to input the real-time millimeter-wave radar point cloud data into a millimeter-wave radar target detection model with velocity estimation capability to obtain target detection results. The target detection results include the target's position and velocity. The millimeter-wave radar target detection model is trained based on any of the millimeter-wave radar target detection model training methods described above.

[0045] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the millimeter-wave radar target detection model training method as described above, and / or the millimeter-wave radar target detection method as described above.

[0046] The millimeter-wave radar target detection model training method, detection method, and apparatus provided by this invention pre-train the millimeter-wave radar target detection model based on multiple consecutive frames of millimeter-wave radar point cloud data and 3D bounding box annotation information to obtain a pre-trained millimeter-wave radar target detection model; generate ground truth (GT) bounding box velocity information based on multiple consecutive frames of millimeter-wave radar point cloud data; and perform velocity regression training on the pre-trained millimeter-wave radar target detection model based on the target box predicted position information and GT bounding box velocity information output by the pre-trained millimeter-wave radar target detection model to obtain a millimeter-wave radar target detection model with velocity estimation capability. Through pre-training and velocity regression training, accurate velocity prediction information can be obtained, meeting user needs. Furthermore, by automatically generating GT bounding box velocity information, the slow convergence problem caused by relying solely on self-supervised learning can be avoided, further improving the velocity estimation accuracy. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating the millimeter-wave radar target detection model training method provided by the present invention;

[0049] Figure 2 This is a schematic diagram of the network structure paradigm of the millimeter-wave radar target detection model provided by the present invention;

[0050] Figure 3 This is a schematic diagram of the target bounding box prediction result output by the millimeter-wave radar target detection model provided by the present invention;

[0051] Figure 4 This is a flowchart illustrating the millimeter-wave radar target detection method provided by the present invention;

[0052] Figure 5 This is a schematic diagram of the structure of the millimeter-wave radar target detection model training device provided by the present invention;

[0053] Figure 6 This is a schematic diagram of the structure of the millimeter-wave radar target detection device provided by the present invention;

[0054] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0056] Figure 1 The flowchart of the millimeter-wave radar target detection model training method provided in the embodiments of the present invention is as follows: Figure 1 As shown, the millimeter-wave radar target detection model training method provided in this embodiment of the invention includes:

[0057] Step 101: Based on multiple consecutive frames of millimeter-wave radar point cloud data and 3D bounding box annotation information, pre-train the millimeter-wave radar target detection model to obtain the pre-trained millimeter-wave radar target detection model.

[0058] In this embodiment of the invention, the 3D frame annotation information includes the position and size of the 3D frame.

[0059] Step 102: Generate GT bounding box velocity information based on multiple consecutive frames of millimeter-wave radar point cloud data;

[0060] In this embodiment of the invention, the ground truth box (GT box) velocity information refers to the correct velocity annotation information. It should be noted that other GT box information, such as the target's location, size, and category, can also be generated based on multiple consecutive frames of millimeter-wave radar point cloud data.

[0061] Step 103: Perform velocity regression training on the pre-trained millimeter-wave radar target detection model based on the target bounding box predicted position information and GT bounding box velocity information output by the pre-trained millimeter-wave radar target detection model to obtain a millimeter-wave radar target detection model with velocity estimation capability.

[0062] It should be noted that the pre-trained millimeter-wave radar target detection model can also output information such as the target's category and size.

[0063] In this embodiment of the invention, velocity regression training of the pre-trained millimeter-wave radar target detection model based on the target's predicted position is achieved by estimating the target's velocity based on information such as the target's shape and position at different times. Based on the GT box velocity information, correct velocity annotation information can be directly provided for model training, thereby accelerating the speed and accuracy of model training.

[0064] Traditional millimeter-wave radar target detection models require 3D labeled data, which is usually in the form of 3D bounding boxes containing (x, y, z, w, h, l). Here, x, y, and z represent the target's position in the vehicle's Cartesian coordinate system, and w, h, and l represent the target's length, width, and height, respectively. The target's velocity cannot be obtained directly from these data. It is usually necessary to deduce the velocity of the labeled bounding boxes from the spatiotemporal relationship of the 3D bounding boxes labeled in consecutive frames. This usually depends on consecutive frame labeling, which increases the labeling cost and thus the velocity labeling cost. Therefore, traditional millimeter-wave radar target detection models lack velocity inference capabilities.

[0065] The millimeter-wave radar target detection model training method provided in this invention pre-trains the millimeter-wave radar target detection model based on multiple consecutive frames of millimeter-wave radar point cloud data and 3D bounding box annotation information to obtain a pre-trained millimeter-wave radar target detection model; generates ground truth (GT) bounding box velocity information based on multiple consecutive frames of millimeter-wave radar point cloud data; and performs velocity regression training on the pre-trained millimeter-wave radar target detection model based on the target box predicted position information and GT bounding box velocity information output by the pre-trained millimeter-wave radar target detection model to obtain a millimeter-wave radar target detection model with velocity estimation capability. Through pre-training and velocity regression training, accurate velocity prediction information can be obtained, meeting user needs. Furthermore, by automatically generating GT bounding box velocity information, the slow convergence problem caused by relying solely on self-supervised learning can be avoided, further improving the velocity estimation accuracy.

[0066] Based on any of the above embodiments, in this embodiment of the invention, a schematic diagram of the network structure paradigm of the millimeter-wave radar target detection model is shown below. Figure 2 As shown, a grid-based structure is used to process millimeter-wave radar point cloud data. The Pointpillars method is used to project the point cloud onto a bird's-eye view (BEV) grid. Then, a 2D convolutional backbone network is used to learn features. Target category, bounding box position, shape, and velocity attributes are predicted through the neck and multitask head. Based on the above network structure, step 101 pre-trains the millimeter-wave radar target detection model using multiple consecutive frames of millimeter-wave radar point cloud data and 3D bounding box annotation information. Specifically, this includes:

[0067] Step 1011: Perform coarse feature extraction on multiple consecutive frames of millimeter-wave radar point cloud data to obtain multi-frame coarse feature information from a bird's-eye view.

[0068] In this embodiment of the invention, n consecutive frames of millimeter-wave radar point cloud data form a millimeter-wave radar point cloud set p = {p1, p2, ..., pn}, where the i-th frame of point cloud data pi = {x, y, z, doppler, rcs, snr, t}, where x, y, z are the coordinate positions in a Cartesian coordinate system centered on the vehicle, doppler represents the Doppler velocity (which is the radar velocity relative to the stationary ground; typically, in operation, the vehicle's motion offset needs to be compensated for on top of the original doppler), rcs represents the radar backscattering cross section, snr represents the signal-to-noise ratio, and t represents the time difference between the time the point cloud data was acquired and the current timestamp. n can be flexibly selected based on sensor specifications and the region of interest. Typically, n = 5 can ensure better detection accuracy while avoiding a dramatic increase in computational load.

[0069] Introducing multi-frame point cloud data can effectively address the issue of excessive sparseness in millimeter-wave radar, while simultaneously providing the model with rich spatiotemporal semantic information. For example, after overlaying 500ms of point cloud data, the multiple frames of point cloud data need to be compensated to reflect the current pose of the vehicle.

[0070] Multi-frame point cloud data from millimeter-wave radar are processed using grid-based feature extraction to obtain coarse features for each frame under BEV (Browser-Earth Variable Object) conditions. These coarse features are a set of features with low feature dimension, including data such as point cloud acquisition time, backscatter cross section (RSC), and signal-to-noise ratio (SNR). Appropriate feature extraction methods can be selected as needed, such as the pointpillars approach.

[0071] Step 1012: After superimposing the coarse feature information from multiple frames, feature learning is performed to obtain multidimensional feature information;

[0072] In this embodiment of the invention, coarse feature information from multiple frames is superimposed to obtain feature input under multi-frame point cloud BEV, which is then fed into the backbone network for feature learning. This can yield more detailed multi-dimensional feature information. The backbone network can flexibly select appropriate methods as needed, such as Res-Net.

[0073] Step 1013: Obtain the predicted target box information based on the multi-dimensional feature information;

[0074] In this embodiment of the invention, multidimensional feature information is input into the neck and head modules in the network structure, and the target category, 3D bounding box and velocity information are output through the multitask head. The neck and head can be flexibly selected according to the needs of the appropriate network, such as the neck being the FPN feature pyramid network and the head being the centerpoint network.

[0075] Step 1014: Compare the predicted target box information with the ground truth box information, and use the comparison result as the pre-training loss value;

[0076] In this embodiment of the invention, the predicted target bounding box information includes a set of target bounding box 3D information b = {b1, b2, ..., bk}, where bk = {x, y, z, w, h, l}, x, y, z represent the position of the target in the vehicular Cartesian coordinate system, and w, h, l represent the length, width, and height of the target, respectively.

[0077] The predicted target location includes classification and regression results. Classification results include the target's category, while regression results include 3D information about the target bounding box, such as its location, dimensions, and height. The pre-training loss is calculated using the following methods:

[0078] Pre-training loss value L det =W cls L cls +W box L box Among them, W cls For the classification task weights, L cls For the classification task loss; W box To regress task weights, L box Losses due to the return mission.

[0079] By comparing the predicted category of the target bounding box with the actual category in the ground truth bounding box information, the classification task loss can be calculated. By comparing the predicted shape and position information of the target bounding box with the actual shape and position information in the ground truth bounding box information, the regression task loss can be calculated. Then, by selecting the classification task weights and regression task weights according to the actual situation, the pre-training loss value can be obtained.

[0080] Step 1015: Adjust the model parameters based on the pre-training loss value to obtain the pre-trained millimeter-wave radar target detection model.

[0081] In this embodiment of the invention, the model parameters are adjusted, for example, by adjusting the parameters in the methods selected for the backbone, neck, and head modules, thereby obtaining a pre-trained millimeter-wave radar target detection model.

[0082] In some embodiments of the present invention, pre-training the millimeter-wave radar target detection model based on the original millimeter-wave radar point cloud dataset further includes:

[0083] Create a Doppler velocity map from a bird's-eye view, which includes multiple grids;

[0084] Projecting multiple frames of point cloud data from continuous millimeter-wave radar point cloud data onto a Doppler velocity map;

[0085] Extract the maximum Doppler velocity value from each grid in multiple grids as the corresponding Doppler velocity feature information of the grid;

[0086] The multidimensional feature information is concatenated with the Doppler velocity feature information of each grid to obtain the concatenation result;

[0087] The predicted target box information is obtained based on the concatenation results.

[0088] After obtaining the BEV features through the backbone, they are concatenated with the previously obtained time difference features. This ensures that the network can learn spatiotemporal information better, determine which point cloud data participates in subsequent model calculations, whether there is data loss, obtain information on the velocity distribution of the point cloud, and improve the accuracy of velocity regression.

[0089] Based on any of the above embodiments, in this embodiment of the invention, step 102, which generates GT bounding box velocity information based on multiple consecutive frames of millimeter-wave radar point cloud data, specifically includes:

[0090] Step 1021: Perform foreground point segmentation on multiple consecutive frames of millimeter-wave radar point cloud data to obtain point cloud clusters belonging to each ground truth (GT) box;

[0091] By matching millimeter-wave radar point cloud data with ground truth (GT) boxes, point cloud clusters belonging to each GT box can be obtained.

[0092] Step 1022: Calculate the Doppler velocity of each point cloud based on the azimuth angle of the point cloud cluster and the orientation angle of the GT frame;

[0093] Let the azimuth angle of the point cloud be θ, and the orientation angle of the GT frame be... The velocity of each point cloud is then: v r This represents the Doppler velocity of the point cloud.

[0094] Step 1023: Remove outlier point clouds based on the difference between the Doppler velocity of each point cloud and the median Doppler velocity of all point clouds in the point cloud cluster;

[0095] For example, point clouds whose difference from the median Doppler velocity of all point clouds in the point cloud cluster is greater than 2 m / s are considered outliers and removed.

[0096] Step 1024: Based on the relationship between the Doppler velocity and the target box position in the point cloud cluster after removing outliers, solve the system of equations simultaneously;

[0097] The system of equations is as follows:

[0098] Among them, v r The doppler velocity of the point cloud is represented by θ, and the azimuth angle of the point cloud relative to the radar is represented by v. x ,v yThese represent the velocities of the entire point cloud cluster in the x-axis and y-axis directions, respectively.

[0099] Step 1025: Obtain the velocity information of the GT frame by solving the system of equations using least squares.

[0100] By solving the system of equations using least squares, the velocity of the target in the x-axis and y-axis directions can be obtained. The velocity of the target in the x-axis and y-axis directions is used as the velocity information of the ground plane (GT) frame.

[0101] The above steps can automatically generate ground truth bounding box velocity information, eliminating the need for manual velocity annotation and saving costs.

[0102] Based on any of the above embodiments, in this embodiment of the invention, step 103, which involves performing velocity regression training on the pre-trained millimeter-wave radar target detection model based on the target predicted position and GT box velocity information, specifically includes:

[0103] Step 1031: Calculate the first loss value based on the predicted target position and the second loss value based on the GT box velocity information, respectively;

[0104] In this embodiment of the invention, calculating the first loss value based on the predicted target location includes:

[0105] Step 10311: Input the point cloud data at time t-1 from the original millimeter-wave radar point cloud dataset into the pre-trained millimeter-wave radar target detection model to obtain the predicted target box position at time t-1.

[0106] The predicted position of the target bounding box at time t-1 is Bt-1={b1…bk}, bk={x,y,z,w,h,l}, where x, y, and z represent the position of the target in the Cartesian coordinate system, and w, h, and l represent the length, width, and height of the target, respectively.

[0107] Step 10312: Based on the vehicle's pose change from time t-1 to time t, perform motion compensation on the predicted position of the target box at time t-1.

[0108] Step 10313: Based on the predicted velocity, predict the position of the Bt-1 after motion compensation. Then, the predicted position of the target box at time t is: x' = x + v x *Δt, y'=y+v y *Δt, Δt = time t - time t-1;

[0109] Step 10314: Compare the predicted location of the target box at time t with the ground truth box information, and use the comparison result as the velocity regression training loss value;

[0110] Obtain the GT bounding box information at time t: GT bounding box Bt = {b1…bm}, bm = {x,y,z,w,h,l}. x,y,z represent the actual position of the target in the vehicle's Cartesian coordinate system, and w, h,l represent the actual length, width, and height of the target, respectively.

[0111] In some embodiments of the present invention, the predicted location of the target bounding box at time t is compared with the ground truth bounding box information, and the comparison result is used as the first loss value, including:

[0112] The predicted target location at time t is matched with the ground truth bounding box information at time t to obtain multiple matching pairs;

[0113] Calculate the distance between the center points of multiple matching pairs, and use the average of the center point distances as the first loss value.

[0114] like Figure 3 As shown, the left side displays the target position predicted by the model based on the point cloud input at time t-1. Based on the target's position and velocity at time t-1, the target's position at time t can be predicted. The solid box on the right represents the predicted target position, and the dashed box represents the position of the ground truth (GT) box. A matching algorithm obtains matching pairs between the detection and GT. The distance between the center points of the matching pairs can be used to characterize the velocity estimation loss. The target point matching method can employ the Hungarian matching algorithm to obtain one-to-one matching pairs. If a match is unsuccessful, the unmatched target point is ignored. For the obtained n matching pairs, the distance between the center points of each matching pair, D = {d1, d2, ..., dn}, is calculated.

[0115] Therefore, the first loss value is Velocity prediction is monitored by the positional error of velocity prediction.

[0116] The second loss value, predicted based on ground truth bounding box velocity information, is calculated using the following methods:

[0117] Calculate the absolute value of the error between the predicted velocity output by the model and the corresponding ground truth bounding box velocity information;

[0118] The absolute values ​​corresponding to multiple predicted velocities are summed, and the summation result is used as the second loss value.

[0119] Step 1032: Use the weighted sum of the first loss value and the second loss value as the velocity regression training loss value;

[0120] Velocity regression training loss value L vel =W1L vel1 +W2L vel2 Among them, L vel1 W1 is the weight corresponding to the second loss value, and L is the second loss value. vel2W1 is the first loss value, and W2 is the weight corresponding to the first loss value. In the early stage of training, W1 can be set larger to ensure faster convergence. In the later stage of training, W2 can be increased to ensure the accuracy of velocity regression.

[0121] Step 1033: Adjust the model parameters based on the velocity regression training loss value to obtain a trained millimeter-wave radar target detection model.

[0122] Combining the pre-training and speed training processes, the final loss value of the millimeter-wave radar target detection model is: L det =W cls L cls +W box L box +W vel L vel Among them, W cls For the classification task weights, L cls For the classification task loss; W box To regress task weights, L box Loss due to regression mission; W vel To estimate task weights for speed, L vel This provides a loss for velocity estimation. A well-trained model can output velocity predictions faster and more accurately.

[0123] The millimeter-wave radar target detection model training method provided in this invention learns position information during the pre-training stage and improves velocity prediction speed and accuracy during the velocity regression training stage by using the generated ground truth bounding box velocity information for weakly supervised learning and self-supervised learning based on position estimation. Since millimeter-wave radar point clouds only contain radial velocity information and have large position errors, relying solely on self-labeling or position error supervision leads to slow convergence and low accuracy. The method provided in this invention, which uses hybrid supervised learning without velocity labeling, can achieve over 70% of the velocity regression accuracy with velocity labeling, significantly reducing labeling costs and decreasing the algorithm's dependence on data.

[0124] Figure 4 The flowchart of the millimeter-wave radar target detection method provided in the embodiments of the present invention is as follows: Figure 4 As shown, the millimeter-wave radar target detection method provided in this embodiment of the invention includes:

[0125] Step 401: Acquire real-time millimeter-wave radar point cloud data;

[0126] Step 402: Input the real-time millimeter-wave radar point cloud data into the millimeter-wave radar target detection model with velocity estimation capability to obtain the target detection result. The target detection result includes the target's position and velocity. The millimeter-wave radar target detection model is trained based on the millimeter-wave radar target detection model training method described in any of the above embodiments.

[0127] In this embodiment of the invention, by inputting millimeter-wave radar point cloud data into a trained millimeter-wave radar target detection model, target detection results are obtained. The target detection results include at least one of the position, shape, and velocity of the target box. This can reduce model training costs, improve model convergence speed, and make the estimated velocity more accurate, thus meeting user requirements.

[0128] The millimeter-wave radar target detection model training device provided by the present invention will be described below. The millimeter-wave radar target detection model training device described below can be referred to in correspondence with the millimeter-wave radar target detection model training method described above.

[0129] Figure 5 This is a schematic diagram of a millimeter-wave radar target detection model training device provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the millimeter-wave radar target detection model training device provided in this embodiment of the invention includes:

[0130] The pre-training module 501 is used to pre-train the millimeter-wave radar target detection model based on continuous multi-frame millimeter-wave radar point cloud data and 3D bounding box annotation information, so as to obtain the pre-trained millimeter-wave radar target detection model.

[0131] The generation module 502 is used to generate GT box velocity information based on multiple consecutive frames of millimeter-wave radar point cloud data;

[0132] The velocity regression training module 503 is used to perform velocity regression training on the pre-trained millimeter-wave radar target detection model based on the target bounding box predicted position information and ground truth bounding box velocity information output by the pre-trained millimeter-wave radar target detection model, thereby obtaining a millimeter-wave radar target detection model with velocity estimation capabilities.

[0133] The millimeter-wave radar target detection model training device provided in this embodiment of the invention pre-trains the millimeter-wave radar target detection model based on multiple consecutive frames of millimeter-wave radar point cloud data and 3D bounding box annotation information to obtain a pre-trained millimeter-wave radar target detection model; generates ground truth (GT) bounding box velocity information based on multiple consecutive frames of millimeter-wave radar point cloud data; and performs velocity regression training on the pre-trained millimeter-wave radar target detection model based on the target box predicted position information and GT bounding box velocity information output by the pre-trained millimeter-wave radar target detection model to obtain a millimeter-wave radar target detection model with velocity estimation capability. Through pre-training and velocity regression training, accurate velocity prediction information can be obtained, meeting user needs. Furthermore, by automatically generating GT bounding box velocity information, the slow convergence problem caused by relying solely on self-supervised learning can be avoided, further improving the velocity estimation accuracy.

[0134] The millimeter-wave radar target detection device provided by the present invention is described below. The millimeter-wave radar target detection device described below can be referred to in correspondence with the millimeter-wave radar target detection method described above.

[0135] Figure 6 This is a schematic diagram of a millimeter-wave radar target detection device provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the millimeter-wave radar target detection device provided in this embodiment of the invention includes:

[0136] The acquisition module 601 is used to acquire real-time millimeter-wave radar point cloud data;

[0137] The detection module 602 is used to input real-time millimeter-wave radar point cloud data into a millimeter-wave radar target detection model with velocity estimation capability to obtain target detection results. The target detection results include the position and velocity of the target. The millimeter-wave radar target detection model is trained based on any of the millimeter-wave radar target detection model training methods described above.

[0138] The millimeter-wave radar target detection device provided in this embodiment of the invention obtains target detection results by inputting millimeter-wave radar point cloud data into a trained millimeter-wave radar target detection model. The target detection results include at least one of the target box position, shape, and velocity. This can reduce model training costs, improve model convergence speed, and estimate the velocity more accurately, thus meeting user requirements.

[0139] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logic instructions in the memory 730 to execute a millimeter-wave radar target detection model training method. This method includes: pre-training the millimeter-wave radar target detection model based on multiple consecutive frames of millimeter-wave radar point cloud data and 3D bounding box annotation information to obtain a pre-trained millimeter-wave radar target detection model; generating ground truth (GT) bounding box velocity information based on multiple consecutive frames of millimeter-wave radar point cloud data; performing velocity regression training on the pre-trained millimeter-wave radar target detection model based on the target bounding box predicted position information and GT bounding box velocity information output by the pre-trained millimeter-wave radar target detection model to obtain a millimeter-wave radar target detection model with velocity estimation capability; and / or executing a millimeter-wave radar target detection method, which includes: acquiring real-time millimeter-wave radar point cloud data; inputting the real-time millimeter-wave radar point cloud data into the millimeter-wave radar target detection model with velocity estimation capability to obtain a target detection result, wherein the target detection result includes the target's position and velocity, and the millimeter-wave radar target detection model is trained based on the millimeter-wave radar target detection model training method described in any of the above embodiments.

[0140] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0142] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a millimeter-wave radar target detection model, characterized in that, include: Based on continuous multi-frame millimeter-wave radar point cloud data and 3D bounding box annotation information, the millimeter-wave radar target detection model is pre-trained to obtain the pre-trained millimeter-wave radar target detection model. Generate GT frame velocity information based on the continuous multi-frame millimeter-wave radar point cloud data; Based on the target bounding box predicted position information and the ground truth bounding box velocity information output by the pre-trained millimeter-wave radar target detection model, velocity regression training is performed on the pre-trained millimeter-wave radar target detection model to obtain a millimeter-wave radar target detection model with velocity estimation capability. The step of modifying the pre-trained millimeter-wave radar target detection model based on the target bounding box predicted position information and the ground truth bounding box velocity information output by the pre-trained millimeter-wave radar target detection model includes: Calculate the first loss value based on the predicted location information of the target box and the second loss value based on the predicted velocity information of the ground truth box, respectively. The weighted sum of the first loss value and the second loss value is used as the velocity regression training loss value; The model parameters are adjusted based on the velocity regression training loss value to obtain a trained millimeter-wave radar target detection model.

2. The millimeter-wave radar target detection model training method according to claim 1, characterized in that, The pre-training of the millimeter-wave radar target detection model includes: Coarse feature extraction is performed on the continuous multi-frame millimeter-wave radar point cloud data to obtain multi-frame coarse feature information from a bird's-eye view. Multi-dimensional feature information is obtained by superimposing the coarse feature information of the multiple frames and then performing feature learning. The predicted target bounding box information is obtained based on the multidimensional feature information; Calculate the pre-training loss value based on the predicted target box information and the preset standard target box information; The model parameters are adjusted based on the pre-training loss value to obtain a pre-trained millimeter-wave radar target detection model.

3. The millimeter-wave radar target detection model training method according to claim 2, characterized in that, The predicted target bounding box information obtained based on the multidimensional feature information includes: Create a Doppler velocity map from a bird's-eye view, the Doppler velocity map comprising multiple grids; Projecting multiple frames of point cloud data from the continuous multi-frame millimeter-wave radar point cloud data onto the Doppler velocity map; The maximum Doppler velocity value in each of the multiple grids is extracted as the Doppler velocity feature information of the corresponding grid. The multidimensional feature information is concatenated with the Doppler velocity feature information of each grid to obtain the concatenation result; The predicted target box information is obtained based on the concatenation result.

4. The millimeter-wave radar target detection model training method according to claim 1, characterized in that, The step of generating ground truth (GT) bounding box velocity information based on the continuous multi-frame millimeter-wave radar point cloud data includes: Foreground point segmentation is performed on the continuous multi-frame millimeter-wave radar point cloud data to obtain point cloud clusters belonging to each ground truth (GT) box; The Doppler velocity of each point cloud is calculated based on the point cloud azimuth angle and the GT frame orientation angle in the point cloud cluster. Outlier point clouds are removed based on the difference between the Doppler velocity of each point cloud and the median Doppler velocity of all point clouds in the point cloud cluster. The GT box velocity information is obtained by considering the relationship between the Doppler velocity and the target box position in the point cloud cluster after removing outliers.

5. The millimeter-wave radar target detection model training method according to claim 1, characterized in that, The calculation of the first loss value based on the target predicted location includes: Input the point cloud data at time t-1 from multiple consecutive frames of millimeter-wave radar point cloud data into the pre-trained millimeter-wave radar target detection model to obtain the predicted position of the target box at time t-1. Based on the vehicle's pose change from time t-1 to time t, motion compensation is performed on the predicted position of the target box at time t-1. The target position at time t is calculated based on the predicted position and velocity of the target box at time t-1 after motion compensation, thus obtaining the predicted position of the target box at time t. The predicted location of the target box at time t is matched with the 3D bounding box annotation information at time t to obtain multiple matching pairs; The average distance between the center points of multiple matching pairs is used as the first loss value.

6. A millimeter-wave radar target detection method, characterized in that, include: Acquire real-time millimeter-wave radar point cloud data; The real-time millimeter-wave radar point cloud data is input into a millimeter-wave radar target detection model with velocity estimation capability to obtain target detection results. The target detection results include the target's position and velocity. The millimeter-wave radar target detection model is trained based on the millimeter-wave radar target detection model training method according to any one of claims 1 to 5.

7. A millimeter-wave radar target detection model training device, characterized in that, include: The pre-training module is used to pre-train the millimeter-wave radar target detection model based on multiple consecutive frames of millimeter-wave radar point cloud data and 3D bounding box annotation information, so as to obtain the pre-trained millimeter-wave radar target detection model. The generation module is used to generate GT frame velocity information based on the continuous multi-frame millimeter-wave radar point cloud data; The velocity regression training module is used to perform velocity regression training on the pre-trained millimeter-wave radar target detection model based on the target box prediction position information and the GT box velocity information output by the pre-trained millimeter-wave radar target detection model, so as to obtain a millimeter-wave radar target detection model with velocity estimation capability. The step of modifying the pre-trained millimeter-wave radar target detection model based on the target bounding box predicted position information and the ground truth bounding box velocity information output by the pre-trained millimeter-wave radar target detection model includes: Calculate the first loss value based on the predicted location information of the target box and the second loss value based on the predicted velocity information of the ground truth box, respectively. The weighted sum of the first loss value and the second loss value is used as the velocity regression training loss value; The model parameters are adjusted based on the velocity regression training loss value to obtain a trained millimeter-wave radar target detection model.

8. A millimeter-wave radar target detection device, characterized in that, include: The acquisition module is used to acquire millimeter-wave radar point cloud data; The detection module is used to input the millimeter-wave radar point cloud data into a millimeter-wave radar target detection model with velocity estimation capability to obtain target detection results. The target detection results include the position and velocity of the target. The millimeter-wave radar target detection model is trained based on the millimeter-wave radar target detection model training method according to any one of claims 1 to 5.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the millimeter-wave radar target detection model training method as described in any one of claims 1 to 5, and / or the millimeter-wave radar target detection method as described in claim 6.

Images