Method for target detection and trajectory prediction based on millimeter wave radar
Through the voxelization processing and feature extraction of 4D millimeter wave radar, the problem that traditional trajectory prediction methods are difficult to deal with complex environments and real-time data is solved, low-cost and efficient trajectory prediction is achieved, and the safety and traffic of the autonomous driving system are improved. efficiency.
Patent Information
- Application Number
- CN202411223948.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-09-03
AI Technical Summary
Traditional trajectory prediction methods are difficult to deal with complex road environments and real-time data, and are costly, so they cannot effectively predict the future trajectory of vehicles and pedestrians, affecting traffic efficiency and safety.
Using a 4D millimeter wave radar method, through voxelization processing and feature extraction, multi-frame data of the vehicle is obtained, target detection and trajectory prediction are carried out, dynamic prediction of future trajectories, and dependence on high-precision maps and historical trajectories is reduced. .
It realizes accurate prediction of the future trajectory of vehicles and pedestrians under low-cost conditions, improves driving safety and traffic efficiency of the autonomous driving system, and is suitable for all-weather environments.
Smart Images

Figure CN118731893B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of millimeter wave radar technology, and in particular to the field of trajectory prediction, and specifically refers to a method, device, processor and computer-readable storage medium thereof for realizing target detection and trajectory prediction based on millimeter wave radar. Background Art
[0002] Trajectory prediction is a key technology in the fields of intelligent transportation systems, self-driving cars, robot navigation, etc. With the increase in the number of vehicles and pedestrians, traffic congestion, frequent accidents and other problems are becoming increasingly serious. In order to improve traffic efficiency and safety, it is necessary to predict the trajectories of vehicles and pedestrians in order to carry out reasonable traffic planning and control.
[0003] Traditional trajectory prediction methods mainly rely on physical models and statistical models. The physical model predicts the future position of the vehicle by building a vehicle dynamics model. However, this method requires a large number of vehicle dynamics parameters and has difficulty handling complex road environments and traffic conditions. The statistical model predicts future trajectories by analyzing historical data and building statistical models of vehicle and pedestrian behavior. However, this method has difficulty handling real-time data and emergencies.
[0004] In recent years, with the rapid development of artificial intelligence technology, trajectory prediction methods based on deep learning have been widely used. Deep learning has powerful feature extraction and pattern recognition capabilities, and can handle complex data and real-time changes. Trajectory prediction methods based on deep learning mainly include the following:
[0005] Trajectory prediction based on convolutional neural network (CNN), trajectory prediction based on recurrent neural network (RNN), trajectory prediction based on attention mechanism, and trajectory prediction based on multimodal fusion. Among them, multimodal fusion can combine data of different modalities to improve the accuracy of prediction. This method is suitable for processing multiple sensor data, such as lidar and camera data. And trajectory prediction contains high-precision map information or map information acquisition module, which is very expensive.
[0006] The point cloud of 4D imaging millimeter-wave radar is denser than that of 3D millimeter-wave radar and lower in cost than that of lidar. The fusion solution of 4D millimeter-wave and vision is seen as the next generation of autonomous driving perception fusion solution, which will have great advantages in cost and performance. Summary of the invention
[0007] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a method, device, processor and computer-readable storage medium for realizing target detection and trajectory prediction based on millimeter-wave radar.
[0008] In order to achieve the above objectives, the method, device, processor and computer-readable storage medium for target detection and trajectory prediction based on millimeter wave radar of the present invention are as follows:
[0009] The method for realizing target detection and trajectory prediction based on millimeter wave radar has the following main features:
[0010] (1) Obtain multi-frame 4D millimeter-wave radar data of the current vehicle and perform data preprocessing;
[0011] (2) Using voxelization to extract features from preprocessed data;
[0012] (3) The acquired BEV features are sent to the trajectory prediction detection module to obtain the position information of the current vehicle's future multi-frame targets;
[0013] (4) The acquired multi-frame position information is used to perform trajectory association to obtain the future trajectory, thereby realizing the prediction of the future trajectory of the target vehicle.
[0014] Preferably, the step (1) comprises the following steps:
[0015] (1.1) Dividing the multi-frame 4D millimeter-wave radar data acquired within a preset time interval into a past frame point cloud and a current frame point cloud;
[0016] (1.2) Perform data cropping on multiple frames of 4D millimeter-wave radar data to extract important features.
[0017] Preferably, the step (1.2) is specifically to extract the point cloud features after filtering in the following manner: :
[0018] ;
[0019] in, is the position coordinate of the i-th point cloud data in each frame, is the radial velocity is the point cloud velocity after compensation, is the characteristic power value, is the signal-to-noise ratio, Point is the original point cloud, The maximum and minimum values in the x and y range of the retained point cloud are respectively.
[0020] Preferably, the step (2) specifically includes the following steps:
[0021] (2.1) Convert the past frame point cloud and the current frame point cloud into voxelized grids according to the preset voxel size and detection distance;
[0022] (2.2) The voxelized grid features are sent to the feature extraction module for feature extraction to obtain the current frame bev features And the past frame bev features .
[0023] Preferably, the step (3) specifically comprises the following steps:
[0024] (3.1) Input the first detection head of the trajectory prediction detection module as the current frame bev feature After detection processing, the confidence cls, position xyz, heading angle rot and speed information vel are output, and the current detection head features are output after convolution processing. , as follows:
[0025] ;
[0026] (3.2) When the detection process is performed by the i-th detection head, the bev feature of the past frame And the output feature of the i-1th detection head , as follows:
[0027] .
[0028] Preferably, the step (4) is specifically as follows:
[0029] According to the obtained confidence information and the position information, shape information and heading angle information of the detection frame, the detection frame of the current frame is associated and matched with the detection frame of the future frame, and a complete trajectory prediction path is obtained based on the matching path.
[0030] Preferably, the step (4) specifically includes the following steps:
[0031] (4.1) Filter out low-confidence detection boxes based on the confidence threshold in the following way: :
[0032] ;
[0033] in, is the original detection box set, is the confidence threshold, is the heading angle of the i-th point cloud, is the velocity information of the i-th point cloud, is the confidence of the i-th point cloud;
[0034] (4.2) In the detection frame In the t-th frame, and the detection box of frame t+1 , calculate the Euclidean distance between the two center points as follows :
[0035] ;
[0036] in, is the position coordinate of the i-th point cloud at the t-th frame, is the position coordinate of the jth point cloud at the t+1th frame;
[0037] (4.3) For each detection box in the tth frame , traverse all detection boxes in the future frame t+1 and find the index value that meets the following conditions :
[0038] ;
[0039] in, is the detection box index value that meets the above conditions. is the distance threshold, if it exceeds this threshold, the detection boxes are considered not to be the same target;
[0040] (4.4) If the above conditions are met, then it is considered that and If the match is successful, after all detection frames of frame t have been traversed, the unmatched detection frames are deleted and the above process is repeated until all frames are processed.
[0041] The device for realizing target detection and trajectory prediction based on millimeter wave radar has the following main features:
[0042] a processor configured to execute computer-executable instructions;
[0043] The memory stores one or more computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the method for realizing target detection and trajectory prediction based on millimeter wave radar are implemented.
[0044] The main feature of the processor for realizing target detection and trajectory prediction based on millimeter-wave radar is that the processor is configured to execute computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method for realizing target detection and trajectory prediction based on millimeter-wave radar are implemented.
[0045] The main feature of the computer-readable storage medium is that a computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method for realizing target detection and trajectory prediction based on millimeter-wave radar as described above.
[0046] The method, device, processor and computer-readable storage medium for realizing target detection and trajectory prediction based on millimeter-wave radar of the present invention are adopted, which do not rely on high-precision maps and historical trajectories. The trajectory prediction of dynamic objects can be completed by inputting multiple frames of historical point clouds, which can be used for planning and control of vehicles in subsequent autonomous driving. At the same time, compared with laser radar, millimeter-wave radar has extremely low cost and great advantages in scheme design. It can effectively ensure the functional stability of all weather conditions such as daytime, nighttime and bad weather. The technical scheme can effectively predict the future trajectory of vehicles or pedestrians, thereby helping autonomous driving to plan the driving trajectory of the vehicle and improve driving safety. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 The present invention is a process flow chart of a method for realizing target detection and trajectory prediction based on millimeter wave radar. DETAILED DESCRIPTION
[0048] In order to more clearly describe the technical content of the present invention, further description is given below in conjunction with specific embodiments.
[0049] Before describing in detail embodiments according to the present invention, it should be noted that, hereinafter, the terms "comprises", "includes" or any other variations are intended to cover non-exclusive inclusion, whereby a process, method, article or apparatus comprising a series of elements includes not only these elements, but also other elements not explicitly listed or inherent to such process, method, article or apparatus.
[0050] See also Figure 1 As shown, the method for realizing target detection and trajectory prediction based on millimeter wave radar, wherein the method comprises the following steps:
[0051] (1) Obtain multi-frame 4D millimeter-wave radar data of the current vehicle and perform data preprocessing;
[0052] (2) Using voxelization to extract features from preprocessed data;
[0053] (3) The acquired BEV features are sent to the trajectory prediction detection module to obtain the position information of the current vehicle's future multi-frame targets;
[0054] (4) The acquired multi-frame position information is used to perform trajectory association to obtain the future trajectory, thereby realizing the prediction of the future trajectory of the target vehicle.
[0055] As a preferred embodiment of the present invention, the step (1) comprises the following steps:
[0056] (1.1) Dividing the multi-frame 4D millimeter-wave radar data acquired within a preset time interval into a past frame point cloud and a current frame point cloud;
[0057] (1.2) Perform data cropping on multiple frames of 4D millimeter-wave radar data to extract important features.
[0058] As a preferred embodiment of the present invention, the step (1.2) is specifically to extract the point cloud features after filtering in the following manner: :
[0059] ;
[0060] in, is the position coordinate of the i-th point cloud data in each frame, is the radial velocity is the point cloud velocity after compensation, is the characteristic power value, is the signal-to-noise ratio, Point is the original point cloud, The maximum and minimum values in the x and y range of the retained point cloud are respectively.
[0061] As a preferred embodiment of the present invention, the step (2) specifically includes the following steps:
[0062] (2.1) Convert the past frame point cloud and the current frame point cloud into voxelized grids according to the preset voxel size and detection distance;
[0063] (2.2) The voxelized grid features are sent to the feature extraction module for feature extraction to obtain the current frame bev features And the past frame bev features .
[0064] As a preferred embodiment of the present invention, the step (3) specifically includes the following steps:
[0065] (3.1) Input the first detection head of the trajectory prediction detection module as the current frame bev feature After detection processing, the confidence cls, position xyz, heading angle rot and speed information vel are output, and the current detection head features are output after convolution processing. , as follows:
[0066] ;
[0067] (3.2) When the detection process is performed by the i-th detection head, the bev feature of the past frame And the output feature of the i-1th detection head , as follows:
[0068] .
[0069] As a preferred embodiment of the present invention, the step (4) is specifically as follows:
[0070] According to the obtained confidence information and the position information, shape information and heading angle information of the detection frame, the detection frame of the current frame is associated and matched with the detection frame of the future frame, and a complete trajectory prediction path is obtained based on the matching path.
[0071] As a preferred embodiment of the present invention, the step (4) specifically includes the following steps:
[0072] (4.1) Filter out low-confidence detection boxes based on the confidence threshold in the following way: :
[0073] ;
[0074] in, is the original detection box set, is the confidence threshold, is the heading angle of the i-th point cloud, is the velocity information of the i-th point cloud, is the confidence of the i-th point cloud;
[0075] (4.2) In the detection frame In the t-th frame, and the detection box of frame t+1 , calculate the Euclidean distance between the two center points as follows :
[0076] ;
[0077] in, is the position coordinate of the i-th point cloud at the t-th frame, is the position coordinate of the jth point cloud at the t+1th frame;
[0078] (4.3) For each detection box in the tth frame , traverse all detection boxes in the future frame t+1 and find the index value that meets the following conditions :
[0079] ;
[0080] in, is the detection box index value that meets the above conditions. is the distance threshold, if it exceeds this threshold, the detection boxes are considered not to be the same target;
[0081] (4.4) If the above conditions are met, then it is considered that and If the match is successful, after all detection frames of frame t have been traversed, the unmatched detection frames are deleted and the above process is repeated until all frames are processed.
[0082] In a specific embodiment of the present invention, the method for realizing target detection and trajectory prediction based on millimeter wave radar has the following specific implementation steps:
[0083] Wherein, step 1) is specifically:
[0084] Select 11 frames of millimeter-wave radar point cloud data with a time interval of 500ms, of which the first ten frames are used as the past frame point cloud and the last frame is used as the current frame point cloud. The point cloud data is cropped to extract important features. The preprocessing of each frame of point cloud data is as follows:
[0085] ;
[0086] in, , , , , , , are the position coordinates, radial velocity, point cloud velocity after compensation, characteristic power value and signal-to-noise ratio of the i-th point cloud data in each frame, Point is the original point cloud, The maximum and minimum values of the xy range of the retained point cloud are respectively, is the filtered point cloud feature.
[0087] Convert the past frame millimeter wave radar point cloud data to the current frame coordinate system. For each frame of point cloud data, the conversion process is as follows:
[0088] ;
[0089] in, are the position information of the millimeter-wave radar point cloud data of the past i-th frame converted to the point cloud of the current frame coordinate system, and is the rotation matrix and translation matrix of the millimeter-wave radar point cloud data of the past i-th frame converted to the current frame coordinate system, is the position information of the millimeter-wave radar point cloud of the past i-th frame.
[0090] Step 2) is specifically:
[0091] The past frame point cloud and the current frame point cloud are voxelized. Taking the current frame point cloud as an example, it is converted into a voxelized grid according to the set voxel size and detection distance, where the height direction grid size is constant to 1. The voxelized grid features are sent to the feature extraction module for feature extraction. The feature extraction module includes resnet18 and an upsampling module. First, the grid features are downsampled through the resnet18 network, and the features are downsampled by 2 times, 4 times, and 8 times respectively to obtain the downsampled features. The formula is as follows:
[0092] ;
[0093] in , is the feature after downsampling by 1, 2, 4, and 8 times, Radar_Voxel_Feature is the millimeter-wave point cloud feature after voxelization, and Resnet18 is the convolutional neural network model.
[0094] Next, After upsampling, After channel splicing, convolution is used to keep the feature channels consistent, and the first upsampling result is obtained. :
[0095] ;
[0096] Among them, upasmple is the upsampling operation, cat is the data splicing operation, and conv is the convolution operation, including convolution, normalization and activation function.
[0097] Will Upsampling and After splicing along the channel, convolution is used to keep the feature channel consistent, and the second upsampling result is obtained. :
[0098] ;
[0099] Will After upsampling and splicing along the channel of F, convolution is used to keep the feature channel consistent, and the second upsampling result is obtained. :
[0100] ;
[0101] Finally, After upsampling And the bev feature of the current frame extracted after Radar_Voxel_Feature splicing and convolution output :
[0102] ;
[0103] Similarly, the past frame point cloud is processed and output as the past frame bev feature in the above manner .
[0104] Step 3) is specifically:
[0105] To predict the target point attributes of multiple future frames, the bev features of the current frame obtained in 2) are and past frame bev features Multiple detection heads are sent to predict the confidence, position, heading angle and speed information of the target in multiple frames.
[0106] 1. Centerhead detection head The first detection head input is the bev feature of the current frame After the head, the confidence cls, position xyz, heading angle rot and speed information vel are output, and the current detection head features are output after convolution. :
[0107] ;
[0108] 2. If it is the i-th detection head, the bev feature of the past frame is required And the output characteristics of the i-1th detection head :
[0109] ;
[0110] Wherein, step 4) is specifically:
[0111] The confidence, position, heading angle and speed information obtained in 3) are processed according to the confidence and the position information, shape information and heading angle information of the detection frame, and the detection frame of the current frame is matched with the detection frame of the future frame, and finally matched into a complete trajectory.
[0112] First, filter out detection boxes with low confidence based on the confidence threshold:
[0113] ;
[0114] in, is the original detection box set, is the confidence threshold.
[0115] exist In the t-th frame, and the detection box of frame t+1 , calculate the Euclidean distance between their center points:
[0116] ;
[0117] For each detection box in the tth frame , traverse all detection boxes in the future frame t+1 and find the index value that meets the following conditions :
[0118] ;
[0119] in, is the index value of the detection box that meets the above conditions. is the Euclidean distance between the i-th detection box in the t-th frame and the j-th detection box in the t+1-th frame. is the distance threshold. If the upper limit is exceeded, the detection boxes are considered not to be the same target.
[0120] If the above conditions are met, it is considered and If the match is successful, after all detection frames of frame t have been traversed, the unmatched detection frames are deleted and the above process is repeated until all frames are processed.
[0121] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.
[0122] It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device.
[0123] A person of ordinary skill in the art may understand that all or part of the steps of the method for implementing the above-mentioned embodiment may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0124] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.
[0125] In the description of this specification, the description with reference to the terms "an embodiment", "some embodiments", "example", "specific example", or "embodiment" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0126] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.
[0127] The method, device, processor and computer-readable storage medium for realizing target detection and trajectory prediction based on millimeter-wave radar of the present invention are adopted, which do not rely on high-precision maps and historical trajectories. The trajectory prediction of dynamic objects can be completed by inputting multiple frames of historical point clouds, which can be used for planning and control of vehicles in subsequent autonomous driving. At the same time, compared with laser radar, millimeter-wave radar has extremely low cost and great advantages in scheme design. It can effectively ensure the functional stability of all weather conditions such as daytime, nighttime and bad weather. The technical scheme can effectively predict the future trajectory of vehicles or pedestrians, thereby helping autonomous driving to plan the driving trajectory of the vehicle and improve driving safety.
[0128] In this specification, the present invention has been described with reference to specific embodiments thereof. However, it is apparent that various modifications and variations may be made without departing from the spirit and scope of the present invention. Therefore, the specification and drawings should be regarded as illustrative rather than restrictive.
Claims
1. A method for target detection and trajectory prediction based on millimeter wave radar, It is characterized in that The method comprises the following steps: (1) Obtain multi-frame 4D millimeter-wave radar data of the current vehicle and perform data preprocessing; (2) Using voxelization to extract features from preprocessed data; (3) The acquired BEV features are sent to the trajectory prediction detection module to obtain the position information of the current vehicle's future multi-frame targets; (4) Correlate the acquired multi-frame position information to obtain the future trajectory, thereby predicting the future trajectory of the target vehicle; The step (1) comprises the following steps: (1.1) Dividing the multi-frame 4D millimeter-wave radar data acquired within a preset time interval into a past frame point cloud and a current frame point cloud; (1.2) Perform data cropping on multiple frames of 4D millimeter-wave radar data to extract important features; The step (2) specifically includes the following steps: (2.1) Convert the past frame point cloud and the current frame point cloud into voxelized grids according to the preset voxel size and detection distance; (2.2) The voxelized grid features are sent to the feature extraction module for feature extraction to obtain the current frame bev features And the past frame bev features ; The step (3) specifically includes the following steps: (3.1) Input the first detection head of the trajectory prediction detection module as the current frame bev feature After detection processing, the confidence cls, position xyz, heading angle rot and speed information vel are output, and the current detection head features are output after convolution processing. , as follows: ; (3.2) When the detection process is performed by the i-th detection head, the bev feature of the past frame And the output feature of the i-1th detection head , as follows: 。 2. The method for target detection and trajectory prediction based on millimeter wave radar according to claim 1, It is characterized in that The step (1.2) is specifically to extract the point cloud features after filtering in the following way: : ; in, is the position coordinate of the i-th point cloud data in each frame, is the radial velocity is the point cloud velocity after compensation, is the characteristic power value, is the signal-to-noise ratio, Point is the original point cloud, The maximum and minimum values in the x and y range of the retained point cloud are respectively.
3. The method for target detection and trajectory prediction based on millimeter wave radar according to claim 1, It is characterized in that The step (4) is specifically as follows: According to the obtained confidence information and the position information, shape information and heading angle information of the detection frame, the detection frame of the current frame is associated and matched with the detection frame of the future frame, and a complete trajectory prediction path is obtained based on the matching path.
4. The method for target detection and trajectory prediction based on millimeter wave radar according to claim 1, It is characterized in that The step (4) specifically includes the following steps: (4.1) Filter out low-confidence detection boxes based on the confidence threshold in the following way: : ; in, is the original detection box set, is the confidence threshold, is the heading angle of the i-th point cloud, is the velocity information of the i-th point cloud, is the confidence of the i-th point cloud; (4.2) In the detection frame For the i-th detection box in the t-th frame and the jth detection box of the t+1th frame , calculate the Euclidean distance between the two center points as follows : ; in, is the position coordinate of the i-th point cloud at the t-th frame, is the position coordinate of the jth point cloud at the t+1th frame; (4.3) For each detection box in the tth frame , traverse all detection boxes in the future frame t+1 and find the index value that meets the following conditions : ; in, is the index value of the detection box that meets the above conditions. is the distance threshold, if it exceeds this threshold, the detection boxes are considered not to be the same target; (4.4) If the above conditions are met, then it is considered that and If the match is successful, after all detection frames of frame t have been traversed, the unmatched detection frames are deleted and the above process is repeated until all frames are processed.
5. A device for target detection and trajectory prediction based on millimeter wave radar, It is characterized in that The device comprises: a processor configured to execute computer executable instructions; A memory storing one or more computer executable instructions, wherein when the computer executable instructions are executed by the processor, the steps of the method for realizing target detection and trajectory prediction based on millimeter wave radar according to any one of claims 1 to 4 are implemented.
6. A processor for target detection and trajectory prediction based on millimeter-wave radar, It is characterized in that The processor is configured to execute computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the method for realizing target detection and trajectory prediction based on millimeter wave radar described in any one of claims 1 to 4 are implemented.
7. A computer-readable storage medium, It is characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method for realizing target detection and trajectory prediction based on millimeter-wave radar as described in any one of claims 1 to 4.
Citation Information
Patent Citations
Intelligent traffic cloud management platform of HDMap and 5G vehicle and road cooperation
CN116469249A
Perception fusion system, electronic equipment and storage medium
CN116664997A
End-to-end automatic driving method, system, equipment and medium
CN118306419A
Joint perception prediction method for automatic driving motorcade in high-speed scene
CN118411701A
End-to-end systems and methods for streaming 3D detection and forecasting from lidar point clouds
US20230290153A1
