A Data Augmentation Method for 3D Target Detection with FMCW LiDAR

By constructing an FMCW lidar point cloud 3D target detection dataset through Doppler velocity information, the problem of insufficient existing data is solved, richer data support is achieved, and the training and optimization effect of autonomous driving system is improved.

CN117348020BActive Publication Date: 2026-11-17ZHEJIANG UNIV +2
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311037111.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-08-17
Publication Date
2026-11-17
Estimated Expiration
2043-08-17

AI Technical Summary

Technical Problem

The lack of FMCW LiDAR point cloud 3D target detection datasets in existing technologies results in insufficient data to support effective training and optimization of autonomous driving systems.

Method used

By calculating Doppler velocity information, a 3D target detection dataset containing Doppler velocities is constructed, including the Doppler velocities of target objects and background points, and data augmentation is performed using FMCW lidar point cloud data.

Benefits of technology

It expands the existing publicly available 3D object detection dataset, provides rich Doppler velocity information, fills the gap in the FMCW lidar point cloud 3D object detection dataset, and supports more effective training and optimization of autonomous driving systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117348020B_ABST
    Figure CN117348020B_ABST
Patent Text Reader

Abstract

This invention discloses a data augmentation method for 3D target detection using an FMCW LiDAR system. The method transforms the target point cloud coordinates and the target object's orientation rotation angle from the camera coordinate system to the radar coordinate system; initializes the motion velocities of the radar and the target object in each frame of data, thus obtaining the velocity of each point cloud; calculates the radial velocities of the radar and the target object point clouds; calculates the Doppler velocities of stationary background points and moving target object points respectively; and merges the data information of background points and target object points to obtain a 3D target detection dataset with Doppler velocities. This invention expands upon existing publicly available 3D target detection datasets with Doppler velocity information to address the current lack of open-source FMCW LiDAR target detection datasets. The data augmentation method of this invention is applicable to open-source real-world LiDAR 3D target detection datasets to obtain experimental FMCW LiDAR target detection data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to a data augmentation method in the field of 3D target detection, specifically a data augmentation method for 3D target detection using an FMCW lidar. Background Technology

[0002] Currently, most publicly available LiDAR point cloud 3D target detection datasets in the field of autonomous driving are obtained from LiDAR data collected using Time-of-Flight (ToF) measurement methods, such as the KITTI, Waymo, and Nuscenes datasets. There is no publicly available FMCW (Free-Moving-Wide) LiDAR point cloud 3D target detection dataset. This invention, based on publicly available datasets and utilizing the principle of Doppler velocity acquisition, calculates the Doppler velocity information of the point cloud and proposes an FMCW LiDAR 3D target detection data augmentation method, filling the gap in this field. Summary of the Invention

[0003] To address the needs and problems in the background technology, this invention provides a method for enhancing 3D target detection data of FMCW lidar.

[0004] The technical solution of the present invention is as follows:

[0005] Step 1: Obtain the original 3D object detection dataset. If the coordinates of the center point of the 3D object in the original 3D object detection dataset and / or the target rotation angle are in the camera coordinate system, then proceed to Step 2; otherwise, proceed to Step 3.

[0006] Step 2: Transform the coordinates of the center point of the 3D object and / or the target rotation angle in the original 3D target detection dataset from the camera coordinate system to the radar coordinate system;

[0007] Step 3: If the point cloud data in the original 3D object detection dataset contains velocity information, proceed to Step 4; otherwise, initialize the motion velocity of the point cloud data to obtain the velocity information corresponding to the point cloud data.

[0008] Step 4: Based on the current 3D target detection dataset, calculate the horizontal component of each radar velocity and the Doppler velocity of each target object's point cloud;

[0009] Step 5: Calculate the Doppler velocity of the background points based on the horizontal component of each radar velocity, and then construct a 3D target detection dataset containing the Doppler velocity.

[0010] In step three, for each frame of point cloud data, the velocity magnitude of the point cloud corresponding to each target object is randomly set based on the velocity constraint, and the rotation angle of the corresponding target in the original 3D target detection dataset is used as the velocity direction of the point cloud corresponding to each target object.

[0011] In step four, the calculation steps for the Doppler velocity of each target object point cloud and the corresponding radar point cloud are as follows:

[0012] S1: Combining the velocities of the current target object and the corresponding radar, calculate the horizontal components of the target object's velocity on each laser beam and the horizontal components of the radar's velocity, using the following formulas:

[0013]

[0014]

[0015]

[0016] Among them, v pi Let v be the horizontal component of the current target object's velocity on the i-th laser beam of the radar. 0_i Let v be the horizontal component of the velocity on the i-th laser beam of the radar. pi p is the i-th surface scan point of the current target object. i The speed of motion, v 0i Let β be the velocity of the i-th laser beam of the radar. i For point cloud p on the horizontal plane i The angle θ between the velocity direction and the i-th laser beam. i Let x be the angle between the radar's own velocity on the horizontal plane and the i-th laser beam. pi y pi p represents the i-th surface scan point of the current target object. i The x and y coordinates are given, where x0 and y0 are the x and y coordinates of the origin of the radar coordinate system, respectively, and rotation_z is the target rotation angle of the current target object in the radar coordinate system.

[0017] S2: Based on the horizontal components of the target object's velocity on each laser beam and the horizontal component of the radar's velocity, calculate the vertical components of the target object's velocity on each laser beam and the vertical component of the radar's velocity. The calculation formula is as follows:

[0018]

[0019]

[0020] Among them, v pi_dopplerLet v be the vertical component of the current target object's velocity on the i-th laser beam of the radar. 0_i_doppler Let γ be the vertical component of the radar velocity on the i-th laser beam. i Let zi be the angle between the i-th laser beam of the radar and the horizontal plane, and z0 be the z-axis coordinate value of the origin of the radar coordinate system. i p is the i-th surface scan point of the current target object. i The z-axis coordinate value;

[0021] S3: Based on the vertical components of the target object's velocity along each laser beam and the vertical component of the radar velocity, the Doppler velocity of the target object is calculated using the following formula:

[0022] v pi_0_doppler =v pi_doppler -v 0_i_doppler

[0023] Among them, v pi_0_doppler p is the i-th surface scan point of the current target object. i Doppler velocity.

[0024] In step five, the inverse of the horizontal component of the radar velocity corresponding to each background point is taken as its Doppler velocity, as shown in the following formula:

[0025] v background_j_doppler =-v 0_j_doppler

[0026] Among them, v background_j_doppler Let v be the Doppler velocity of the j-th background point. 0_j_doppler Let be the horizontal component of the radar velocity on the radar laser beam where the j-th background point is located.

[0027] In step five, the 3D object detection dataset containing Doppler velocity is composed of the original 3D object detection dataset and the Doppler velocities of each object point cloud and background point.

[0028] The beneficial effects of this invention are:

[0029] This invention augments existing publicly available 3D object detection datasets with Doppler velocity information to obtain FMCW LiDAR object detection experimental data, thus addressing the current issue that the FMCW LiDAR object detection dataset is not yet open source. Attached Figure Description

[0030] Figure 1 This is a flowchart of the method of the present invention.

[0031] Figure 2 This is a three-dimensional spatial diagram.

[0032] Figure 3 This represents the center point of the LiDAR and the target object in the horizontal plane view.

[0033] Figure 4 This represents the component of the velocity along the laser beam direction on the horizontal plane.

[0034] Figure 5 This represents the Doppler velocity in the vertical plane containing the laser beam. Detailed Implementation

[0035] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0036] like Figure 1 As shown, the present invention includes the following steps:

[0037] Step 1: Obtain the original 3D object detection dataset. If the coordinates of the center point of the 3D object in the original 3D object detection dataset and / or the target rotation angle (rotation_y, i.e., the vehicle orientation) are in the camera coordinate system, then proceed to Step 2; otherwise, proceed to Step 3.

[0038] In this embodiment, the 3D Object Detection dataset in KITTI is used as an example. Point cloud data and annotation information are obtained through the 3D Object Detection dataset in KITTI. The point cloud data includes the three-dimensional coordinates of the point cloud in the radar coordinate system. The annotation information includes the target category, the size of the three-dimensional object (height, width, length, i.e., h, w, l), and the coordinates p(x, y, z) of the center point of the three-dimensional object in the camera coordinate system, the target rotation angle (rotation_y, i.e., the vehicle orientation), etc.

[0039] Step 2: Transform the coordinates of the center point of the 3D object and / or the target rotation angle (rotation_y, i.e., the vehicle orientation) in the original 3D object detection dataset from the camera coordinate system to the radar coordinate system, and update the 3D object detection dataset;

[0040] In this embodiment, the coordinates of the center point of the three-dimensional object and the orientation (i.e., rotation angle) of the target object are transformed from the camera coordinate system to the radar coordinate system to obtain the coordinates of the center point of the three-dimensional object and the target rotation angle (rotation_z, i.e., vehicle orientation) in the radar coordinate system.

[0041] Step 3: If the point cloud data in the original 3D object detection dataset contains velocity information, proceed to Step 4; otherwise, initialize the motion velocity of the point cloud data, obtain the velocity information corresponding to the point cloud data, and update the 3D object detection dataset.

[0042] In step three, for each frame of point cloud data, the velocity magnitude of the point cloud corresponding to each target object is randomly set based on the velocity constraint. The rotation angle of the corresponding target in the original 3D target detection dataset is used as the velocity direction of the point cloud corresponding to each target object, which is consistent with the orientation of the target object marked in the label file in the dataset.

[0043] In this embodiment, the target object is categorized into vehicles, pedestrians, and cyclists, satisfying the equation object = car, pedestrian, cyclist, and the target object's speed v. object Satisfying v object =v car ,v pedestrian ,v cyclist According to road speed limits, the following speed restrictions apply:

[0044] ① Maximum speed limit is denoted as: v max_car ,v max_pedestrian ,v max_cyclist The unit is m / s;

[0045] ②The lower limit of velocity is denoted as: μv max_car μv max_pedestrian μv max_cyclist , μ∈[0,1].

[0046] Therefore, the range of the object's velocity is [μv]. max_car v max_car The value is randomly initialized according to a Gaussian distribution.

[0047] Step 4: Based on the current 3D target detection dataset, calculate the horizontal component of each radar velocity and the Doppler velocity (i.e., radial velocity) of each target object's point cloud;

[0048] like Figure 2 As shown, a radar coordinate system XYZ is established with the FMCW LiDAR as the center. The positive X-axis is the direction directly in front of the vehicle where the FMCW LiDAR is located. The FMCW LiDAR coordinates are O(x0,y0,z0), the speed is denoted as v0, the left side of the vehicle is the positive Y-axis, and the positive Z-axis is the direction perpendicular to the ground upwards.

[0049] In step four, the calculation steps for the Doppler velocity of each target object's point cloud and the corresponding radar point cloud are as follows:

[0050] Since the center point of an object is not located on its surface, the center point p(x,y,z) of the target object is generally a virtual point. The real points obtained by scanning the object's surface are denoted as the set P = {p1, p2, p3, ..., p...}. n}

[0051] Assuming all target objects are in rigid motion, meaning that at any given moment: all points obtained by a single lidar scanning a single target object have the same motion state. The velocity of a single target object is denoted as v. p The velocity of all points obtained after the object is scanned by the FMCW lidar is denoted as the set:

[0052] {v p1 ,v p2 ,v p3 ,…,v p_n}, and v p =v p1 =v p2 =…=v pn

[0053] like Figure 3 As shown, due to the different angles and beams of the lidar, the Doppler radial velocities at different points on the same object are mostly not equal. The Doppler velocities at different points on the target object are denoted as the set: {v p1_doppler ,v p2_doppler ,…,v pn_doppler The unit is m / s. The coordinates of each point are marked as follows:

[0054] {(x p1 ,y p1 ,z p1 ),(x p2 ,y p2 ,z p2 ),…,(x pn ,y pn ,z pn )}

[0055] Since the object is in rigid body motion, the motion (direction and speed) of each point on the same object is consistent, so the size of rotation_z remains unchanged.

[0056] S1: On the horizontal plane, the angle between the target object's velocity and the laser beam is denoted as the horizontal deflection angle, represented by the symbol β; the angle between the radar's own velocity and the laser beam is represented by the symbol θ. β can be calculated from the target rotation angle (i.e., vehicle orientation) rotation_z in the radar coordinate system. Combining the current target object's velocity and the corresponding radar's velocity, the horizontal components of the current target object's velocity on each laser beam and the horizontal components of the radar's velocity are calculated, such as... Figure 4 As shown, the calculation formula is as follows:

[0057]

[0058]

[0059]

[0060] Among them, v pi Let v be the horizontal component of the current target object's velocity on the i-th laser beam of the radar. 0_i Let v be the horizontal component of the velocity on the i-th laser beam of the radar. pi p is the i-th surface scan point of the current target object. i The velocity of the movement is parallel to the ground horizontally and parallel to the length l of the 3D Box in the dataset's label vertically. 0i Let β be the velocity of the i-th laser beam of the radar. i For point cloud p on the horizontal plane i The angle between the velocity direction and the i-th laser beam, i.e., the horizontal deflection angle, θ i Let x be the angle between the radar's own velocity on the horizontal plane and the i-th laser beam. pi y pi p represents the i-th surface scan point of the current target object. i The x and y coordinates are given, where x0 and y0 are the x and y coordinates of the origin of the radar coordinate system, respectively, and rotation_z is the target rotation angle of the current target object in the radar coordinate system.

[0061] S2: Based on the horizontal components of the current target object's velocity on each laser beam and the horizontal component of the radar velocity, such as... Figure 5 As shown, due to the existence of the vertical field of view of the lidar, it is necessary to further calculate v in the vertical plane. 0|| and v pi|| The component v on the laser beam 0_i_doppler and v pi_doppler The angle between the laser beam and the horizontal plane is denoted as γ. The vertical components of the target object's velocity along each laser beam and the vertical component of the radar velocity are then calculated using the following formula:

[0062]

[0063]

[0064] Among them, v pi_doppler Let v be the vertical component of the current target object's velocity on the i-th laser beam of the radar. 0_i_doppler Let γ be the vertical component of the radar's velocity on the i-th laser beam. i Let zi be the angle between the i-th laser beam of the radar and the horizontal plane, and z0 be the z-axis coordinate value of the origin of the radar coordinate system. i p is the i-th surface scan point of the current target object. i The z-axis coordinate value;

[0065] S3: Based on the vertical components of the target object's velocity along each laser beam and the vertical component of the radar velocity, the Doppler velocity of the target object is calculated using the following formula:

[0066] v pi_0_doppler =v pi_doppler -v 0_i_doppler

[0067] Among them, v pi_0_doppler p is the i-th surface scan point of the current target object. i The Doppler velocity, i.e., the point p scanned on the surface of the target object. i The velocity component of the FMCW LiDAR on this harness.

[0068] Similarly, other points {p1, p2, p3, ..., p} can be obtained. n Doppler velocity {v} p1_0_doppler ,v p2_0_doppler ,v p3_0_doppler ,…,v pn_0_doppler}

[0069] Step 5: Calculate the Doppler velocity (radial velocity) of the background points based on the horizontal component of each radar velocity, and then construct a 3D target detection dataset containing Doppler velocity. The 3D target detection dataset containing Doppler velocity is composed of the original 3D target detection dataset and the Doppler velocities of each target object point cloud and background points.

[0070] Relative to the world coordinate system, the velocity of all background points is zero. Relative to the radar coordinate system, the magnitude of the velocity of all background points is equal to the magnitude of the radar's velocity, but in the opposite direction; that is, the velocity of background point P is zero. j The Doppler velocity information is the component of the radar's motion velocity along the laser beam direction. Therefore, in step five, the inverse of the horizontal component of the radar velocity corresponding to each background point is taken as its Doppler velocity, as shown in the following formula:

[0071] v background_j_doppler =-v 0_j_doppler

[0072] Among them, v background_j_doppler Let v be the Doppler velocity of the j-th background point. 0_j_doppler Let be the horizontal component of the radar velocity on the radar laser beam where the j-th background point is located.

[0073] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A FMCW lidar 3D target detection data augmentation method, characterized in that, Includes the following steps: Step 1: Obtain the original 3D object detection dataset. If the coordinates of the center point of the 3D object in the original 3D object detection dataset and / or the target rotation angle are located in the camera coordinate system, proceed to Step 2; otherwise, proceed to Step 3. Step 2: Transform the coordinates of the center point of the 3D object and / or the target rotation angle in the original 3D target detection dataset from the camera coordinate system to the radar coordinate system; Step 3: If the point cloud data in the original 3D object detection dataset contains velocity information, proceed to Step 4; otherwise, initialize the motion velocity of the point cloud data to obtain the velocity information corresponding to the point cloud data. In step three, for each frame of point cloud data, the velocity magnitude of the point cloud corresponding to each target object is randomly set on the basis of satisfying the velocity constraint, and the rotation angle of the corresponding target in the original 3D target detection dataset is used as the velocity direction of the point cloud corresponding to each target object. Step 4: Based on the current 3D target detection dataset, calculate the horizontal component of each radar velocity and the Doppler velocity of each target object's point cloud; Step 5: Calculate the Doppler velocity of the background points based on the horizontal component of each radar velocity, and then construct a 3D target detection dataset containing the Doppler velocity. In step five, the inverse of the horizontal component of the radar velocity corresponding to each background point is taken as its Doppler velocity, as shown in the following formula: wherein, is the Doppler velocity of the jthbackground point, is the horizontal component of the radar velocity on the radar laser beam on which the jthbackground point lies.

2. The FMCW lidar 3D target detection data enhancement method according to claim 1, characterized in that, In step four, the calculation steps for the Doppler velocity of each target object point cloud and the corresponding radar point cloud are as follows: S1: Combining the velocities of the current target object and the corresponding radar, calculate the horizontal components of the target object's velocity on each laser beam and the horizontal components of the radar's velocity, using the following formulas: in, Let be the horizontal component of the current target object's velocity on the i-th laser beam of the radar. Let be the horizontal component of the velocity on the i-th laser beam of the radar. The i-th surface scan point of the current target object The speed of movement, Let be the velocity of the i-th laser beam of the radar. Point clouds on the horizontal plane The angle between the velocity direction and the i-th laser beam. Let be the angle between the radar's own velocity on the horizontal plane and the i-th laser beam. , These are the i-th surface scan points of the current target object. xy-axis coordinates, , These are the x and y coordinates of the origin of the radar coordinate system. This represents the target rotation angle of the current target object in the radar coordinate system. S2: Based on the horizontal components of the target object's velocity on each laser beam and the horizontal component of the radar's velocity, calculate the vertical components of the target object's velocity on each laser beam and the vertical component of the radar's velocity. The calculation formula is as follows: in, Let be the vertical component of the current target object's velocity on the i-th laser beam of the radar. Let be the vertical component of the radar velocity on the i-th laser beam. Let be the angle between the i-th laser beam of the radar and the horizontal plane. The z-axis coordinate value of the origin of the radar coordinate system. The i-th surface scan point of the current target object The z-axis coordinate value; The i-th surface scan point of the current target object x-axis coordinate value; S3: Based on the vertical components of the target object's velocity along each laser beam and the vertical component of the radar velocity, the Doppler velocity of the target object is calculated using the following formula: in, The i-th surface scan point of the current target object Doppler velocity.

3. The method for enhancing 3D target detection data of FMCW lidar according to claim 1, characterized in that, In step five, the 3D object detection dataset containing Doppler velocity is composed of the original 3D object detection dataset and the Doppler velocities of each object point cloud and background point.

Citation Information

Patent Citations

  • Systems and methods for enhanced distance estimation by a mono-camera using radar and motion data

    CN111223135A

  • Simulation method of frequency-modulated continuous wave laser radar

    CN111830490A