Millimeter wave radar point cloud data augmentation method and system, and electronic device

Abstract

The application provides a millimeter wave radar point cloud data augmentation method, system and electronic equipment, the method comprises the following steps: superimposing multiple frames of point clouds on a current frame to obtain multiple first point clouds with non-zero Doppler velocity; setting the Doppler velocity of the multiple first point clouds to zero, wherein the multiple first point clouds are matched with corresponding labeled boxes; taking the multiple first point clouds with zero Doppler velocity as first augmented point clouds, and taking the labeled boxes corresponding to the multiple first point clouds with zero velocity as the labeled boxes corresponding to the first augmented point clouds. In view of the problem that the millimeter wave radar has poor detection of static targets, the application innovatively provides a data augmentation method for static scene reconstruction, improves the problem of unbalanced static target data, and improves the detection capability of static targets.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method, system, and electronic device for augmenting millimeter-wave radar point cloud data. Background Technology

[0002] Millimeter-wave radar uses radiated electromagnetic energy to measure targets within the sensor's field of view, exhibiting strong long-range target detection capabilities. Compared to other automotive sensors, automotive radar provides unique target velocity (Doppler) measurements, maintains good robustness in adverse weather and strong light conditions, and is relatively inexpensive. Therefore, it is considered the most reliable sensor in automotive perception technology. With advancements in radar and chip technology, millimeter-wave radar is becoming increasingly smaller, enabling it to perform well in complex multi-target environments and provide better early warning for drivers, leading to its growing importance.

[0003] As millimeter-wave radar expands from Advanced Driver Assistance Systems (ADAS) to higher-level autonomous driving applications such as L2 and L3, its perception capabilities demand higher detection accuracy, higher detection rates, and lower false detection probabilities, while also providing more precise target information. However, due to the limitations of millimeter-wave radar's angular resolution, its point cloud distribution is relatively sparse. For the same target, the point cloud distribution in different frames is somewhat random, lacking clear contour features. Furthermore, the impact of lateral detection accuracy leads to low target position accuracy, posing a significant challenge to millimeter-wave radar perception.

[0004] With the rapid development of deep learning in computer vision, autonomous driving is also beginning to adopt deep learning methods to replace traditional algorithms. Currently, 3D detection methods based on vision or LiDAR point clouds have made significant progress. With the rise of deep learning-based millimeter-wave radar detection algorithms, deep learning is playing an increasingly important role in millimeter-wave radar detection and tracking. However, because deep learning technology based on millimeter-wave radar started relatively late, many aspects are still immature, especially in terms of data format and representation, which differs greatly from images and LiDAR, resulting in a lack of corresponding systematic data augmentation techniques. Therefore, it is crucial to research a universal point cloud augmentation technique suitable for millimeter-wave radar target detection.

[0005] Existing deep learning-based millimeter-wave radar target detection algorithms rely on manual annotation. Deep neural networks undergo supervised learning through manual annotation, adjusting model parameters to complete pre-defined tasks. Autonomous driving is an open scenario, and to maintain good model performance in such environments, deeper and larger networks are needed, along with massive amounts of data. However, the cost and efficiency of manual data annotation for autonomous driving are enormous, and the amount of data varies across different scenarios. Therefore, it is necessary to augment existing data to meet the performance requirements of algorithms in different scenarios.

[0006] Currently, data augmentation techniques based on images and LiDAR point clouds are relatively mature. However, deep learning algorithms for millimeter-wave radar have a shorter development history and differ significantly from traditional visual and LiDAR data representations, primarily in that: point cloud density is sparser; the information content of a single point is more abundant; and features of different dimensions at each point are coupled. Therefore, data augmentation methods for millimeter-wave radar differ greatly from those used in traditional methods. Consequently, a systematic approach to millimeter-wave radar data augmentation is needed to compensate for performance deficiencies caused by insufficient and imbalanced data.

[0007] The information disclosed in this background section is intended only to enhance the understanding of the general background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0008] To address the problems existing in the prior art, this invention provides a method, system, and electronic device for augmenting millimeter-wave radar point cloud data.

[0009] This invention provides a method for augmenting millimeter-wave radar point cloud data, the method comprising:

[0010] Multiple point clouds are superimposed onto the current frame to obtain multiple first point clouds with non-zero Doppler velocity measurements. The Doppler velocity measurements of the multiple first point clouds are then set to zero. Each of the multiple first point clouds is matched with a corresponding bounding box.

[0011] The multiple first point clouds after the Doppler velocity measurement is zeroed are used as the first augmented point cloud, and the corresponding annotation boxes of the multiple first point clouds are used as the annotation boxes corresponding to the first augmented point cloud after the velocity is zeroed.

[0012] The present invention provides a method for augmenting millimeter-wave radar point cloud data.

[0013] The multi-frame point clouds are superimposed onto the current frame after motion compensation to obtain multiple second point clouds where all Doppler velocities are zero;

[0014] The multiple second point clouds are superimposed with the first augmented point cloud to form the current frame after data augmentation, and the annotation boxes corresponding to the multiple second point clouds and the annotation boxes corresponding to the multiple first point clouds are superimposed as the annotation boxes corresponding to the current frame after data augmentation.

[0015] According to a method for augmenting millimeter-wave radar point cloud data provided by the present invention, the method further includes:

[0016] The velocities of multiple third point clouds in the vehicle coordinate system are obtained, wherein the multiple third point clouds are matched with corresponding bounding boxes;

[0017] Based on the predetermined rotation angle and the coordinates of the plurality of third point clouds in the vehicle coordinate system, the coordinates of the plurality of fourth point clouds obtained after rotating the plurality of third point clouds according to the rotation angle are obtained in the vehicle coordinate system.

[0018] Based on the coordinates of the plurality of fourth point clouds in the vehicle coordinate system, the rotation angle, and the velocity of the plurality of third point clouds in the vehicle coordinate system, Doppler velocity measurements of the plurality of fourth point clouds are obtained.

[0019] Based on the coordinates of the plurality of fourth point clouds and Doppler velocity measurements, a second augmented point cloud is generated, and the annotation boxes corresponding to the plurality of third point clouds are rotated according to the rotation angle and used as the annotation boxes corresponding to the second augmented point cloud.

[0020] According to a method for augmenting millimeter-wave radar point cloud data provided by the present invention, the method further includes:

[0021] The velocities of multiple fifth point clouds in the vehicle coordinate system are obtained, wherein the multiple fifth point clouds are matched with corresponding bounding boxes;

[0022] Based on the predetermined translation amount and the coordinates of the plurality of fifth point clouds in the vehicle coordinate system, obtain the coordinates of the plurality of sixth point clouds obtained after translating the plurality of fifth point clouds according to the translation amount in the vehicle coordinate system.

[0023] Based on the coordinates of the plurality of sixth point clouds in the vehicle coordinate system and the velocity of the plurality of fifth point clouds in the vehicle coordinate system, Doppler velocity measurements of the plurality of sixth point clouds are obtained.

[0024] Based on the coordinates of the plurality of sixth point clouds and Doppler velocity measurements, a third augmented point cloud is generated, and the annotation boxes corresponding to the plurality of fifth point clouds are translated according to the translation amount and used as the annotation boxes corresponding to the third augmented point cloud.

[0025] According to a millimeter-wave radar point cloud data augmentation method provided by the present invention, the method for acquiring the velocities of multiple third point clouds in the vehicle coordinate system, or the method for acquiring the velocities of multiple fifth point clouds in the vehicle coordinate system, includes the following:

[0026] If the bounding boxes corresponding to multiple point clouds include labeled velocities, then the labeled velocities are taken as the velocities of the multiple point clouds in the vehicle coordinate system;

[0027] If the annotation boxes corresponding to multiple point clouds do not include the annotation speed, the speed of the multiple point clouds in the vehicle coordinate system is calculated based on a system of simultaneous equations. Specifically, for each point cloud in the multiple point clouds, an equation is formed based on the relationship between the angle of each point cloud in the vehicle coordinate system, the Doppler velocity of each point cloud, and the speed of the multiple point clouds in the vehicle coordinate system.

[0028] According to a method for augmenting millimeter-wave radar point cloud data provided by the present invention, the method further includes:

[0029] Obtain the seventh point cloud regarding gantry cranes, and / or tunnels, and / or viaducts;

[0030] The seventh point cloud is superimposed onto the current frame as the fourth augmented point cloud.

[0031] According to a method for augmenting millimeter-wave radar point cloud data provided by the present invention, the method further includes:

[0032] Add white noise to the lateral coordinates of the augmented point cloud;

[0033] And / or, add white noise to the radar cross section of the augmented point cloud.

[0034] This invention also provides a millimeter-wave radar point cloud data augmentation system, the system comprising:

[0035] The overlay module is used to overlay multiple frame point clouds onto the current frame, obtain multiple first point clouds with non-zero Doppler velocity, and set the Doppler velocity of the multiple first point clouds to zero. The multiple first point clouds are matched with corresponding bounding boxes.

[0036] An augmentation module is used to take the plurality of first point clouds after the Doppler velocity measurement is zeroed as the first augmented point cloud, and to take the annotation boxes corresponding to the plurality of first point clouds after the velocity is zeroed as the annotation boxes corresponding to the first augmented point cloud.

[0037] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the millimeter-wave radar point cloud data augmentation method as described in any of the preceding claims.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the millimeter-wave radar point cloud data augmentation method as described in any of the preceding claims.

[0039] The millimeter-wave radar point cloud data augmentation method, system, and electronic equipment provided by this invention address the problem of poor detection of stationary targets by millimeter-wave radar by innovatively proposing a data augmentation method for static scene reconstruction, which improves the problem of unbalanced stationary target data and enhances the detection capability of stationary targets. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of a left-right mirror image in the prior art;

[0042] Figure 2 A flowchart illustrating a millimeter-wave radar point cloud data augmentation method provided by the present invention;

[0043] Figure 3 This is a schematic diagram of static target point cloud augmentation provided by the present invention;

[0044] Figure 4 A schematic diagram of a rotationally augmented point cloud provided by the present invention;

[0045] Figure 5 A schematic diagram illustrating a method for replicating augmented point clouds provided by the present invention;

[0046] Figure 6 This invention provides a schematic diagram of a millimeter-wave radar point cloud data augmentation system.

[0047] Figure 7 This is a schematic diagram of the physical structure of an electronic device provided by the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0049] It should be noted that the vehicle coordinate system mentioned in this invention is a special moving coordinate system used to describe the motion of a vehicle. Its origin coincides with the center of mass. When the vehicle is stationary on a level road, the X-axis is parallel to the ground and points forward, the Z-axis passes through the vehicle's center of mass and points upward, and the Y-axis points to the driver's left. Furthermore, a polar coordinate scheme using radius and angle is adopted in the XY-axis plane.

[0050] To better illustrate the technical solution of this invention, we will first introduce conventional data augmentation methods in the prior art, specifically including:

[0051] 1) Mirroring can solve the problem of imbalance between left and right samples.

[0052] Figure 1 This is a schematic diagram of a left-right mirror image in the prior art, such as... Figure 1 As shown, the main approach is as follows: the global millimeter-wave radar point cloud 2 is flipped horizontally, that is, the y-axis coordinate of point cloud 2 becomes negative, forming a new augmented point cloud. At the same time, the bounding box 2 (Ground Truth, GT) is mirrored, and the velocity direction and yaw angle are symmetrical with the original value axes.

[0053] 2) Multi-frame overlay to solve the problem of sparse point clouds in millimeter-wave radar. Due to the sparseness of point clouds and the existence of certain errors, the reliability of a single frame of point cloud is relatively low, so multi-frame point cloud overlay is required.

[0054] The main steps are as follows: record the pose information of the vehicle with multiple frames of point cloud information and the timestamps corresponding to the point clouds; record the relative pose changes of the vehicle from multiple frames of point cloud to the current time; compensate the vehicle's motion with the point cloud of the corresponding frame to obtain the distribution of the point cloud of the corresponding frame in the current pose state of the vehicle; and superimpose the multiple frames of point cloud to obtain the multi-frame point cloud result after motion compensation.

[0055] The millimeter-wave radar point cloud data augmentation method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0056] Figure 2 This is a flowchart illustrating a millimeter-wave radar point cloud data augmentation method provided by the present invention, as shown below. Figure 2 As shown, the present invention provides a method for augmenting millimeter-wave radar point cloud data, which may include the following steps.

[0057] It should be noted that this invention is applicable to vehicles equipped with autonomous or assisted driving systems, and the point clouds described below are all obtained by the millimeter-wave radar detecting obstacles.

[0058] S100: Overlay multiple frame point clouds onto the current frame to obtain multiple first point clouds with non-zero Doppler velocity measurements, and set the Doppler velocity measurements of the multiple first point clouds to zero. Among them, the multiple first point clouds are matched with corresponding bounding boxes.

[0059] It should be noted that the point clouds from multiple frames are superimposed onto the current frame directly without motion compensation.

[0060] Preferably, the multi-frame point cloud is a continuous multi-frame point cloud within a short period of time.

[0061] Preferably, the bounding boxes corresponding to the multiple first point clouds are vehicles, pedestrians, and other obstacles on the road, and these vehicles, pedestrians, and other obstacles all have a moving speed relative to the ground.

[0062] Preferably, the bounding boxes corresponding to the multiple first point clouds are other vehicles detected by the vehicle itself, and the bounding boxes include the vehicle's relative speed to the ground.

[0063] S200. After zeroing the Doppler velocity measurement, multiple first point clouds are used as the first augmented point cloud, and the velocity of the corresponding annotation boxes of multiple first point clouds is zeroed and used as the annotation boxes corresponding to the first augmented point cloud.

[0064] Preferably, setting the velocity of the annotation boxes corresponding to multiple first point clouds to zero includes setting the velocity of the vehicle relative to the ground included in the annotation box to zero.

[0065] Optionally, multiple point clouds are superimposed onto the current frame after motion compensation to obtain multiple second point clouds where all Doppler velocities are zero;

[0066] Multiple second point clouds are superimposed with the first augmented point cloud to form the current frame after data augmentation, and the annotation boxes corresponding to multiple second point clouds and multiple first point clouds are superimposed to form the annotation boxes corresponding to the current frame after data augmentation.

[0067] Figure 3 This is a schematic diagram of static target point cloud augmentation provided by the present invention, such as... Figure 3 As shown, the original point cloud 10 of multiple frames is superimposed with motion compensation and the moving point cloud is removed. The remaining point cloud is the static scene point cloud 20. The static scene point cloud 20 is then removed from the point cloud formed by direct superposition without motion compensation to obtain the moving point cloud 30 used to simulate the static point cloud. The static scene point cloud 20 and the moving point cloud 30 are then reconstructed to form a new static scene 40.

[0068] It's important to note that millimeter-wave radar is the primary sensor for detecting static obstacles in autonomous and assisted driving systems. Therefore, it's crucial to ensure the detection capability of millimeter-wave radar for static obstacles. Typically, autonomous vehicles traveling at high speeds require longer braking distances, necessitating even greater detection range. However, in most scenarios, it's difficult to capture static obstacles in the center of the road during high-speed driving. Consequently, the distribution of dynamic and static obstacle samples in conventional data is highly unbalanced, resulting in a weaker detection capability for static obstacles in the model. Therefore, by augmenting the current frame with the first augmented point cloud, the point cloud distribution of static targets can be approximated, significantly increasing the quantity and diversity of static point clouds and avoiding the time-consuming and expensive work of collecting and labeling uncommon data.

[0069] This embodiment addresses the challenge of poor stationary target detection in millimeter-wave radar by innovatively proposing a data augmentation method for stationary scene reconstruction. This method improves the problem of unbalanced stationary target data and enhances the detection capability of stationary targets.

[0070] Optionally, firstly, based on the bounding box and multiple point clouds, the bounding box and point clouds are matched to complete the segmentation of the foreground points, and then the velocities of multiple point clouds in the vehicle coordinate system are obtained based on the following method.

[0071] If multiple point clouds correspond to bounding boxes that include labeled velocities, then the labeled velocities are used as the velocities of the multiple point clouds in the vehicle coordinate system; that is, the velocity (v) of the bounding boxes corresponding to the foreground point clouds is used. x v y v, as the point cloud x v y .

[0072] If the bounding boxes corresponding to multiple point clouds do not include labeled velocities, then the velocities of the multiple point clouds in the vehicle coordinate system are calculated based on a system of simultaneous equations. Specifically, for each point cloud in the multiple point clouds, an equation is formed based on the relationship between the angle of each point cloud in the vehicle coordinate system, the Doppler velocity of each point cloud, and the velocities of the multiple point clouds in the vehicle coordinate system. See the following formula for details:

[0073]

[0074] Among them, v r The doppler velocity of the point cloud is represented by θ, and the azimuth angle of the point cloud relative to the radar is represented by v. x v y v represents the velocity of the entire point cloud in the x-axis and y-axis directions, respectively. r The superscript of θ and the subscript of θ represent the point cloud number.

[0075] v is obtained through least squares. x v yThe estimate will be v x v y Assign a corresponding point cloud.

[0076] Optionally, Figure 4 This is a schematic diagram of a rotationally augmented point cloud provided by the present invention, as shown below. Figure 4 As shown, the method also includes:

[0077] Obtain the velocities of multiple third point clouds in the vehicle coordinate system, where each third point cloud is matched with a corresponding bounding box;

[0078] Based on the predetermined rotation angle and the coordinates of multiple third point clouds in the vehicle coordinate system, obtain the coordinates of multiple fourth point clouds obtained by rotating multiple third point clouds according to the rotation angle in the vehicle coordinate system.

[0079] Based on the coordinates and rotation angles of multiple fourth point clouds in the vehicle coordinate system and the velocities of multiple third point clouds in the vehicle coordinate system, Doppler velocity measurements of multiple fourth point clouds are obtained.

[0080] Based on the coordinates of multiple fourth point clouds and Doppler velocity measurements, a second augmented point cloud is generated, and the annotation boxes corresponding to multiple third point clouds are rotated according to the rotation angle and used as the annotation boxes corresponding to the second augmented point cloud.

[0081] Preferably, after rotation, based on the position of the fourth point cloud, v x v y The projection of the point cloud velocity onto the radar electric axis is recalculated to obtain the predicted value of the rotated Doppler velocity measurement, as shown in the following formula:

[0082]

[0083] Among them, v x v y Let x and y represent the velocities of the third point cloud along the x and y axes, respectively, and let x and y represent the position coordinates of the third point cloud, respectively. r ' represents the position coordinates of the fourth point cloud and the Doppler velocity measurement, respectively; Δθ represents the rotation angle; and θ represents the azimuth angle of the third point cloud.

[0084] Furthermore, it should be noted that since the shape of the point cloud distribution is related to the relative position of the radar, the angle during rotation should not be too large, and is generally selected to be within 10°.

[0085] It should be noted that rotating the global point cloud by a small angle is a common data augmentation method. However, since millimeter-wave radar point clouds have a Doppler observation dimension, which is the projection of the target velocity onto the radar's electrical axis, Doppler velocity measurement is related to the target's position. Rotating the point cloud, whether globally or locally, will lead to inaccurate Doppler velocity measurements, thus reducing algorithm performance. This invention overcomes the problem of inaccurate Doppler velocity measurement through re-prediction of Doppler velocity measurements, making the point cloud rotation augmentation method usable.

[0086] Optionally, the method further includes:

[0087] Obtain the velocities of multiple fifth point clouds in the vehicle coordinate system, where each fifth point cloud is matched with a corresponding bounding box;

[0088] Based on the predetermined translation amount and the coordinates of multiple fifth point clouds in the vehicle coordinate system, obtain the coordinates of multiple sixth point clouds obtained after translating multiple fifth point clouds according to the translation amount in the vehicle coordinate system.

[0089] Based on the coordinates of multiple sixth point clouds in the vehicle coordinate system and the velocities of multiple fifth point clouds in the vehicle coordinate system, Doppler velocity measurements of multiple sixth point clouds are obtained.

[0090] Based on the coordinates of multiple sixth point clouds and Doppler velocity measurements, a third augmented point cloud is generated, and the corresponding bounding boxes of multiple fifth point clouds are translated according to the translation amount and used as the corresponding bounding boxes of the third augmented point cloud.

[0091] Preferably, the fifth point cloud is translated according to a preset translation amount, and the Doppler velocity is re-predicted based on the position of the sixth point cloud, specifically according to the following formula:

[0092]

[0093] Among them, v x v y Let x and y represent the velocities of the fifth point cloud along the x and y axes, respectively, and let x and y represent the position coordinates of the fifth point cloud, respectively. r ' represents the position coordinates of the sixth point cloud and the Doppler velocity measurement, respectively, and Δx and Δy represent the translation amount.

[0094] Furthermore, it should be noted that since the distribution and location of point clouds are related, the translation distance generally does not exceed 10m.

[0095] To address the problem that the coupling between radar point cloud velocity and position prevents translation and rotation, a data augmentation method based on Doppler velocity re-prediction is proposed.

[0096] Optionally, Figure 5This invention provides a schematic diagram of a replicated augmented point cloud, as shown below. Figure 5 As shown, the method also includes:

[0097] Get the seventh point cloud 3 regarding gantry cranes, and / or tunnels, and / or viaducts;

[0098] The seventh point cloud 3 is superimposed onto the current frame as the fourth augmented point cloud.

[0099] Optionally, the method further includes:

[0100] Add white noise to the horizontal coordinates of the augmented point cloud;

[0101] And / or, add white noise to the radar cross section of the augmented point cloud.

[0102] Preferably, due to the low lateral detection accuracy and large fluctuations in the radar cross section (RCS) of millimeter-wave radar, the enhancement is mainly achieved by superimposing Gaussian white noise on these two dimensions, as shown in the following formula:

[0103]

[0104] Where x = x represents the x-axis coordinate remaining unchanged after adding noise, y = y + Δy represents the y-axis coordinate after adding noise equal to the y-axis coordinate before adding noise plus Δy, and Δy follows a Gaussian distribution of N(0, rB / 2), where r represents the point cloud distance and B represents the radar angular resolution. Similarly, RCS = RCS + ΔRCS represents the radar cross section after adding noise equal to the radar cross section before adding noise plus ΔRCS, and ΔRCS follows a Gaussian distribution of N(0, σ). 2 / 4), where σ is the standard deviation of the prior statistical RCS distribution.

[0105] Superimposing Gaussian white noise on data from various dimensions can effectively improve the robustness of the algorithm, enhance detection stability in cluttered environments, and reduce the algorithm's sensitivity to inaccurate annotations.

[0106] Preferably, a point cloud database of gantry cranes, and / or tunnels, and / or viaducts is established, and during the training of the target detection model, the point cloud of the gantry crane / tunnel is copied into the training scene with a predetermined probability.

[0107] It should be noted that, due to the limited elevation resolution of traditional millimeter-wave radar and the lack of height differentiation in point clouds, static radar point clouds will appear on the tunnel roof and metal poles in scenarios such as tunnels, gantry cranes, and viaducts, resulting in a large number of stationary radar point clouds and generating numerous false detections. Therefore, data augmentation is needed to enhance the model's ability to distinguish stationary objects.

[0108] The millimeter-wave radar point cloud data augmentation system provided by the present invention is described below. The millimeter-wave radar point cloud data augmentation system described below can be referred to in correspondence with the millimeter-wave radar point cloud data augmentation method described above.

[0109] Figure 6 This invention provides a schematic diagram of a millimeter-wave radar point cloud data augmentation system, as shown below. Figure 6 As shown, the present invention also provides a millimeter-wave radar point cloud data augmentation system, the system comprising:

[0110] The overlay module is used to overlay point clouds from multiple frames onto the current frame, obtain multiple first point clouds with non-zero Doppler velocities, and set the Doppler velocities of multiple first point clouds to zero. Among them, multiple first point clouds are matched with corresponding bounding boxes.

[0111] The augmentation module is used to take multiple first point clouds after the Doppler velocity measurement is zeroed as the first augmented point cloud, and to take the corresponding annotation boxes of multiple first point clouds after the velocity is zeroed as the annotation boxes corresponding to the first augmented point cloud.

[0112] This embodiment addresses the challenge of poor stationary target detection in millimeter-wave radar by innovatively proposing a data augmentation method for stationary scene reconstruction. This method improves the problem of unbalanced stationary target data and enhances the detection capability of stationary targets.

[0113] Figure 7 A schematic diagram of the physical structure of an electronic device provided by the present invention, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a millimeter-wave radar point cloud data augmentation method, the method including:

[0114] Multiple point clouds are superimposed onto the current frame to obtain multiple first point clouds with non-zero Doppler velocity measurements. The Doppler velocity measurements of the multiple first point clouds are then set to zero. Each of the multiple first point clouds is matched with a corresponding bounding box.

[0115] The multiple first point clouds after the Doppler velocity measurement is zeroed are used as the first augmented point cloud, and the corresponding annotation boxes of the multiple first point clouds are used as the annotation boxes corresponding to the first augmented point cloud after the velocity is zeroed.

[0116] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the millimeter-wave radar point cloud data augmentation method provided by the above methods, the method comprising:

[0118] Multiple point clouds are superimposed onto the current frame to obtain multiple first point clouds with non-zero Doppler velocity measurements. The Doppler velocity measurements of the multiple first point clouds are then set to zero. Each of the multiple first point clouds is matched with a corresponding bounding box.

[0119] The multiple first point clouds after the Doppler velocity measurement is zeroed are used as the first augmented point cloud, and the corresponding annotation boxes of the multiple first point clouds are used as the annotation boxes corresponding to the first augmented point cloud after the velocity is zeroed.

[0120] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the millimeter-wave radar point cloud data augmentation methods provided above, the methods comprising:

[0121] Multiple point clouds are superimposed onto the current frame to obtain multiple first point clouds with non-zero Doppler velocity measurements. The Doppler velocity measurements of the multiple first point clouds are then set to zero. Each of the multiple first point clouds is matched with a corresponding bounding box.

[0122] The multiple first point clouds after the Doppler velocity measurement is zeroed are used as the first augmented point cloud, and the corresponding annotation boxes of the multiple first point clouds are used as the annotation boxes corresponding to the first augmented point cloud after the velocity is zeroed.

[0123] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for augmenting millimeter wave radar point cloud data, the method comprising: The method includes: Multiple point clouds are superimposed onto the current frame to obtain multiple first point clouds with non-zero Doppler velocity measurements. The Doppler velocity measurements of the multiple first point clouds are then set to zero. Each of the multiple first point clouds is matched with a corresponding bounding box. The multiple first point clouds after the Doppler velocity measurement is zeroed are used as the first augmented point cloud, and the velocity of the corresponding annotation boxes of the multiple first point clouds is zeroed and used as the annotation boxes corresponding to the first augmented point cloud; the velocity of the annotation box is the velocity of the vehicle relative to the ground.

2. The millimeter-wave radar point cloud data augmentation method according to claim 1, characterized in that, The multi-frame point clouds are superimposed onto the current frame after motion compensation to obtain multiple second point clouds where all Doppler velocities are zero; The multiple second point clouds are superimposed with the first augmented point cloud to form the current frame after data augmentation, and the annotation boxes corresponding to the multiple second point clouds and the annotation boxes corresponding to the multiple first point clouds are superimposed as the annotation boxes corresponding to the current frame after data augmentation.

3. The method of claim 1, wherein, The method further includes: The velocities of multiple third point clouds in the vehicle coordinate system are obtained, wherein the multiple third point clouds are matched with corresponding bounding boxes; Based on the predetermined rotation angle and the coordinates of the plurality of third point clouds in the vehicle coordinate system, the coordinates of the plurality of fourth point clouds obtained after rotating the plurality of third point clouds according to the rotation angle are obtained in the vehicle coordinate system. Based on the coordinates of the plurality of fourth point clouds in the vehicle coordinate system, the rotation angle, and the velocity of the plurality of third point clouds in the vehicle coordinate system, Doppler velocity measurements of the plurality of fourth point clouds are obtained. Based on the coordinates of the plurality of fourth point clouds and Doppler velocity measurements, a second augmented point cloud is generated, and the annotation boxes corresponding to the plurality of third point clouds are rotated according to the rotation angle and used as the annotation boxes corresponding to the second augmented point cloud.

4. The method of claim 1, wherein, The method further includes: The velocities of multiple fifth point clouds in the vehicle coordinate system are obtained, wherein the multiple fifth point clouds are matched with corresponding bounding boxes; Based on the predetermined translation amount and the coordinates of the plurality of fifth point clouds in the vehicle coordinate system, obtain the coordinates of the plurality of sixth point clouds obtained after translating the plurality of fifth point clouds according to the translation amount in the vehicle coordinate system. Based on the coordinates of the plurality of sixth point clouds in the vehicle coordinate system and the velocity of the plurality of fifth point clouds in the vehicle coordinate system, Doppler velocity measurements of the plurality of sixth point clouds are obtained. Based on the coordinates of the plurality of sixth point clouds and Doppler velocity measurements, a third augmented point cloud is generated, and the annotation boxes corresponding to the plurality of fifth point clouds are translated according to the translation amount and used as the annotation boxes corresponding to the third augmented point cloud.

5. The method of augmenting millimeter wave radar point cloud data according to claim 3 or 4, characterized in that, The method for obtaining the velocities of multiple third point clouds in the vehicle coordinate system, or the method for obtaining the velocities of multiple fifth point clouds in the vehicle coordinate system, includes the following: If the bounding boxes corresponding to multiple point clouds include labeled velocities, then the labeled velocities are taken as the velocities of the multiple point clouds in the vehicle coordinate system; If the annotation boxes corresponding to multiple point clouds do not include the annotation speed, the speed of the multiple point clouds in the vehicle coordinate system is calculated based on a system of simultaneous equations. Specifically, for each point cloud in the multiple point clouds, an equation is formed based on the relationship between the angle of each point cloud in the vehicle coordinate system, the Doppler velocity of each point cloud, and the speed of the multiple point clouds in the vehicle coordinate system.

6. The millimeter-wave radar point cloud data augmentation method according to claim 1, characterized in that, The method further includes: Obtain the seventh point cloud regarding gantry cranes, and / or tunnels, and / or viaducts; The seventh point cloud is superimposed onto the current frame as the fourth augmented point cloud.

7. The millimeter-wave radar point cloud data augmentation method according to claim 1, characterized in that, The method further includes: Add white noise to the lateral coordinates of the augmented point cloud; And / or, add white noise to the radar cross section of the augmented point cloud.

8. A millimeter-wave radar point cloud data augmentation system, characterized in that, The system includes: The overlay module is used to overlay multiple frame point clouds onto the current frame, obtain multiple first point clouds with non-zero Doppler velocity, and set the Doppler velocity of the multiple first point clouds to zero. The multiple first point clouds are matched with corresponding bounding boxes. An augmentation module is used to take the plurality of first point clouds after the Doppler velocity measurement is zeroed as the first augmented point cloud, and to take the label frame velocity corresponding to the plurality of first point clouds after the velocity is zeroed as the label frame corresponding to the first augmented point cloud; the label frame velocity is the labeled vehicle velocity relative to the ground.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the millimeter-wave radar point cloud data augmentation method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the millimeter-wave radar point cloud data augmentation method as described in any one of claims 1-7.

Images