Point Cloud-Based Radar Data Enhancement Method and Device

Abstract

The present invention provides a method and apparatus for point cloud-based radar data enhancement. The method comprises: obtaining environmental point cloud data collected by radar; the environmental point cloud data includes a first obstacle; performing a data enhancement operation on the first obstacle to generate new environmental point cloud data, wherein the point cloud data of a second obstacle obtained by performing the data enhancement operation is consistent with the measurement resolution of the second obstacle's location. By guiding the generation of new datasets, embodiments of the present invention not only ensure sample diversity, rationality, and authenticity, but also mitigate the problem of training divergence caused by an excessive number of difficult samples.

Description

Technical Field

[0001] The present invention relates to the field of data enhancement technology, and in particular to a method and device for radar data enhancement based on point cloud. Background Art

[0002] With the increasing maturity of lidar technology, it has been widely adopted as a 3D perception sensor in autonomous driving in recent years. Currently, perception algorithms based on lidar data mainly include traditional methods, deep learning methods, and hybrid approaches. In the perception field, deep learning models trained with large amounts of sample data significantly outperform traditional methods. Due to limited sample data, training often suffers from overfitting caused by small sample sets. Data augmentation is a commonly used measure. Data augmentation is a data expansion technique that aims to maximize the value of limited data.

[0003] Existing point cloud-based radar data enhancement methods focus on increasing data diversity, but the resulting data has authenticity issues. Summary of the Invention

[0004] The problem solved by the present invention is that the data obtained by the existing point cloud-based data enhancement method has authenticity issues.

[0005] To address the above-mentioned problems, the present invention provides a point cloud-based radar data enhancement method, comprising: obtaining environmental point cloud data collected by a radar; the environmental point cloud data includes a first obstacle; performing a data enhancement operation on the first obstacle to generate new environmental point cloud data; the data enhancement operation includes at least one of the following: rotation, translation, scaling, occlusion, random noise addition, and partial area culling operations, and the point cloud data of a second obstacle obtained by performing the data enhancement operation is consistent with the measurement resolution of the location of the second obstacle.

[0006] Optionally, after performing the data enhancement operation on the first obstacle, the method further includes: reconstructing a bounding box of the point cloud data obtained by performing the data enhancement operation.

[0007] Optionally, the translation operation includes the following steps: determining the initial rotation angle of the obstacle within the visible angle range; the visible angle range is determined by the horizontal field of view angle of the radar; controlling the obstacle to rotate around the Z axis of the radar coordinate system according to the rotation matrix corresponding to the rotation angle, and the Z axis of the radar coordinate system is perpendicular to the horizontal plane; if the ratio of the visible point cloud before and after the rotation transformation of the obstacle is greater than or equal to a first preset threshold, and the position of the obstacle after the rotation transformation is consistent with the placement range corresponding to its category, then the obstacle after the rotation transformation is retained.

[0008] Optionally, the rotation operation includes the following steps: transforming the coordinate origin of the environmental point cloud data to the center of gravity of the obstacle to obtain an obstacle coordinate system; controlling the obstacle to rotate around at least one coordinate axis of the obstacle coordinate system by a preset angle to obtain a rotated obstacle; if the ratio of the visible point cloud before and after the rotation transformation of the obstacle is greater than or equal to a first preset threshold, and the position of the obstacle after the rotation transformation is consistent with the placement range corresponding to its category, then the obstacle after the rotation transformation is retained.

[0009] Optionally, the scaling operation includes the following steps: obtaining a random movement distance of the obstacle, and decomposing the random movement distance into a lateral movement distance and a longitudinal movement distance according to the orientation angle of the obstacle; moving the obstacle according to the lateral movement distance and the longitudinal movement distance to obtain a transformed obstacle; if the transformed obstacle is away from the radar, counting the number of point clouds in the grid of the transformed obstacle, if the grid is not empty and the number of point clouds in the grid is greater than the number of initial point clouds in the grid, retaining the point in the grid with the smallest distance from the radar and discarding the remaining points; if the transformed obstacle is close to the radar, performing an upsampling operation on the point cloud of the transformed obstacle.

[0010] Optionally, the occlusion operation includes the following steps: if the target obstacle meets the occlusion condition, another obstacle is translated to the front of the target obstacle according to the translation operation; the occlusion condition includes that the initial occlusion state is no occlusion or slight occlusion, and the number of point clouds is greater than a second preset threshold; if the ratio of the visible point cloud of the occluded target obstacle to the initial target obstacle is greater than or equal to the first preset threshold, the occluded target obstacle is retained and the status annotation information is updated.

[0011] Optionally, the operation of adding random noise includes the following steps: assuming that the random noise conforms to a Gaussian distribution, calculating the mean and variance of the obstacle point cloud and its respective neighborhood; using the mode of the obstacle point cloud mean set as the mean of the random noise, and using the mode of the obstacle point cloud variance set as the random noise variance; and adding the random noise to a preset proportion of randomly selected obstacle point clouds.

[0012] Optionally, the partial area removal operation includes the following steps: sorting the point clouds of the obstacles according to the echo intensity amplitude from small to large; removing the point clouds of the obstacles in order from low to high according to the echo intensity amplitude; and stopping the removal when the ratio of the remaining point cloud of the obstacle to the point cloud of the obstacle before removal meets the third preset threshold condition.

[0013] Optionally, the method further includes: if a hole appears during the data enhancement operation, performing the following point cloud hole filling step: searching for a point with a minimum height in the surrounding area of the point cloud hole as a ground point; and reconstructing the ground point based on the radar line emission angle and angular resolution corresponding to the point cloud hole.

[0014] The present invention provides a point cloud-based radar data enhancement device, comprising: an acquisition module for acquiring environmental point cloud data collected by a radar; the environmental point cloud data includes a first obstacle; and a data enhancement module for performing a data enhancement operation on the first obstacle to generate new environmental point cloud data; the data enhancement operation includes at least one of the following: rotation, translation, scaling, occlusion, random noise addition, and partial area culling operations, and the point cloud data of a second obstacle obtained by performing the data enhancement operation is consistent with the measurement resolution of the position where the second obstacle is located.

[0015] The point cloud-based radar data enhancement method and device provided by the embodiments of the present invention perform data enhancement operations on obstacles in environmental point cloud data to generate new environmental point cloud data. The obstacles obtained by performing the data enhancement operation are consistent with the measurement resolution of their locations. By guiding the generation of new data sets, training difficulties caused by too many difficult samples are avoided. Multiple obstacle data enhancement methods are used to increase sample diversity, which not only increases sample diversity but also ensures the authenticity of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0017] Figure 1 4 is a schematic flow chart of a point cloud-based radar data enhancement method in an embodiment of the present invention;

[0018] Figure 2 Schematic diagram of a radar coordinate system in an embodiment of the present invention;

[0019] Figure 3 Schematic diagram of the local coordinate system of an obstacle in scaling transformation in an embodiment of the present invention;

[0020] Figure 4 The figure is a schematic structural diagram of a point cloud-based radar data enhancement device in one embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0022] Existing point cloud data enhancement methods usually remove, rotate, translate, occlude and other operations on obstacles to obtain new obstacles to increase data diversity, but do not consider the impact of data enhancement on the resolution of point cloud measurements, resulting in data authenticity issues, and the failure to reconstruct bounding boxes, resulting in accuracy issues in the labeled data.

[0023] An embodiment of the present invention provides a point cloud-based radar data enhancement method, which can enhance obstacle data according to the actual radar acquisition situation, guide the generation of data-enhanced point clouds, avoid training difficulties caused by too many difficult samples, increase sample diversity, and ensure the authenticity of the data.

[0024] Figure 1 FIG. 1 is a schematic flow chart of a method for enhancing radar data based on point clouds in one embodiment of the present invention, the method comprising:

[0025] S102, obtaining environmental point cloud data collected by radar.

[0026] The environmental point cloud data includes obstacles. For example, a laser radar is used to collect environmental point cloud data. The laser radar is installed on a vehicle, robot, or other carrier to collect environmental data. If the radar coordinate system has pitch, roll, or yaw angles relative to the carrier coordinate system, the point cloud needs to be calibrated. The formula is as follows:

[0027] p′=R·p

[0028] Where R represents the posture correction matrix, p represents the point cloud coordinates before transformation, and p' represents the point cloud coordinates after transformation.

[0029] Figure 2 FIG. 4 shows a schematic diagram of a radar coordinate system, in which the positive direction of the X-axis is along the forward direction, the square of the Y-axis is along the left side, and the positive direction of the Z-axis is perpendicular to the horizontal plane and upward.

[0030] S104: Perform a data enhancement operation on the first obstacle to generate new environment point cloud data.

[0031] The data augmentation operation includes at least one of the following: rotation, translation, scaling, occlusion, random noise addition, and partial region culling. To ensure data authenticity, the point cloud data of the second obstacle obtained by performing the data augmentation operation must be consistent with the measurement resolution of the second obstacle's location.

[0032] Since the point cloud data structure is sparse, the rectangular coordinate system data of the point cloud is converted to spherical coordinates for storage to facilitate the determination of neighborhood relationships and then rasterized; the rectangular coordinate system data of the point cloud can also be converted to the image coordinate system.

[0033] Taking spherical coordinates as an example, each point in the point cloud coordinate system is represented by (x, y, z), and each point in the spherical coordinate system is represented by (α, β, d). The conversion formula between the two is as follows:

[0034]

[0035] Where α represents the elevation angle of the laser emission line of the point cloud, β represents the azimuth angle of the laser emission line, and d represents the distance from the obstacle to the lidar.

[0036] As can be seen from the above, the spherical coordinate system is further gridded into polar coordinates with horizontal and vertical angular resolutions as intervals and no radial division is done. There may be more than one point cloud in each grid. The conversion formula is as follows:

[0037]

[0038] Where row represents the vertical index of the polar grid image, col represents the horizontal coordinate of the polar grid image, ver_resolution and hor_resolution represent the vertical angular resolution and horizontal angular resolution of the sensor, respectively, and int represents the rounding operation.

[0039] Each frame of the environmental point cloud data generated in the above steps needs to be annotated with the vehicle path, pedestrian path, and other three areas. Obstacle data consists of the corresponding point cloud coordinates, echo intensity, 3D bounding box, motion direction, category, and occlusion status. The occlusion status can be categorized as 0, 0.5, and 1, indicating no obstruction, slight obstruction, and major obstruction, respectively.

[0040] When the obstacle is moved, a hole appears in the initial obstacle area. To approximate the actual point cloud, the point with the minimum height in the area around the obstacle in the scene is searched as the ground point. The ground point is reconstructed based on the laser line emission angle and angular resolution corresponding to the grid index of the hole area. The filled point cloud is estimated using the following formula.

[0041]

[0042] Where z represents the Z-axis minimum point within the initial obstacle area, α represents the pitch angle of the laser emission line, β represents the azimuth angle of the laser emission line, r represents the radial distance from the point cloud to the lidar, and x and y represent the horizontal and vertical coordinates of the point cloud, respectively.

[0043] In this embodiment, data enhancement includes rotation, translation, scaling, occlusion, random noise addition, and partial region culling. These data enhancement operations primarily involve performing data enhancement and subsequent fusion on obstacles in the current frame or between point clouds of different frames. The new data generation process must adhere to LiDAR acquisition principles, including ensuring that obstacles are present in real-world areas and considering the difficulty of model training with the new data.

[0044] For example, when occlusion exists between point clouds of different obstacles, the obscured portions of the obstacle must be removed, otherwise it violates the LiDAR acquisition principle. However, excessive point cloud removal can lead to blurring during model training, so the proportion of obstacles that are obscured must be determined in advance. For example, when an obstacle undergoes a translational transformation, directly applying displacement to the point cloud violates the LiDAR's measurement resolution. After data augmentation, obstacles should be located within their respective defined point cloud regions according to their category. For example, cars should not be parked in ponds, and pedestrians should not be embedded in trees or buildings.

[0045] The point cloud-based radar data enhancement method provided by an embodiment of the present invention performs data enhancement operations on obstacles in environmental point cloud data to generate new environmental point cloud data. The obstacles obtained by performing the data enhancement operation are consistent with the measurement resolution of their locations. By guiding the generation of new data sets, training difficulties caused by too many difficult samples are avoided. Multiple obstacle data enhancement methods are used to increase sample diversity, which not only increases sample diversity but also ensures the authenticity of the data.

[0046] Considering that various data enhancement methods will destroy the 3D bounding box constraints of the initial obstacles, the above-mentioned method of the embodiment of the present invention, after performing the data enhancement operation on the obstacles in the environmental point cloud data, also includes the following steps: reconstructing the bounding box of the point cloud data obtained by performing the data enhancement operation.

[0047] Specifically, the bounding box reconstruction based on the transformed point cloud is performed according to the following steps: first, the point cloud is projected onto the XY plane; second, the convex hull detection is implemented according to the Graham scanning method, and the 2D minimum circumscribed bounding box of the obstacle is calculated based on the convex hull detection results combined with the minimum area principle; then, the four corner points of the 2D minimum circumscribed bounding box are restored to the point cloud coordinate system according to the maximum and minimum values of the obstacle's Z axis, and the coordinate values of the eight corner points of the 3D bounding box are obtained.

[0048] The following introduces the feasible methods of each point cloud data enhancement operation.

[0049] Translation transformation is used to simulate the lateral movement of obstacles. Obstacles are generally placed on the ground, and the displacement is mainly in the horizontal and vertical directions. The laser radar scanning method is as follows: the laser emission lines on the same horizontal line are approximately circular, and scan the surrounding environment at equally spaced angles, and the measurement resolution of the point cloud on the same radius is approximately equal. If the coordinates are directly translated, the resolution of the position before the translation and the position after the translation are not the same, resulting in the inability to ensure that the point cloud measurement resolution conforms to the actual situation. An embodiment of the present invention provides a translation rule that conforms to the laser radar scanning mechanism, which may include the following steps:

[0050] First, the initial obstacle rotation angle is determined within the visible angle range. This visible angle range is determined by the radar's horizontal field of view. The rotation angle range is set by the lidar's horizontal field of view, and a random rotation angle is generated within this range.

[0051] Next, the obstacle is rotated around the Z axis of the radar coordinate system using the rotation matrix corresponding to the rotation angle. As mentioned above, the Z axis of the radar coordinate system is perpendicular to the horizontal plane. This rotation operation ensures that the measurement resolution of the point cloud remains unchanged before and after the transformation, and the obstacle is effectively displaced horizontally and vertically.

[0052] Then, if the ratio of the visible point cloud of the obstacle after the rotational transformation to the original obstacle is greater than or equal to a first preset threshold, and the position of the obstacle after the orientation angle correction is within the placement range corresponding to its category, the obstacle after the orientation angle correction is retained. Specifically, the ratio of the visible point cloud of the obstacle before and after the translation operation can be calculated, and obstacles with a ratio value not less than a preset threshold T1 and located within the placement range corresponding to the obstacle category can be retained. For example, T1 can be set to 0.5. New environmental point cloud data is generated based on the obstacles finally obtained by the translation.

[0053] The placement range refers to multiple manually calibrated polygonal areas within each point cloud frame, each corresponding to various obstacle categories. If the ratio of the intersection of the obstacle's bounding box and the placement range containing the obstacle's category exceeds a preset threshold, the obstacle's location is considered within the placement range for that category and can be retained. The threshold range is [0.8, 1.0].

[0054] For example, the ratio of the intersection area of the obstacle's bounding box and the placeable range containing the category to which the obstacle belongs to, to the area of the obstacle, can be calculated as follows: after dividing the obstacle's bounding box and the placeable range into grids of the same size, the three-dimensional point cloud data is converted into a two-dimensional projection; pixel filling is performed on the placeable range and the obstacle area, and the number of pixels in the intersection area and the number of pixels in the obstacle area are calculated, and then the ratio of the two numbers is calculated to obtain the ratio of the above-mentioned intersection area to the area of the obstacle.

[0055] In 3D space, conventional translation has three degrees of freedom: forward, backward, left, right, and up and down. Given that objects or obstacles are typically above ground, ignoring up and down movement, any other translation alone will result in a point cloud with a different number of visible points than the original, violating the actual point cloud measurement resolution.

[0056] Angular resolution determines a lidar's ability to discern objects at different distances. For example, a 16-line lidar with an angular resolution of 0.2° × 0.2° images an object at 100 meters, representing a physical size of approximately 0.35 × 0.35 square meters for a single point cloud. When capturing an object at 50 meters, the value of 0.2 / 180 × π × 50 represents a physical size of approximately 0.17 square meters for a single point cloud. For example, assuming a standing pedestrian is approximately 40 centimeters wide and 170 meters tall, a pedestrian would occupy 5 points at 100 meters and 24 points at 50 meters. This relationship becomes more pronounced as the angular resolution decreases.

[0057] Considering the above issues, simply translating the marked obstacles is not an option. Therefore, we rotate the target points around the origin, that is, around the Z axis of the radar coordinate system, to ensure that the scale of each point remains unchanged after rotation. This operation is equivalent to translating the marked obstacles both forward and backward, and left and right, similar to revolution. Later, the term "autorotation transformation" refers to rotation around its own coordinate axis, similar to rotation.

[0058] Self-rotation transformation is used to simulate the attitude change of obstacles. Assuming that the obstacle stands on the ground, the yaw angle mainly affects the attitude change of the obstacle. Considering that the ground is usually bumpy, setting a smaller random pitch or roll angle is beneficial to improving the diversity of the data set. Therefore, the yaw angle is set to [-30°, 30°], and the pitch and roll angles are set to [-6°, 6°]. Self-rotation transformation will cause the point cloud facing the radar to turn away from it, resulting in a reduction in the number of visible point clouds of the obstacle. Therefore, it is necessary to determine whether to perform this operation based on the orientation angle of the obstacle point cloud. The orientation angle of the obstacle is the angle θ between the target movement direction and the X-axis. Obstacles with an orientation angle that meets the [-15~15°] will be subjected to rotation transformation. Obstacles whose number of visible point clouds before and after the transformation is not less than the preset threshold T1 and are located within their placement range are retained. An embodiment of the present invention provides a self-rotation rule, which may include the following steps:

[0059] First, the coordinate origin of the environmental point cloud data is transformed to the center of gravity of the obstacle to obtain the obstacle coordinate system. For example, the range of the orientation angle of the obstacle suitable for rotation is [-15 to 15 degrees].

[0060] Secondly, the obstacle is controlled to rotate by a preset angle around at least one coordinate axis of the obstacle coordinate system to obtain a rotated obstacle. The rotation transformation can be achieved by using a single axis or a combination of multiple axes.

[0061] The preset angle includes at least one of the following: a yaw angle, a pitch angle, and a roll angle. For example, the yaw angle ranges from [-30°, 30°], the pitch angle ranges from [-6°, 6°], and the roll angle ranges from [-6°, 6°].

[0062] If the ratio of visible point clouds before and after the obstacle's rotational transformation is greater than or equal to a first preset threshold, and the obstacle's position falls within the placement range corresponding to its category, the obstacle after the rotational transformation is retained. Determining whether the obstacle's position falls within the placement range for its category can be done using the same method described above for translational transformation, which will not be further described here.

[0063] Holes caused by rotational transformations of obstacles are addressed using the aforementioned hole-filling method. Obstructed point cloud areas are eliminated according to the following rule: Count the number of point clouds within a single grid. If the grid is not empty and the number is greater than or equal to 2, then the point with the minimum distance in the point cloud is retained and the remaining points are eliminated. For example, T1 can be set to 0.5. It is assumed that each grid contains only one point before the transformation. If the number of point clouds in a grid after the transformation is greater than or equal to 2, the point within the grid with the minimum distance is retained. The minimum distance indicates that the point is closest to the radar, is easier to observe, and better reflects the target's contour information.

[0064] Scaling transformation is used to simulate the movement of obstacles. The center of gravity of the obstacle is used as the origin, and a local coordinate system parallel to the X and Y axes of the radar coordinate system is constructed. The direction angle of the obstacle is the angle θ between the target movement direction and the X axis. Figure 3 A schematic diagram of the local coordinate system of an obstacle in a scaling transformation is shown, where θ is the angle between the direction of motion and the X axis. An embodiment of the present invention provides a scaling rule that may include the following steps:

[0065] First, the random movement distance of the obstacle is obtained and decomposed into lateral and longitudinal movement distances based on the obstacle's orientation angle. In this embodiment, the scaling operation does not utilize a scaling factor, but instead randomly generates a movement distance to displace the obstacle along its orientation. The range of the random movement distance is set based on the type of obstacle. Assuming the obstacle is a car moving at a constant speed of 36 km / h and traveling for a maximum of 1 second, the movement range is set to 0 to 10 meters.

[0066] Secondly, the obstacle is moved according to the lateral movement distance and the longitudinal movement distance to obtain the transformed obstacle.

[0067] Then, if the transformed obstacle is far from the radar, the number of point clouds within the transformed obstacle grid is counted. If the grid is not empty and the number of point clouds is greater than the initial number of point clouds in the grid, the point with the smallest distance from the radar within the grid is retained and the remaining points are discarded. If the transformed obstacle is close to the radar, the transformed obstacle point cloud is upsampled. If the obstacle is close to the radar after the transformation, the point cloud is upsampled to simulate the measurement resolution of the obstacle when it moves far or near. Holes caused by the scaling transformation of the obstacle are filled using the hole filling method described above.

[0068] It should be noted here that when calculating the scale of the point cloud after the transformation of the labeled obstacle, the obstacle is divided into grids according to the transformed distance. In theory, a single grid contains only one point. If upsampling or downsampling is not performed according to the actual situation, the number of point clouds in the grid does not meet the requirements.

[0069] If an obstacle moves closer to the radar, the point cloud's measurement resolution will theoretically decrease, potentially leading to holes in the grid. Holes in the point cloud can be processed by interpolating the neighboring regions, a process known as upsampling. If the obstacle moves farther from the radar, the point cloud's measurement resolution increases, increasing the number of points in the grid and retaining the point with the minimum distance within the grid.

[0070] The occlusion operation needs to be implemented with the help of the above-mentioned translation operation. The embodiment of the present invention provides an occlusion rule, which may include the following steps:

[0071] If the target obstacle meets the occlusion condition, another obstacle is translated to the front of the target obstacle using the translation operation. This occlusion condition includes the initial occlusion state being either no occlusion or slight occlusion, and the number of point clouds being greater than a second preset threshold. In this embodiment, the occlusion operation is only applied to obstacles that are no occlusion or slight occlusion, and whose number of point clouds is greater than a threshold value T2. For example, T2 is 30.

[0072] If the ratio of the visible point cloud of the obscured target obstacle to the initial target obstacle is greater than or equal to a first preset threshold, the obscured target obstacle is retained and the status annotation information is updated.

[0073] Assuming an obstacle is in front of or behind another obstacle, a rotation matrix can be calculated based on the horizontal angle between the center of gravity of the two obstacles to transform one obstacle directly in front of the other. Fine-tuning the rotation left and right is then performed. When the visible point cloud ratio of the occluded obstacle is no less than the preset threshold T1, fine-tuning stops, the new obstacle is retained, and the occlusion status annotation information is updated.

[0074] The random noise addition operation can introduce the acquisition error of the random noise operation simulation device. The embodiment of the present invention provides a random noise addition operation rule, which can include the following steps:

[0075] Assuming random noise follows a Gaussian distribution, calculate the mean and variance of the obstacle point cloud and its neighborhood. Use the mode of the obstacle point cloud mean set as the mean of the random noise, and the mode of the obstacle point cloud variance set as the random noise variance. Add random noise to a preset percentage of randomly selected obstacle point clouds. For example, the preset obstacle percentage can be set to 5%, meaning that random noise is added to 5% of the randomly selected obstacle point clouds.

[0076] The partial area removal operation simulates the phenomenon of missing part of the point cloud due to low reflectivity obstacles, and can select the point cloud of the removal area according to the reflection intensity of the obstacle. The embodiment of the present invention provides a partial area removal operation rule, which can include the following steps:

[0077] The obstacle point clouds are sorted from smallest to largest by echo intensity amplitude. The obstacle point clouds are then removed in ascending order of echo intensity amplitude. Removal stops when the ratio of the remaining obstacle point cloud to the point cloud before removal meets a third preset threshold. The third preset threshold, T3, is a randomly varying value that may vary for different obstacles and is randomly generated between 0.5 and 1.0.

[0078] It should be noted that if holes appear during the data augmentation operation, the following point cloud hole filling steps are performed: search for the point with the minimum height in the area around the point cloud hole as the ground point; reconstruct the ground point based on the radar line emission angle and angular resolution corresponding to the point cloud hole.

[0079] The embodiments of the present invention can enhance obstacle data based on the actual radar acquisition situation. By generating data-enhanced point clouds in a guided manner, training difficulties caused by too many difficult samples are avoided. Multiple obstacle data enhancement methods are adopted to increase sample diversity and ensure data authenticity. After obstacle point cloud data enhancement, the annotation data is updated and the 3D bounding box is reconstructed to ensure the accuracy of the annotation data.

[0080] Figure 4 FIG. 1 is a schematic structural diagram of a point cloud-based radar data enhancement device according to an embodiment of the present invention, the device comprising:

[0081] An acquisition module 401 is configured to acquire environmental point cloud data collected by a radar; the environmental point cloud data includes a first obstacle;

[0082] A data enhancement module 402 is configured to perform a data enhancement operation on the first obstacle to generate new environmental point cloud data. The data enhancement operation includes at least one of the following: rotation, translation, scaling, occlusion, random noise addition, and partial area culling. The point cloud data of the second obstacle obtained by performing the data enhancement operation is consistent with the measurement resolution of the location of the second obstacle.

[0083] The point cloud-based radar data enhancement device provided by an embodiment of the present invention can perform data enhancement operations on obstacles in environmental point cloud data to generate new environmental point cloud data. The obstacles obtained by performing the data enhancement operation are consistent with the measurement resolution of their locations. By guiding the generation of new data sets, training difficulties caused by too many difficult samples are avoided. A variety of obstacle data enhancement methods are used to increase sample diversity, which not only increases sample diversity but also ensures the authenticity of the data.

[0084] Optionally, as an embodiment, the device further includes a bounding box reconstruction module, configured to reconstruct a bounding box from the point cloud data obtained by performing the data enhancement operation.

[0085] Optionally, as an embodiment, the translation operation includes the following steps: determining the initial rotation angle of the obstacle within the visible angle range; the visible angle range is determined by the horizontal field of view angle of the radar; controlling the obstacle to rotate around the Z axis of the radar coordinate system according to the rotation matrix corresponding to the rotation angle, and the Z axis of the radar coordinate system is perpendicular to the horizontal plane; if the ratio of the visible point cloud before and after the rotation transformation of the obstacle is greater than or equal to a first preset threshold, and the position of the obstacle after the rotation transformation is consistent with the placement range corresponding to its category, then the obstacle after the rotation transformation is retained.

[0086] Optionally, as an embodiment, the self-rotation operation includes the following steps: transforming the coordinate origin of the environmental point cloud data to the center of gravity of the obstacle to obtain an obstacle coordinate system; controlling the obstacle to rotate around at least one coordinate axis of the obstacle coordinate system by a preset angle to obtain a rotated obstacle; if the ratio of the visible point cloud before and after the rotation transformation of the obstacle is greater than or equal to a first preset threshold, and the position of the obstacle after the rotation transformation is consistent with the placement range corresponding to its category, then the obstacle after the rotation transformation is retained.

[0087] Optionally, as an embodiment, the scaling operation includes the following steps: obtaining a random movement distance of the obstacle, and decomposing the random movement distance into a lateral movement distance and a longitudinal movement distance according to the orientation angle of the obstacle; moving the obstacle according to the lateral movement distance and the longitudinal movement distance to obtain a transformed obstacle; if the transformed obstacle is away from the radar, counting the number of point clouds in the grid of the transformed obstacle, if the grid is not empty and the number of point clouds in the grid is greater than the number of initial point clouds in the grid, retaining the point in the grid with the smallest distance from the radar, and discarding the remaining points; if the transformed obstacle is close to the radar, performing an upsampling operation on the point cloud of the transformed obstacle.

[0088] Optionally, as an embodiment, the occlusion operation includes the following steps: if the target obstacle meets the occlusion condition, another obstacle is translated to the front of the target obstacle according to the translation operation; the occlusion condition includes that the initial occlusion state is no occlusion or slight occlusion, and the number of point clouds is greater than a second preset threshold; if the ratio of the visible point cloud of the occluded target obstacle to the initial target obstacle is greater than or equal to the first preset threshold, the occluded target obstacle is retained and the status annotation information is updated.

[0089] Optionally, as an embodiment, the operation of adding random noise includes the following steps: assuming that the random noise conforms to a Gaussian distribution, calculating the mean and variance of the obstacle point cloud and its respective neighborhood; taking the mode of the obstacle point cloud mean set as the mean of the random noise, and taking the mode of the obstacle point cloud variance set as the random noise variance; and adding the random noise to a preset proportion of randomly selected obstacle point clouds.

[0090] Optionally, as an embodiment, the partial area removal operation includes the following steps: sorting the point clouds of the obstacles according to the echo intensity amplitude from small to large; removing the point clouds of the obstacles in order from low to high according to the echo intensity amplitude; and stopping the removal when the ratio of the remaining point cloud of the obstacle to the point cloud of the obstacle before removal meets the third preset threshold condition.

[0091] Optionally, as an embodiment, the device further includes a hole filling module, which is used to: if a hole appears during the data enhancement operation, perform the following point cloud hole filling steps: search for the point with the minimum height in the surrounding area of the point cloud hole as the ground point; and reconstruct the ground point according to the radar line emission angle and angular resolution corresponding to the point cloud hole.

[0092] This embodiment further provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the various processes of the above-described embodiment and achieves the same technical effects. To avoid repetition, the details are not described here. The computer-readable storage medium may be, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0093] Of course, those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment method can be implemented by instructing a control device through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it may include the process of the above-mentioned embodiment of the point cloud-based radar data enhancement method, wherein the storage medium may be a memory, a disk, an optical disk, etc.

[0094] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0095] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other.

[0096] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A radar data enhancement method based on point cloud, characterized in that: include: Obtain environmental point cloud data collected by radar; The environmental point cloud data includes a first obstacle; Performing a data augmentation operation on the first obstacle to generate new environmental point cloud data; the data augmentation operation includes at least one of the following: rotation, translation, scaling, occlusion, random noise addition, and partial region culling, and the point cloud data of the second obstacle obtained by performing the data augmentation operation is consistent with the measurement resolution of the location of the second obstacle; The translation operation includes the following steps: Determining an initial rotation angle of an obstacle within a visual angle range; the visual angle range is determined by the horizontal field of view of the radar; Controlling the obstacle to perform a rotation transformation around the Z axis of the radar coordinate system according to the rotation matrix corresponding to the rotation angle, where the Z axis of the radar coordinate system is perpendicular to the horizontal plane; If the ratio of the visible point cloud before and after the rotation transformation of the obstacle is greater than or equal to a first preset threshold, and the position of the obstacle after the rotation transformation is within the placement range corresponding to its category, then the obstacle after the rotation transformation is retained; The self-spinning operation comprises the following steps: Transforming the coordinate origin of the environmental point cloud data to the center of gravity of the obstacle to obtain an obstacle coordinate system; Controlling the obstacle to rotate around at least one coordinate axis of the obstacle coordinate system by a preset angle to obtain a rotated obstacle; If the ratio of the visible point cloud before and after the rotation transformation of the obstacle is greater than or equal to a first preset threshold, and the position of the obstacle after the rotation transformation is within the placement range corresponding to its category, then the obstacle after the rotation transformation is retained; The scaling operation includes the following steps: Obtaining a random movement distance of an obstacle, and decomposing the random movement distance into a lateral movement distance and a longitudinal movement distance according to the orientation angle of the obstacle; Moving the obstacle according to the lateral movement distance and the longitudinal movement distance to obtain a transformed obstacle; If the transformed obstacle is far away from the radar, the number of point clouds in the grid of the transformed obstacle is counted. If the grid is not empty and the number of point clouds in the grid is greater than the number of initial point clouds in the grid, the point with the smallest distance from the radar in the grid is retained and the remaining points are discarded. If the transformed obstacle is close to the radar, the point cloud of the transformed obstacle is upsampled; The occlusion operation includes the following steps: If the target obstacle meets the occlusion condition, another obstacle is translated to the front of the target obstacle according to the translation operation; the occlusion condition includes that the initial occlusion state is no occlusion or slight occlusion, and the number of point clouds is greater than a second preset threshold; If the ratio of the visible point cloud of the obscured target obstacle to the initial target obstacle is greater than or equal to a first preset threshold, the obscured target obstacle is retained and the state annotation information is updated; The random noise adding operation comprises the following steps: Assuming that random noise conforms to Gaussian distribution, calculate the mean and variance of the obstacle point cloud and its respective neighborhood; The mode of the obstacle point cloud mean set is used as the mean of the random noise, and the mode of the obstacle point cloud variance set is used as the variance of the random noise; Adding the random noise to a point cloud of a randomly selected obstacle with a preset ratio; The partial area removal operation includes the following steps: Sort the point cloud of obstacles from small to large according to the amplitude of echo intensity; The point cloud of obstacles is removed in order from low to high according to the echo intensity amplitude; When the ratio of the remaining obstacle point cloud to the obstacle point cloud before removal meets the third preset threshold condition, the removal is stopped.

2. The method according to claim 1, characterized in that After performing the data augmentation operation on the first obstacle, the method further includes: The bounding box of the point cloud data obtained by performing data augmentation operations is reconstructed.

3. The method according to claim 1 or 2, characterized in that The method further comprises: If holes appear during data augmentation, perform the following point cloud hole filling steps: Search for the point with the minimum height in the surrounding area of the point cloud hole as the ground point; The ground points are reconstructed based on the radar line emission angles and angular resolutions corresponding to the point cloud holes.

4. A radar data enhancement device based on point cloud, characterized in that: The method according to any one of claims 1 to 3, wherein the device comprises: An acquisition module, configured to acquire environmental point cloud data collected by a radar; the environmental point cloud data includes a first obstacle; A data enhancement module is configured to perform a data enhancement operation on the first obstacle to generate new environmental point cloud data; the data enhancement operation includes at least one of the following: rotation, translation, scaling, occlusion, random noise addition, and partial area culling operations; and the point cloud data of the second obstacle obtained by performing the data enhancement operation is consistent with the measurement resolution of the location of the second obstacle.

Images