Radar point cloud data enhancement method, device, smart car and storage medium

Abstract

The present invention relates to a radar point cloud data enhancement method, device, smart car, and storage medium. The method comprises: enumerating annotated boxes in point cloud data, screening out annotated boxes whose portions fall within the radar's near-field blind spot, obtaining a plurality of first target boxes and point cloud data within the first target boxes based on the screened annotated boxes, wherein all the first target boxes and the point cloud data within the first target boxes constitute a field of view boundary scene dataset; selecting a second target box from the field of view boundary scene dataset, and when it is determined that the second target box is to be placed into a target frame of the point cloud data based on its position and that the second target box does not conflict with an object in the target frame, placing the second target box and the point cloud data within the second target box into the target frame to enhance the point cloud data. This method can effectively enhance near-field point cloud data and improve the perception performance of a target perception model in near-field of view boundary scenarios.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a radar point cloud data enhancement method, device, intelligent vehicle, and storage medium. Background Art

[0002] As LiDAR technology continues to improve, it plays an increasingly important role in advanced intelligent driving. It's an essential sensor in vehicles with Level 3 intelligent driving capabilities and above. By processing the 3D point cloud data collected by LiDAR using a deep learning target perception model (hereinafter referred to as the target perception model), the vehicle's surrounding environment can be accurately perceived.

[0003] LiDARs generally have a field of view. For example, the AT128 has a field of view of 120°×25.4°. The area outside the field of view is called a blind spot. In addition, due to the limitations of the installation position of the LiDAR, in tunnels, curves, ramps, cut-ins, cut-outs, overtaking, etc., part of the field of view is blocked, which will also form a blind spot. When the target just enters but does not completely enter the field of view of the LiDAR, the target will be broken. This scene is called the FOV (Field of View) boundary scene (i.e., the field of view boundary scene). The data recorded by the target in the field of view boundary scene is called FOV boundary data. Since the blind spot of the LiDAR is usually at close range, the FOV boundary data is also called close-range point cloud data.

[0004] However, in the point cloud data samples currently used to train target perception models, the amount and proportion of close-range point cloud data are very small. The target perception model lacks training data in scenes with field of view boundaries, resulting in missed detections or inaccurate detection results when the target just enters the field of view of the lidar during autonomous driving.

[0005] Existing point cloud data augmentation methods extract point cloud data and add it to the training data based on whether the number of points within the annotation box meets the requirements. However, in scenes at the boundaries of the field of view, the number of points within the annotation box cannot represent the characteristics of the target, resulting in inaccurate extracted point cloud data. Moreover, adding point cloud data to the training data is prone to data conflicts. As a result, it is impossible to effectively enhance close-range point cloud data, resulting in poor performance of the target perception model in scenes at close distances and boundaries of the field of view. Summary of the Invention

[0006] One of the objectives of the present invention is to provide a radar point cloud data enhancement method to solve the problem in the prior art that close-range point cloud data cannot be effectively enhanced, resulting in poor perception performance of the target perception model in close-range field of view boundary scenarios; a second objective is to provide a radar point cloud data enhancement device; a third objective is to provide an intelligent car; and a fourth objective is to provide a computer-readable storage medium.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] Enumerate the labeled boxes in the point cloud data, filter out the labeled boxes whose parts are within the radar's close-range blind spot, and obtain a plurality of first target boxes and point cloud data within the first target boxes based on the filtered labeled boxes. All of the first target boxes and the point cloud data within the first target boxes constitute a field of view boundary scene dataset;

[0009] A second target frame is selected from the field of view boundary scene dataset. When it is determined that the second target frame is to be placed in the target frame of the point cloud data based on the position of the second target frame, and the second target frame does not conflict with the object in the target frame, the second target frame and the point cloud data within the second target frame are placed in the target frame to enhance the point cloud data.

[0010] According to the above technical means, the point cloud data is first filtered within the annotated frame, accurately extracting the first target frame that matches the field of view boundary scene, obtaining the field of view boundary scene dataset. Then, when the second target frame in the field of view boundary scene dataset is added to the target frame, only the second target frame that does not conflict with the objects in the target frame is added to the target frame, making the enhanced sample data more realistic. Therefore, through the two steps of screening and conflict determination, the close-range point cloud data can be effectively enhanced to meet the data requirements of the target perception model training and improve the perception performance of the target perception model in close-range field of view boundary scenes.

[0011] Furthermore, the step of filtering out the marked boxes of the partial areas within the radar's close-range blind spot includes:

[0012] From a bird's-eye view perspective, a first length between a rear axle center of an ego vehicle and a center of a radar coordinate system is calculated, a second length of a close-range blind spot of the radar in a direction of travel of the ego vehicle is obtained, and a third length of an object represented by the annotation box is obtained, wherein the ego vehicle is equipped with a radar;

[0013] With the center of the rear axle of the own vehicle as the center of the coordinate system, a boundary range in the direction of travel of the own vehicle is obtained based on the first length, the second length, and the third length. When the coordinates of the center of the annotation box along the direction of travel of the own vehicle are within the boundary range, the annotation box is set as a annotation box whose portion is within the close-range blind spot of the radar.

[0014] According to the above technical means, by determining the boundary range of the object represented by the annotation box relative to the object, it is possible to simply and conveniently determine whether the annotation box conforms to the field of view boundary scene and whether the annotation box has a partial area within the radar's close-range blind spot.

[0015] Furthermore, the step of obtaining a plurality of first target frames and point cloud data within the first target frames based on the filtered annotation frames includes:

[0016] Obtaining the projection area of ​​the point cloud within the filtered annotation box on each projection plane;

[0017] When the maximum value of the projected area is greater than a preset area threshold, the filtered annotation box is set as the first target box and the point cloud data in the first target box is acquired.

[0018] According to the above technical means, the annotation box is further filtered by projecting the area, which can filter out difficult examples and avoid the negative impact of difficult example data on the target perception model during training.

[0019] Further, determining whether the second target frame conflicts with an object in the target frame when the second target frame is placed into the target frame of the point cloud data based on the position of the second target frame includes:

[0020] When an object is detected in the target frame based on the position and range of the second target frame, determining that the second target frame conflicts with the object in the target frame;

[0021] When an intersection is detected between the second target frame and the annotation frame or the target frame in the target frame, determining that the second target frame conflicts with the object in the target frame;

[0022] When it is detected that the second target frame is placed in the target frame and is blocked, determining that the second target frame conflicts with the object in the target frame;

[0023] Otherwise, it is determined that the second target frame does not conflict with the object in the target frame.

[0024] According to the above technical means, by judging whether there is a conflict from three angles: whether the second target frame overlaps with the object, whether there is an intersection between the second target frame and the target frame or annotation frame of the target frame, and whether the second target frame is blocked, it is possible to accurately determine whether there will be a data conflict between the point cloud data in the second target frame and the original training data, so that the enhanced close-range point cloud data is more consistent with the actual situation, and the enhanced data is more effective.

[0025] Further, detecting whether the second target frame is blocked when placed in the target frame includes:

[0026] Projecting the second target frame and each annotation frame in the target frame onto respective projection planes, and calculating the intersection area between the projection of the second target frame and the projection of each annotation frame in the target frame on each projection plane;

[0027] When any of the intersection areas is equal to the projected area of ​​the second target frame, it is determined that the second target frame is blocked when placed in the target frame; otherwise, it is determined that the second target frame is not blocked when placed in the target frame.

[0028] According to the above technical means, whether the second target frame is blocked can be determined intuitively and accurately based on the intersection area and the projection area.

[0029] Further, based on the position and range of the second target frame, detecting whether there is an object in the target frame includes:

[0030] Based on the position of the second target frame, obtaining the number of point clouds within the range of the second target frame in the target frame;

[0031] When the number of the point clouds exceeds a preset number threshold, it is determined that an object exists in the target frame; otherwise, it is determined that no object exists in the target frame.

[0032] According to the above technical means, whether the second target frame overlaps with the object in the target frame is determined by the number of point clouds within the range of the second target frame. The calculation is simple and convenient, and there is no need to model based on point cloud data.

[0033] Further, detecting whether there is an intersection between the second target frame and the annotation frame or the target frame in the target frame includes:

[0034] From a bird's-eye view perspective, obtaining a first projection area of ​​the second target frame, a second projection area of ​​the annotation frame, and a third projection area of ​​the target frame placed in the target frame;

[0035] respectively calculating a first intersection-and-union ratio of the first projected area to the second projected area and a second intersection-and-union ratio of the first projected area to the third projected area;

[0036] When the first intersection-in-union ratio is not equal to zero or the second intersection-in-union ratio is not equal to zero, it is determined that there is an intersection between the second target box and the label box or the target box in the target frame; otherwise, it is determined that there is no intersection between the second target box and the label box or the target box in the target frame.

[0037] According to the above technical means, by calculating the intersection-over-union ratio between the projection of the second target frame and the projection of the labeled frame or target frame in the target frame from a bird's-eye view perspective, it is possible to simply and quickly determine whether there is an intersection between the second target frame and the labeled frame or target frame in the target frame.

[0038] Further, placing the second target frame into the target frame includes:

[0039] Searching for the target category's annotation box and target box in the target frame to obtain a total number;

[0040] When the total number is less than or equal to a preset threshold, the second target frame is placed into the target frame.

[0041] According to the above technical means, by counting the total number of annotation boxes and target boxes under the target category, it is possible to prevent too many annotation boxes and target boxes of the same category in a target frame, thereby improving the balance and data quality of the enhanced data.

[0042] A radar point cloud data enhancement device, comprising:

[0043] A dataset module is configured to enumerate the annotation boxes in the point cloud data, filter out the annotation boxes whose parts are within the radar's close-range blind spot, and obtain a plurality of first target boxes and point cloud data within the first target boxes based on the filtered annotation boxes. All of the first target boxes and the point cloud data within the first target boxes constitute a field of view boundary scene dataset.

[0044] An enhancement module is used to select a second target frame from the field of view boundary scene data set, and when it is determined that the second target frame is to be placed in the target frame of the point cloud data based on the position of the second target frame, and the second target frame does not conflict with the object in the target frame, the second target frame and the point cloud data in the second target frame are placed in the target frame to enhance the point cloud data.

[0045] Furthermore, the enhancement module includes a conflict determination unit, which is used to determine that when an object is detected in the target frame based on the position and range of the second target frame, the second target frame is in conflict with the object in the target frame; when an intersection is detected between the second target frame and the annotation frame or target frame in the target frame, the second target frame is in conflict with the object in the target frame; when it is detected that the second target frame is occluded when placed in the target frame, the second target frame is in conflict with the object in the target frame; otherwise, the second target frame is determined not to be in conflict with the object in the target frame.

[0046] According to the above technical means, by determining whether there will be data conflicts between the point cloud data in the second target frame and the original training data, the enhanced close-range point cloud data is made more consistent with the actual situation, and the enhanced data is more effective.

[0047] Furthermore, the data set module includes a screening unit, which is used to calculate a first length between the center of the rear axle of the vehicle and the center of the radar coordinate system from a bird's-eye view perspective, calculate a second length of the radar's close-range blind spot in the direction of travel of the vehicle, and obtain a third length of the object represented by the annotation box, wherein the vehicle is equipped with a radar; with the center of the rear axle of the vehicle as the center of the coordinate system, based on the first length, the second length and the third length, obtain a boundary range in the direction of travel of the vehicle; when the coordinate of the center of the annotation box along the direction of travel of the vehicle is within the boundary range, the annotation box is set as a annotation box whose partial area is within the close-range blind spot of the radar.

[0048] According to the above technical means, it is faster and more effective to determine whether the marked box is a marked box whose part is located within the close-range blind spot of the radar through the boundary range.

[0049] A smart car includes a memory, a processor, and a radar point cloud data enhancement program stored in the memory and executable on the processor, wherein the radar point cloud data enhancement program, when executed by the processor, implements any one of the steps of the above-mentioned radar point cloud data enhancement method.

[0050] A computer-readable storage medium stores a radar point cloud data enhancement program, which, when executed by a processor, implements the steps of any one of the above-mentioned radar point cloud data enhancement methods.

[0051] The beneficial effects of the present invention are as follows: the enhanced data conforms to the field of view boundary scene, and the enhanced data is more consistent with the actual situation. Using the enhanced sample data of this embodiment to train the target perception model can greatly improve the accuracy of the target perception model in target perception in the FOV boundary scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Flowchart of a radar point cloud data enhancement method according to an embodiment of the present invention;

[0053] Figure 2 A flowchart of a filter labeling box according to an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of the scene from a bird's-eye view;

[0055] Figure 4 This is a flowchart of an embodiment of the present invention for determining whether a second target frame conflicts with an object in a target frame;

[0056] Figure 5 Schematic diagram of the intersection-over-union ratio according to an embodiment of the present invention;

[0057] Figure 6 Schematic diagram of a radar point cloud data enhancement device according to an embodiment of the present invention;

[0058] Figure 7 Schematic diagram of a smart car according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following describes the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art will readily appreciate the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through various other specific embodiments, and the various details in this specification may be modified or altered based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are intended only to illustrate the present invention and are not intended to limit the scope of protection of the present invention.

[0060] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0061] There are currently two solutions to the problem of insufficient point cloud data: 1. Resample the tail data, but this will easily lead to overfitting when training the target perception model; 2. Perform data augmentation, that is, extract the annotation box and the point cloud within the box from the existing data based on the number of point clouds within the annotation box, and then directly add the extracted annotation box and the point cloud within the box to the desired training data, so that the number of annotation boxes in the added sample reaches the desired number of annotation boxes. However, when extracting the annotation box and the point cloud data within the box, they are not obtained according to the field of view boundary scene, and the acquired point cloud data is not close-range point cloud data; moreover, when collecting target information, the number of point clouds within the box is used to determine whether the current target meets the training requirements. However, in the field of view boundary scene, the number of point cloud data within the box is insufficient to represent the characteristics of the target, which may cause some difficult examples that are difficult for the deep learning network to learn to be added to the training data, which has a negative impact on the target perception model; moreover, directly adding the extracted annotation box and the point cloud data within the box to the training data may cause data conflicts with the original training data. Therefore, the current data enhancement method cannot enhance close-range point cloud data, and the effect of point cloud data enhancement is not ideal, and cannot improve the perception performance of the target perception model in the field of view boundary scene.

[0062] In order to solve the above problems, the present invention provides a radar point cloud data enhancement method, which obtains a field of view boundary scene data set by quantifying the fov boundary range. When adding close-range point cloud data to training data, only close-range point cloud data that will not conflict with the training data is added to the training data, so that the enhanced sample data is more in line with reality, and more meets the data requirements of target perception model training, thereby improving the perception performance of the target perception model in fov boundary scenes. It should be noted that the radar point cloud data enhancement method of the present invention is not only applicable to fov boundary scenes under the blind spot of the field of view angle, but also applicable to fov boundary scenes under occlusion scenes such as tunnels, curves, ramps, cut-ins, cut-outs, overtaking, etc.; the radar in the present invention refers to a radar device that can collect point cloud data, which can be a laser radar or a millimeter wave radar.

[0063] like Figure 1 As shown, an embodiment of the present invention provides a radar point cloud data enhancement method, which is deployed on an electronic terminal and is used to perform close-range point cloud data enhancement on point cloud data samples used to train a target perception model. This solves the problem of insufficient FOV boundary data causing poor and unstable perception performance of the target perception model in close-range FOV boundary scenarios, thereby improving the perception performance of the trained target perception model. Specifically, the above method includes the following steps:

[0064] Step S100: enumerating the annotated boxes in the point cloud data, filtering out the annotated boxes whose parts are within the radar's close-range blind spot, and obtaining a plurality of first target boxes and point cloud data within the first target boxes based on the filtered annotated boxes. All the first target boxes and the point cloud data within the first target boxes constitute a field of view boundary scene dataset;

[0065] Point cloud data refers to annotated sample data used to train object perception models. Each frame of point cloud data is pre-annotated with several annotation boxes. Each annotation box delineates an object required for target detection during autonomous driving, such as a truck, car, pedestrian, or bicycle. The annotation box information includes its location and extent. Based on this information, the point cloud data within the annotation box can be retrieved from the point cloud data. The first target box refers to the annotation box at the edge of the field of view.

[0066] In some frames of point cloud data, there may be some labeled boxes whose parts are within the radar's short-range blind spot. In other words, the radar can scan the part of the object corresponding to the labeled box. For example, when a truck enters the radar's short-range blind spot from behind the vehicle, the radar cannot scan the front area of ​​the vehicle, but can scan the body area.

[0067] First, select some or all frames of point cloud data and filter all the annotation boxes within those frames. If part of the annotation box falls within the radar's close-range blind spot, the annotation box is set as the first target box and the point cloud data within it is read. All first target boxes and the point cloud data within them constitute the field of view boundary scene dataset.

[0068] In this embodiment, the specific steps of screening the marked box are as follows: Figure 2 Shown, including:

[0069] Step S110: Calculate a first length between the center of the rear axle of the ego vehicle and the center of the radar coordinate system from a bird's-eye view perspective, obtain a second length of the radar's close-range blind spot in the ego vehicle's direction of travel, and obtain a third length of the object represented by the annotation box;

[0070] Step S120: With the center of the rear axle of the vehicle as the center of the coordinate system, based on the first length, the second length, and the third length, a boundary range of the vehicle's traveling direction is obtained. When the coordinates of the center of the annotation box along the vehicle's traveling direction are within the boundary range, the annotation box is set to a annotation box whose partial area is within the radar's close-range blind spot.

[0071] The self-driving vehicle refers to an intelligent driving vehicle equipped with radar. Figure 3It is a schematic diagram of a scene from a bird's-eye view. The lower left rectangular frame represents the ego vehicle. The triangle above the ego vehicle represents the close-range blind area of the radar. The lower right rectangular frame represents the state of other vehicles at the moment when they just enter the FOV of the ego vehicle. The upper right dashed rectangular frame represents the state of other vehicles when they are about to drive out of the FOV of the ego vehicle.

[0072] The center of the rear axle of the ego vehicle is the coordinate axis center of the perception sensor. The forward direction of the ego vehicle is the positive direction of the x-axis ( Figure 3 in the middle is vertically upward), the left side direction of the ego vehicle is the positive direction of the y-axis, and the vertical direction of the ego vehicle is the positive direction of the z-axis. z is the first length from the center of the radar coordinate system to the center of the rear axle of the ego vehicle, a is the second length of the close-range vision blind area range of the radar in the driving direction of the ego vehicle, l is the third length of other vehicles, and the boundary range is [z + a - l / 2, z + a + l / 2]. It should be emphasized that the boundary range is not fixed. Different lengths of other vehicles result in different boundary ranges.

[0073] Since the center of the rear axle of the ego vehicle is the coordinate axis center of the perception sensor, assuming the center point coordinates of the annotation box representing other vehicles are (x, y, z), along the driving direction of the ego vehicle, the coordinate x of the center of the annotation box is the distance between the center of the annotation box and the center of the rear axle of the ego vehicle. Then when x satisfies z + a - l / 2 < x < z + a + l / 2, it means that part of the area of the annotation box is within the close-range vision blind area range of the radar, and this annotation box is the FOV boundary target box.

[0074] By determining the boundary range, it is possible to accurately determine whether the annotation box is an annotation box in the field of view boundary scenario, obtain the point cloud data within the annotation box, and establish a field of view boundary scenario data set.

[0075] In some cases, although the radar point cloud reflected from the surface of the object represented by the annotation box is relatively sparse and the quantity is relatively small, it can generally describe the contour features of the object. At this time, filtering and extracting the target by the quantity of the point cloud may filter out valuable data beneficial to training the model, and the determination based on the boundary range of the present invention can avoid this situation.

[0076] However, in other cases, although the radar point cloud reflected from the surface of the object represented by the annotation box is particularly dense and the quantity is abundant, it may only represent extremely small local features of the object. For example, the reflected point cloud is only the rearview mirror of other vehicles. At this time, these point clouds are called difficult examples. Using such difficult examples to train the target perception model will not only not improve the perception effect of the target perception model, but will instead decline.

[0077] Therefore, to avoid adding difficult examples to the training data, this embodiment also performs a difficult example filter after selecting the annotation box. The projected area of ​​the point cloud on the projection plane is used to determine whether the point cloud within the annotation box can be used as augmented data. Specifically, the projected area of ​​the point cloud within the annotation box on each projection plane (xy projection plane, xz projection plane, yz projection plane) is first obtained. When the maximum of these projected areas is greater than a preset area threshold, the annotation box is set as the first target box.

[0078] Specifically, the point cloud within the annotation box is extracted, and then the coordinates (x, y, z) of the point cloud are projected onto the yz, xz, and xy projection planes of the vehicle coordinate system. Then, the maximum and minimum values ​​of the point cloud coordinates in each projection plane are taken as the difference to obtain the edge of the rectangle formed by the feature point cloud, and the projection area is obtained after multiplication. Assume that the area formed by the point cloud within the annotation box on the three projection planes in the vehicle coordinate system is {s yz , s xz , s xy}, which is calculated as follows:

[0079] s yz =(y max –y min )*(z max -z min );

[0080] s xz =(x max –x min )*(z max -z min );

[0081] s xy =(x max –x min )*(y max -y min );

[0082] Calculate {s yz , s xz , s xy}, assuming that the area filtering threshold is S t , will {s yz , s xz , s xy The maximum value in} and the area filtering threshold S t Compare, if max{s yz , s xz , s xy}>S t, it means that the point cloud features in the annotation box are sufficient to represent the target and can be used as target enhancement data. The annotation box is set as the first target box, and the position, range and point cloud data of the first target box are saved to the field of view boundary scene dataset. Among them, the area filtering threshold can be determined according to the object category corresponding to the annotation box. If max{s yz , s xz , s xy}≤S t , then discard the annotation box, get the next annotation box, and start filtering again.

[0083] In one embodiment, the data in the field of view boundary scene data set is classified and stored. After the first target frame and the point cloud in the first target frame that meet the requirements are extracted, the category of the object corresponding to the first target frame is determined and then placed in the field of view boundary scene data set, and finally the field of view boundary scene data set is classified.

[0084] By performing hard example filtering, meaningless samples can be filtered out when establishing a field of view boundary scene dataset, which can avoid adding hard examples to the training data. This makes the point cloud data used for enhancement more representative of the actual target, improves the effectiveness of the field of view boundary scene dataset, and reduces the negative impact of hard example data on the training target perception model.

[0085] Step S200: Select a second target frame from the field of view boundary scene data set. When it is determined that the second target frame is placed in the target frame of the point cloud data based on the position of the second target frame, and the second target frame does not conflict with the object in the target frame, the second target frame and the point cloud data in the second target frame are placed in the target frame to enhance the point cloud data.

[0086] After obtaining the field of view boundary scene data set, several second target frames can be randomly selected from the field of view boundary scene data set according to the target category that needs to be enhanced, or several second target frames can be randomly selected directly from the field of view boundary scene data set without considering the target category. Then, a conflict judgment is performed on each second target frame. When the second target frame is placed in the target frame of the point cloud data based on the position of the second target frame, and the second target frame does not conflict with the object in the target frame, the second target frame and the point cloud data in the second target frame are placed in the target frame. Among them, the conflict judgment process can be determined according to the scene when the point cloud data sample is collected, as long as the second target frame that is determined not to conflict does not contradict the real scene after being placed in the target frame, and the point clouds surrounded by the second target frame will also be collected under real circumstances.

[0087] In this embodiment, conflict determination primarily considers the following aspects: 1. Whether the second target frame placed based on its position overlaps with other unlabeled objects in the target frame. For example, when a second target frame is added to the target frame, its position coincides with that of a roadside tree, and the added point cloud will overlap with the tree's feature point cloud. However, this situation does not occur in real training data, so a conflict is determined in such cases. 2. Whether multiple second target frames placed before and after the target frame are located at the same position or intersect, or whether the placed second target frame and the labeled frame in the target frame are located at the same position or intersect. This situation does not occur in real training data, so a conflict is determined in such cases. 3. Whether, after the second target frame is placed, multiple objects are located before and after it at a certain viewing angle. For example, if there is a large truck nearby, and the second target frame placed directly represents a small car, the large truck happens to completely obscure the placed car. Normally, a car in the distance obscured by the large truck will not reflect its point cloud features, which is inconsistent with the actual situation. Therefore, a conflict is determined in such cases.

[0088] like Figure 4 As shown, determining whether the second target frame conflicts with the object in the target frame includes the following specific steps:

[0089] Step S210: when an object is detected in the target frame based on the position and range of the second target frame, determining that the second target frame conflicts with the object in the target frame;

[0090] This embodiment determines whether an object exists based on the number of point clouds at the location of a second target frame in the target frame. Specifically, the second target frame is simulated and placed into the target frame based on its location. The range of the second target frame serves as the detection range of the target frame. The number of point clouds within this detection range in the target frame is obtained. If the number of point clouds exceeds a preset threshold, an object is determined to be present; otherwise, the object is determined to be absent.

[0091] Assume the preset threshold is point_n and the number of point clouds is n. If n > point_n, it means that an object already exists at that location in the target frame, making it unsuitable to add a second target frame. A new second target frame needs to be selected. Due to radar point cloud noise, even if there is no object at that location, a small amount of point cloud may exist. Therefore, the value of point_n is generally not set to 0.

[0092] Step S220: when it is detected that there is an intersection between the second target frame and the annotation frame or the target frame in the target frame, it is determined that the second target frame conflicts with the object in the target frame;

[0093] The annotated box in the target frame refers to the original annotation information in the point cloud data sample. The target box in the target frame refers to the second target box of the field of view boundary scene dataset that was previously placed in the target frame. Because the second target box of the field of view boundary scene dataset was extracted from a different data frame and may be located in the same or adjacent positions, conflict determination requires not only checking whether the second target box and the annotated box in the target frame intersect, but also checking whether the second target box intersects with the target frame.

[0094] Specifically, from a bird's-eye view (e.g. Figure 3 Schematic perspective), respectively obtain the first projection area of ​​the second target frame, the second projection area of ​​the annotation frame and the third projection area of ​​the target frame placed in the target frame; respectively calculate the first intersection-and-union ratio of the first projection area to the second projection area, and the second intersection-and-union ratio of the first projection area to the third projection area; when either the first intersection-and-union ratio and the second intersection-and-union ratio are not equal to 0, it indicates that there is an intersection between the second target frame and the annotation frame or the target frame in the target frame, and it is determined that there is a conflict, it is not appropriate to add the second target frame, and it is necessary to reselect another second target frame; otherwise, when the first intersection-and-union ratio and the second intersection-and-union ratio are both equal to 0, it indicates that there is no intersection between the second target frame and the annotation frame or the target frame in the target frame, and it is determined that there is no conflict.

[0095] Among them, the calculation method of intersection-union ratio iou is as follows Figure 5 As shown, S1 represents the area of ​​the first target box / annotation box, S2 represents the area of ​​the second target box / annotation box, and S12 represents the intersection area of ​​the two target boxes / annotation boxes, then iou=S12 / (S1+S2-S12).

[0096] Step S230: when it is detected that the second target frame is placed in the target frame and is blocked, determining that the second target frame conflicts with the object in the target frame;

[0097] When the second target frame is blocked in the target frame, it means that the target corresponding to the second target frame cannot be scanned by the radar under normal circumstances at a certain viewing angle. This means that it is not suitable to add the second target frame and another second target frame needs to be selected.

[0098] Specifically, the method for determining whether it is blocked is as follows: project the second target frame and each annotation frame in the target frame onto respective projection planes, such as xy, xz, yz, etc., and calculate the intersection area S between the projection of the second target frame and the projection of the annotation frame in each target frame on each projection plane. 相交For example, first calculate the intersection area of ​​the second target frame and the label frame 1, label frame 2, and label frame 3 in the target frame on the xy projection plane, and obtain three intersection areas. Then calculate three more intersection areas on the xz projection plane, and finally calculate three more intersection areas on the yz projection plane. When any intersection area S 相交 When the projected area of ​​the second target frame is equal, that is, S 相交 / S 第二目标 =1, it means that the second target frame conflicts with other annotation frames in the target frame, and it is determined that the second target frame is blocked when placed in the target frame, and it is not suitable to add the second target frame; when all the intersection areas S 相交 When the projected area of ​​the first target frame is not equal to the second target frame, that is, S 相交 / S 第二目标 =1, it means that the second target frame does not conflict with other annotation frames in the target frame, and it is determined that the second target frame is not blocked when placed in the target frame. It should be noted that the intersection area S 相交 The projection area S of the second target frame 第二目标 The comparison is made on the same projection plane.

[0099] Step S240: Otherwise, determine that the second target frame does not conflict with the object in the target frame.

[0100] When the three detections in the above steps S210-S230 are all determined to be non-conflicting, it is determined that the second target frame does not conflict with the object in the target frame, and the second target frame and the point cloud data in the second target frame are added to the target frame to form enhanced data.

[0101] By performing the above-mentioned conflict judgment, the second target box and the point cloud within the second target box in the field of view boundary scene dataset are added to the training sample data in a targeted manner. This not only makes up for the shortcoming of insufficient data volume in boundary scenes, but also makes the enhanced data more consistent with the actual situation.

[0102] The specific process of placing the second target frame into the target frame in this embodiment is as follows: first, determine whether there is a point cloud representing other objects at the position where the second target frame is placed. If so, ignore the second target frame and the point cloud, and select the next second target frame for judgment; then calculate the intersection-and-union ratio between the second target frame and the target frame already placed in the target frame, and the intersection-and-union ratio between the second target frame and the labeled frame in the target frame. When any intersection-and-union ratio is not equal to 0, ignore the second target frame and the point cloud, select the next second target frame and restart from the first judgment (whether there is a point cloud representing other objects); then, uniformly project the second target frame and the labeled frame in the target frame onto the yz, xz, and xy planes, calculate the intersection area of ​​the second target frame and the labeled frame under each projection plane, and then calculate the ratio of the intersection area to the target frame area to determine whether the second target frame is blocked when it is placed in the target frame. If the ratio is equal to 1, it means that the second target frame will be completely blocked by the annotation frame in the target frame. At this time, the second target frame and point cloud are ignored, and the next second target frame is selected and restarted from the first judgment. Only when the above three judgments meet the requirements, the second target frame and the point cloud data in the second target frame are placed in the target frame. Then, several second target frames selected from the field of view boundary scene data set are placed in the current target frame in turn, and then several second target frames selected from the field of view boundary scene data set are used to place the next target frame, and the cycle is repeated to achieve close-range point cloud data enhancement. The enhanced sample data is more in line with reality and better meets the data requirements of model training.

[0103] In summary, this embodiment determines the boundary range, filters out boundary data based on the boundary range, and further filters the boundary data based on the projected area to filter out difficult cases that have a negative impact on the target perception model. When using boundary data to enhance point cloud data samples, it also performs conflict determination, making the enhanced data more consistent with the actual situation. Using the enhanced sample data of this embodiment to train the target perception model can greatly improve the accuracy and stability of the target perception model in FOV boundary scenarios.

[0104] In some scenarios, considering the balance of the amount of data of the same type of target in the same point cloud frame, the number of objects of the same type in the same frame is also limited when the second target frame and the point cloud in the second target frame are placed in the target frame to enhance the point cloud data. Specifically, when placing the second target frame in the target frame, the target category (such as cars, pedestrians, bicycles, etc.) is first searched for the annotation frame and target frame in the target frame, and the total number of the two is accumulated. When the total number is less than or equal to the preset threshold, the point cloud data of the second target frame and the second target frame are placed in the target frame. Otherwise, it means that there is no need to add objects of the target category to the target frame, and the enhancement continues on the next target frame.

[0105] like Figure 6 As shown, corresponding to the above radar point cloud data enhancement method, an embodiment of the present invention further provides a radar point cloud data enhancement device, the device comprising:

[0106] The data set module 600 is configured to enumerate the annotated boxes in the point cloud data, select the annotated boxes whose portions are within the radar's close-range blind spot, and obtain a plurality of first target boxes and point cloud data within the first target boxes based on the selected annotated boxes. The first target boxes and the point cloud data within the first target boxes constitute a field of view boundary scene dataset.

[0107] The enhancement module 610 is used to select a second target frame from the field of view boundary scene data set, and when it is determined that the second target frame is to be placed in the target frame of the point cloud data based on the position of the second target frame, and the second target frame does not conflict with the object in the target frame, the second target frame and the point cloud data in the second target frame are placed in the target frame to enhance the point cloud data.

[0108] Optionally, the enhancement module 610 includes a conflict determination unit 611, which is used to determine that when an object is detected in the target frame based on the position and range of the second target frame, the second target frame is in conflict with the object in the target frame; when an intersection is detected between the second target frame and the annotation frame or target frame in the target frame, the second target frame is in conflict with the object in the target frame; when it is detected that the second target frame is occluded when placed in the target frame, the second target frame is in conflict with the object in the target frame; otherwise, the second target frame is determined not to be in conflict with the object in the target frame.

[0109] Optionally, the data set module 600 includes a filtering unit 601, which is used to calculate the first length between the rear axle center of the vehicle and the center of the radar coordinate system from a bird's-eye view perspective, calculate the second length of the radar's close-range blind spot in the direction of travel of the vehicle, and obtain the third length of the object represented by the annotation box; with the rear axle center of the vehicle as the center of the coordinate system, based on the first length, the second length and the third length, obtain the boundary range in the direction of travel of the vehicle; when the coordinates of the center of the annotation box along the direction of travel of the vehicle are within the boundary range, set the annotation box as a annotation box whose partial area is within the close-range blind spot of the radar.

[0110] Specifically, in this embodiment, the specific functions of each module of the above-mentioned radar point cloud data enhancement device can refer to the corresponding description in the above-mentioned radar point cloud data enhancement estimation method, and will not be repeated here.

[0111] Based on the above embodiment, the present invention also provides a smart car. Figure 7 As shown, the smart car includes a processor and memory connected via a system bus. The processor of the smart car is used to provide computing and control capabilities. The memory of the smart car includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a radar point cloud data enhancement program. The internal memory provides an operating environment for the operating system and radar point cloud data enhancement program in the non-volatile storage medium. When executed by the processor, the radar point cloud data enhancement program implements the steps of any of the above-described radar point cloud data enhancement methods.

[0112] An embodiment of the present invention further provides a computer-readable storage medium, on which a radar point cloud data enhancement program is stored. When the radar point cloud data enhancement program is executed by a processor, the steps of any one of the radar point cloud data enhancement methods provided in the embodiments of the present invention are implemented.

[0113] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0114] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "N" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0115] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or N executable instructions for implementing a custom logical function or step of a process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed in a sequence other than that shown or discussed, including in a substantially simultaneous manner or in a reverse order depending on the functions involved, as should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0116] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0117] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0118] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0119] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0120] The above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Any equivalent substitution or modification made by those skilled in the art based on the present invention is within the protection scope of the present invention.

Claims

1. Radar point cloud data enhancement method, characterized in that: The method comprises: Enumerate the labeled boxes in the point cloud data, filter out the labeled boxes whose parts are within the radar's close-range blind spot, and obtain a plurality of first target boxes and point cloud data within the first target boxes based on the filtered labeled boxes. All of the first target boxes and the point cloud data within the first target boxes constitute a field of view boundary scene dataset; selecting a second target frame from the field of view boundary scene dataset, and when it is determined that the second target frame is to be placed in a target frame of the point cloud data based on a position of the second target frame and the second target frame does not conflict with an object in the target frame, placing the second target frame and the point cloud data within the second target frame into the target frame to enhance the point cloud data; Determining whether the second target frame conflicts with an object in the target frame when the second target frame is placed into the target frame of the point cloud data based on the position of the second target frame includes: When an object is detected in the target frame based on the position and range of the second target frame, determining that the second target frame conflicts with the object in the target frame; When an intersection is detected between the second target frame and the annotation frame or the target frame in the target frame, determining that the second target frame conflicts with the object in the target frame; When it is detected that the second target frame is placed in the target frame and is blocked, determining that the second target frame conflicts with the object in the target frame; Otherwise, it is determined that the second target frame does not conflict with the object in the target frame.

2. The radar point cloud data enhancement method according to claim 1, wherein: The part of the area that is selected and marked as being within the radar's short-range blind spot range includes: From a bird's-eye view perspective, a first length between a rear axle center of an ego vehicle and a center of a radar coordinate system is calculated, a second length of a close-range blind spot of the radar in a direction of travel of the ego vehicle is obtained, and a third length of an object represented by the annotation box is obtained, wherein the ego vehicle is equipped with a radar; With the center of the rear axle of the own vehicle as the center of the coordinate system, a boundary range in the direction of travel of the own vehicle is obtained based on the first length, the second length, and the third length. When the coordinates of the center of the annotation box along the direction of travel of the own vehicle are within the boundary range, the annotation box is set as a annotation box whose portion is within the close-range blind spot of the radar.

3. The radar point cloud data enhancement method according to claim 1, wherein: The obtaining of a plurality of first target frames and point cloud data within the first target frames based on the filtered annotation frames includes: Obtaining the projection area of ​​the point cloud within the filtered annotation box on each projection plane; When the maximum value of the projected area is greater than a preset area threshold, the filtered annotation box is set as the first target box and the point cloud data in the first target box is acquired.

4. The radar point cloud data enhancement method according to claim 1, wherein: Detecting whether the second target frame is blocked when placed in the target frame includes: Projecting the second target frame and each annotation frame in the target frame onto respective projection planes, and calculating the intersection area between the projection of the second target frame and the projection of each annotation frame in the target frame on each projection plane; When any of the intersection areas is equal to the projected area of ​​the second target frame, it is determined that the second target frame is blocked when placed in the target frame; otherwise, it is determined that the second target frame is not blocked when placed in the target frame.

5. The radar point cloud data enhancement method according to claim 1, wherein: Detecting whether an object exists in the target frame based on the position and range of the second target frame includes: Based on the position of the second target frame, obtaining the number of point clouds within the range of the second target frame in the target frame; When the number of the point clouds exceeds a preset number threshold, it is determined that an object exists in the target frame; otherwise, it is determined that no object exists in the target frame.

6. The radar point cloud data enhancement method according to claim 1, wherein: Detect the second target frame Whether there is an intersection with the annotation box or target box in the target frame, including: From a bird's-eye view perspective, obtaining a first projection area of ​​the second target frame, a second projection area of ​​the annotation frame, and a third projection area of ​​the target frame placed in the target frame; respectively calculating a first intersection-and-union ratio of the first projected area to the second projected area and a second intersection-and-union ratio of the first projected area to the third projected area; When the first intersection-in-union ratio is not equal to zero or the second intersection-in-union ratio is not equal to zero, it is determined that there is an intersection between the second target box and the label box or the target box in the target frame; otherwise, it is determined that there is no intersection between the second target box and the label box or the target box in the target frame.

7. The radar point cloud data enhancement method according to claim 1, wherein: Putting the second target frame into the target frame includes: Searching for the target category's annotation box and target box in the target frame to obtain a total number; When the total number is less than or equal to a preset threshold, the second target frame is placed into the target frame.

8. Radar point cloud data enhancement device, characterized in that: The device comprises: A dataset module is configured to enumerate the annotation boxes in the point cloud data, filter out the annotation boxes whose parts are within the radar's close-range blind spot, and obtain a plurality of first target boxes and point cloud data within the first target boxes based on the filtered annotation boxes. All of the first target boxes and the point cloud data within the first target boxes constitute a field of view boundary scene dataset. an enhancement module, configured to select a second target frame from the field of view boundary scene dataset, and when it is determined that the second target frame is to be placed in a target frame of the point cloud data based on a position of the second target frame and the second target frame does not conflict with an object in the target frame, place the second target frame and the point cloud data within the second target frame into the target frame to enhance the point cloud data; Determining whether the second target frame conflicts with an object in the target frame when the second target frame is placed into the target frame of the point cloud data based on the position of the second target frame includes: When an object is detected in the target frame based on the position and range of the second target frame, determining that the second target frame conflicts with the object in the target frame; When an intersection is detected between the second target frame and the annotation frame or the target frame in the target frame, determining that the second target frame conflicts with the object in the target frame; When it is detected that the second target frame is placed in the target frame and is blocked, determining that the second target frame conflicts with the object in the target frame; Otherwise, it is determined that the second target frame does not conflict with the object in the target frame.

9. The radar point cloud data enhancement device according to claim 8, characterized in that: The enhancement module includes a conflict determination unit, which is used to determine that when an object is detected in the target frame based on the position and range of the second target frame, the second target frame conflicts with the object in the target frame; when an intersection is detected between the second target frame and the annotation frame or target frame in the target frame, the second target frame conflicts with the object in the target frame; when it is detected that the second target frame is occluded when placed in the target frame, the second target frame conflicts with the object in the target frame; otherwise, the second target frame does not conflict with the object in the target frame.

10. The radar point cloud data enhancement device according to claim 8, characterized in that: The data set module includes a screening unit, which is used to calculate a first length between the center of the rear axle of the vehicle and the center of the radar coordinate system from a bird's-eye view perspective, calculate a second length of the radar's close-range blind spot in the direction of travel of the vehicle, and obtain a third length of the object represented by the annotation box, wherein the vehicle is equipped with a radar; with the center of the rear axle of the vehicle as the center of the coordinate system, obtain a boundary range in the direction of travel of the vehicle based on the first length, the second length, and the third length; when the coordinates of the center of the annotation box along the direction of travel of the vehicle are within the boundary range, set the annotation box as a annotation box whose portion is within the close-range blind spot of the radar.

11. Smart car, characterized by, The smart car includes a memory, a processor, and a radar point cloud data enhancement program stored in the memory and executable on the processor. When the radar point cloud data enhancement program is executed by the processor, the steps of the radar point cloud data enhancement method according to any one of claims 1 to 7 are implemented.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a radar point cloud data enhancement program, which, when executed by a processor, implements the steps of the radar point cloud data enhancement method according to any one of claims 1 to 7.

Images