Point cloud data enhancement method, device, equipment and storage medium

Abstract

The present application relates to a point cloud data enhancement method, apparatus, device, and storage medium, and relates to the field of automotive technology. The method comprises: obtaining at least one paste box corresponding to any one type of point cloud data from a training database, and determining multiple derivative boxes corresponding to each paste box in the at least one paste box; determining the ground height standard deviation threshold and ground height difference threshold corresponding to any one type of point cloud data based on the size of each paste box; determining the target derivative box corresponding to each paste box from the multiple derivative boxes based on the ground height standard deviation threshold and ground height difference threshold; adding the point cloud data corresponding to each paste box to the target derivative box corresponding to each paste box to obtain enhanced point cloud data. In this way, the computational time of ground judgment can be reduced, the rationality of enhanced point cloud data can be ensured, the efficiency of enhanced point cloud data can be improved, and the problem of long ground judgment time and low accuracy can be solved.

Description

Technical Field

[0001] The present invention relates to the field of automotive technology, and in particular to a point cloud data enhancement method, device, equipment and storage medium. Background Art

[0002] With the development of autonomous driving technology, three-dimensional (3D) object perception technology based on LiDAR is becoming increasingly important in the field. Large amounts of point cloud data can be used to train and optimize detection models such as object detection and semantic segmentation in 3D object perception, accelerating the convergence of detection models and improving their accuracy and robustness. However, the current number of available point cloud data frames is insufficient, or the number of target objects in each frame is insufficient, resulting in an imbalance in categories and a lack of diversity, resulting in poor richness of point cloud data.

[0003] Currently, the diversity of point cloud data can be increased by performing enhancement operations (such as scaling, rotation, flipping and noise transformation) on the global point cloud data and the point cloud data within the local target. In addition, the data volume and richness of the point cloud data can be increased by placing the target object in the appropriate position by judging whether the position of the added target object (i.e., point cloud data) is a ground point cloud and whether it interferes with the object in the original point cloud data. However, when performing enhancement operations on the global point cloud data and the point cloud data within the local target, the enhanced point cloud will deviate from the real scene, resulting in unreasonable scenes and targets. It takes a long time to judge the ground point cloud and interfering objects, which is not conducive to real-time training of the model. In addition, the accuracy of judging the ground point cloud and interfering objects is low, resulting in poor rationality and efficiency in enhancing the point cloud data. Summary of the Invention

[0004] The purpose of the present invention is to provide a point cloud data enhancement method, apparatus, device, and storage medium to address the technical problem that when adding target point cloud data, determining ground point clouds and interfering objects takes a long time and has low accuracy, resulting in poor rationality and efficiency in enhancing point cloud data. The technical solutions of this application are as follows:

[0005] According to a first aspect of the present application, a point cloud data enhancement method is provided, comprising: for any type of point cloud data among multiple types of point cloud data included in a training database corresponding to a target vehicle, obtaining at least one paste box corresponding to the any type of point cloud data from the training database, and determining multiple derivative boxes corresponding to each paste box in the at least one paste box, wherein the size range of the paste boxes corresponding to each type of point cloud data is different, the paste box is used to indicate a virtual obstacle corresponding to the any type of point cloud data, and the derivative box is used to indicate a duplicate obstacle of the virtual obstacle corresponding to the paste box; determining a ground height standard deviation threshold and a ground height difference threshold corresponding to the any type of point cloud data based on the size of each paste box in the at least one paste box; determining a target derivative box corresponding to each paste box from the multiple derivative boxes corresponding to each paste box in the at least one paste box based on the ground height standard deviation threshold and the ground height difference threshold corresponding to the any type of point cloud data, wherein each paste box corresponds to one target derivative box; and adding the point cloud data corresponding to each paste box in the at least one paste box to the target derivative box corresponding to each paste box to obtain enhanced point cloud data.

[0006] According to the above technical means, the present application can determine the target derivative frame that can be placed on the ground position from the derivative frames of the paste frame corresponding to any type of point cloud data at multiple positions through the ground height standard deviation threshold and the ground height difference threshold corresponding to any type of point cloud data, so as to add the point cloud data corresponding to the paste frame to the target derivative frame to obtain enhanced point cloud data, thereby increasing the possibility of successfully placing the pasted target on the ground and ensuring the rationality of the enhanced point cloud data. In addition, the ground judgment through the ground height standard deviation threshold and the ground height difference threshold reduces the calculation time and improves the efficiency of the enhanced point cloud data.

[0007] In one possible implementation, determining multiple derivative frames corresponding to each paste box in at least one paste box includes: taking the target vehicle as the origin, rotating each paste box in the at least one paste box based on the original position of each paste box to obtain multiple derivative frames corresponding to each paste box in the at least one paste box; and determining the position of each derivative frame in the multiple derivative frames corresponding to each paste box in the at least one paste box based on the position of each paste box in the at least one paste box and the rotation angle corresponding to each derivative frame.

[0008] According to the above technical means, the present application can determine multiple rotational positions of the pasting frame by rotation, so as to perform ground judgment and collision judgment on the derived frames at multiple rotational positions, thereby improving the possibility of successful placement of the pasting target.

[0009] In a possible implementation, each paste box corresponds to multiple point clouds; based on the size of each paste box in at least one paste box, a ground height standard deviation threshold and a ground height difference threshold corresponding to any type of point cloud data are determined, including: based on the height value of each point cloud data in the multiple point cloud data corresponding to each paste box in at least one paste box, the maximum height value, minimum height value and height standard deviation corresponding to each paste box are determined; based on the height standard deviation corresponding to each paste box in at least one paste box, the ground height standard deviation threshold corresponding to any type of point cloud data is determined; based on the maximum height value and minimum height value corresponding to each paste box in at least one paste box, the ground height difference threshold corresponding to any type of point cloud data is determined.

[0010] Based on the above technical means, this application can count the ground height standard deviation threshold and ground height difference threshold corresponding to the ground point cloud of various types of point cloud data in the vertical block, which can be used for subsequent judgment of whether the pasted target can be placed on the ground at the corresponding position, thereby improving the rationality of target pasting.

[0011] In a possible implementation, based on a ground height standard deviation threshold and a ground height difference threshold corresponding to any type of point cloud data, a target derivative frame corresponding to each paste frame is determined from multiple derivative frames corresponding to each paste frame in at least one paste frame, including: determining a point cloud height standard deviation and a point cloud height difference corresponding to each derivative frame in multiple derivative frames corresponding to each paste frame in at least one paste frame; based on a ground height standard deviation threshold and a ground height difference threshold corresponding to any type of point cloud data, determining at least one derivative frame corresponding to each paste frame from multiple derivative frames corresponding to each paste frame in at least one paste frame, the point cloud height standard deviation corresponding to each derivative frame in at least one derivative frame is less than the ground height standard deviation threshold, and the point cloud height difference corresponding to each derivative frame is less than the ground height difference threshold; determining an intersection-over-union (IOU) corresponding to each derivative frame in at least one derivative frame corresponding to each paste frame in at least one paste frame, and determining a derivative frame with an IOU of zero in at least one derivative frame corresponding to each paste frame as a target derivative frame, where the IOU is the overlap rate between the derivative frame and the original point cloud data.

[0012] According to the above technical means, the present application can determine the ground position where the target can be pasted from the position of the derived frame based on the point cloud height standard deviation and point cloud height difference corresponding to the position of the derived frame, and based on the ground height standard deviation threshold and ground height difference threshold corresponding to the point cloud data at the position of the derived frame, and further determine the ground position that does not overlap with the original point cloud data from the ground position of the derived frame through IOU, so as to ensure that the ground position for target pasting does not collide with other obstacles, thereby improving the rationality of target pasting.

[0013] In a possible implementation, based on the category of point cloud data, point cloud data corresponding to each paste box in at least one paste box corresponding to each category of point cloud data in multiple categories of point cloud data are added to a target derivative box corresponding to each paste box in turn to obtain enhanced point cloud data; for each paste box in at least one paste box corresponding to each category of point cloud data in the multiple categories of point cloud data, and a target derivative box corresponding to each paste box, a center point of each paste box or each target derivative box is connected to multiple corner points respectively to divide each paste box or each target derivative box into multiple vertebrae; transformation processing is performed on the multiple vertebrae corresponding to each paste box or each target derivative box to obtain further enhanced point cloud data, and the transformation processing includes at least one of the following: encryption supplementation, sparsification, noise addition, exchange, mixing, and discarding.

[0014] Based on the above technical means, this application can perform corresponding point cloud data enhancement for different categories of point cloud data, balance the category distribution of point cloud data in each frame of the point cloud, improve the detection rate of small point cloud data samples, and enhance the convergence speed and detection performance of the detection model. By transforming the multiple vertebrae corresponding to the pasted frame or the target derived frame, the diversity of the point cloud can be enhanced, thereby improving the detection accuracy and robustness of the point cloud detection model.

[0015] According to a second aspect of the present application, a point cloud data enhancement device is provided, comprising an acquisition module, a determination module, and a processing module. The acquisition module is configured to acquire, from a training database corresponding to a target vehicle, at least one paste box corresponding to any one of multiple types of point cloud data included in the training database, wherein the size range of the paste box corresponding to each type of point cloud data is different, and the paste box is used to indicate a virtual obstacle corresponding to the point cloud data of the target vehicle. The determination module is configured to determine multiple derivative boxes corresponding to each of the at least one paste box, wherein the derivative boxes are used to indicate duplicate obstacles of the virtual obstacle corresponding to the paste box. The determination module is further configured to determine, based on the size of each paste box in the at least one paste box, a ground height standard deviation threshold and a ground height difference threshold corresponding to the point cloud data of the target vehicle. The determination module is further configured to determine, based on the ground height standard deviation threshold and the ground height difference threshold corresponding to the point cloud data of the target vehicle, a target derivative box corresponding to each paste box from the multiple derivative boxes corresponding to each paste box in the at least one paste box, wherein each paste box corresponds to one target derivative box. The processing module is configured to add the point cloud data corresponding to each paste box in the at least one paste box to the target derivative box corresponding to each paste box, thereby obtaining enhanced point cloud data.

[0016] In one possible embodiment, the processing module is further used to rotate each paste box in at least one paste box based on the original position of each paste box in the at least one paste box with the target vehicle as the origin to obtain multiple derivative boxes corresponding to each paste box in the at least one paste box; the determination module is further used to determine the position of each derivative box in the multiple derivative boxes corresponding to each paste box in the at least one paste box based on the position of each paste box in the at least one paste box and the rotation angle corresponding to each derivative box.

[0017] In one possible implementation, each paste box corresponds to multiple point clouds; the determination module is further used to determine the maximum height value, minimum height value and height standard deviation corresponding to each paste box based on the height value of each point cloud data corresponding to each paste box in at least one paste box; the determination module is further used to determine the ground height standard deviation threshold corresponding to any type of point cloud data based on the height standard deviation corresponding to each paste box in at least one paste box; the determination module is further used to determine the ground height difference threshold corresponding to any type of point cloud data based on the maximum height value and minimum height value corresponding to each paste box in at least one paste box.

[0018] In one possible embodiment, the determination module is further used to determine the point cloud height standard deviation and point cloud height difference corresponding to each derivative box in the multiple derivative boxes corresponding to each paste box in the at least one paste box; the determination module is further used to determine at least one derivative box corresponding to each paste box in the at least one paste box from the multiple derivative boxes corresponding to each paste box in the at least one paste box based on the ground height standard deviation threshold and the ground height difference threshold corresponding to any type of point cloud data, the point cloud height standard deviation corresponding to each derivative box in the at least one derivative box is less than the ground height standard deviation threshold, and the point cloud height difference corresponding to each derivative box is less than the ground height difference threshold; the determination module is further used to determine the intersection-and-union (IOU) corresponding to each derivative box in the at least one derivative box corresponding to each paste box in the at least one paste box, and determine the derivative box with an IOU of zero in the at least one derivative box corresponding to each paste box as the target derivative box, where the IOU is the overlap rate between the derivative box and the original point cloud data.

[0019] In one possible embodiment, the processing module is further used to, based on the category of the point cloud data, add the point cloud data corresponding to each paste box in at least one paste box corresponding to each category of point cloud data in the multiple categories of point cloud data to the target derivative box corresponding to each paste box, so as to obtain enhanced point cloud data; the processing module is further used to, for each paste box in at least one paste box corresponding to each category of point cloud data in the multiple categories of point cloud data, and the target derivative box corresponding to each paste box, connect the center point of each paste box or each target derivative box with multiple corner points, so as to divide each paste box or each target derivative box into multiple vertebrae; the processing module is further used to perform transformation processing on the multiple vertebrae corresponding to each paste box or each target derivative box, so as to obtain further enhanced point cloud data, and the transformation processing includes at least one of the following: encryption supplementation, sparsification, noise addition, exchange, mixing, and discarding.

[0020] According to the third aspect provided by the present application, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement the method of the above-mentioned first aspect and any possible implementation method thereof.

[0021] According to the fourth aspect provided by the present application, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by the processor of an electronic device, the electronic device is enabled to execute the method in the above-mentioned first aspect and any possible implementation method thereof.

[0022] According to the fifth aspect provided by the present application, a vehicle is provided, comprising: a point cloud data enhancement device for implementing the method of the above-mentioned first aspect and any possible implementation method thereof.

[0023] According to the sixth aspect provided by the present application, a computer program product is provided, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method of the above-mentioned first aspect and any possible implementation method thereof.

[0024] Therefore, the above technical features of this application have the following beneficial effects:

[0025] (1) The target derivative frame that can be placed on the ground can be determined from the derivative frames of the pasting frame corresponding to any type of point cloud data at multiple positions through the ground height standard deviation threshold and the ground height difference threshold corresponding to any type of point cloud data, so as to add the point cloud data corresponding to the pasting frame to the target derivative frame to obtain enhanced point cloud data, thereby increasing the possibility of successfully placing the pasted target on the ground and ensuring the rationality of the enhanced point cloud data. In addition, the ground judgment through the ground height standard deviation threshold and the ground height difference threshold reduces the calculation time and improves the efficiency of the enhanced point cloud data.

[0026] (2) Multiple rotational positions of the pasting frame can be determined by rotation, so as to perform ground judgment and collision judgment on the derived frames at multiple rotational positions, thereby increasing the possibility of successful placement of the pasted target.

[0027] (3) The ground height standard deviation threshold and ground height difference threshold corresponding to the ground point cloud of various point cloud data in the vertical block can be counted, which can be used to subsequently determine whether the pasted target can be placed on the ground at the corresponding position, thereby improving the rationality of target pasting.

[0028] (4) According to the point cloud height standard deviation and point cloud height difference corresponding to the position of the derived frame, and based on the ground height standard deviation threshold and ground height difference threshold corresponding to the point cloud data at the position of the derived frame, the ground position where the target can be pasted can be determined from the position of the derived frame, and further, the ground position that does not overlap with the original point cloud data can be determined from the ground position of the derived frame through IOU to ensure that the ground position where the target is pasted does not collide with other obstacles, thereby improving the rationality of the target pasting.

[0029] (5) Point cloud data can be enhanced for different categories of point cloud data, balancing the category distribution of point cloud data in each frame, improving the detection rate of a small number of point cloud data samples, and improving the convergence speed and detection performance of the detection model. By transforming multiple vertebrae corresponding to the pasted box or the target derived box, the diversity of the point cloud can be enhanced, thereby improving the detection accuracy and robustness of the point cloud detection model.

[0030] It should be noted that the technical effects brought about by any implementation method in the second to sixth aspects can refer to the technical effects brought about by the corresponding implementation method in the first aspect, and will not be repeated here.

[0031] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application, and do not constitute an improper limitation on the present application.

[0033] Figure 1 is a schematic structural diagram of a point cloud data enhancement system according to an exemplary embodiment;

[0034] Figure 2 is a flow chart of a point cloud data enhancement method according to an exemplary embodiment;

[0035] Figure 3is a flowchart of another point cloud data enhancement method according to an exemplary embodiment;

[0036] Figure 4 is a schematic diagram showing a rotational amplification of a pasting frame according to an exemplary embodiment;

[0037] Figure 5 is a flowchart of another point cloud data enhancement method according to an exemplary embodiment;

[0038] Figure 6 is a flowchart of another point cloud data enhancement method according to an exemplary embodiment;

[0039] Figure 7 is a flowchart of another point cloud data enhancement method according to an exemplary embodiment;

[0040] Figure 8 is a schematic diagram showing the exchange and mixing of vertebral point clouds of a pasting frame according to an exemplary embodiment;

[0041] Figure 9 is a flowchart of point cloud data enhancement according to an exemplary embodiment;

[0042] Figure 10 is a block diagram of a point cloud data enhancement device according to an exemplary embodiment;

[0043] Figure 11 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0044] The following describes the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art will readily appreciate the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through various other specific embodiments, and the various details in this specification may be modified or altered based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are intended only to illustrate the present invention and are not intended to limit the scope of protection of the present invention.

[0045] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0046] With the development of autonomous driving technology, LiDAR-based 3D object perception technology has gradually attracted attention and exploration. Large amounts of point cloud data can be used to train and optimize detection models such as object detection and semantic segmentation in the 3D object perception field. However, the number of annotated point cloud data frames currently available is insufficient, or the number of target objects in each frame of point cloud data is insufficient, resulting in an imbalance in categories and a lack of diversity, resulting in poor richness of point cloud data. Point cloud data augmentation during model training can accelerate model convergence and improve model accuracy and robustness.

[0047] Currently, the amount of real point cloud data can be increased and its diversity improved by removing the initial obstacles and placing new obstacles within the preset area. However, when the original scene is diverse, it is impossible to determine whether there are entities (such as trees, flower beds, fences, etc.) within the preset area. This may cause the placed obstacles to conflict with the original obstacles, resulting in unreasonable enhanced point cloud data.

[0048] Currently, point cloud enhancement can also be performed based on true value replication. Specifically, the target frame (i.e., obstacle) in the dataset scene can be first cut out, and a specific number and type of target frames can be randomly extracted. Then, it is determined whether the position of the target frame in the target point cloud frame is on the ground and whether it interferes with the obstacles in the original point cloud. If the position of the target frame is not on the ground or there is obstacle interference, the target frame is discarded and the next target frame is determined. Finally, when placing the target frame, the point cloud at the position of this target frame in the original point cloud is deleted to increase the data volume of the point cloud data.

[0049] The original point cloud can be divided into sectors and the average height of each unit point can be calculated to obtain the ground points and non-ground points corresponding to the original point cloud data. The average height can be calculated again by dividing the grid to obtain the ground height of each grid area corresponding to the ground point. The ground height of each grid area corresponding to the ground point, non-ground point and ground point corresponding to the original point cloud data can be used to determine whether the position of the target box is on the ground, or the ground plane can be fitted using a random sampling consensus (RANSAC) algorithm.

[0050] While the above method can accurately position obstacles, addressing the increased false detection rate caused by inappropriate data augmentation and avoiding interference with entities in the original point cloud, by adding various types of objects (i.e., obstacles) to balance the distribution of obstacle categories in each frame, it can improve model convergence speed and post-training detection accuracy. However, the method of fanning the original point cloud, calculating the average height of each unit point, dividing the grid, recalculating the average height, and fitting the ground plane using RANSAC is time-consuming and inaccurate, with high computational costs, making it unsuitable for real-time processing during online training, resulting in poor efficiency in enhancing point cloud data. Furthermore, the method of filtering ground points by the average height of ground point units lacks rationality when nearby vehicles are densely populated. Due to the relatively sparse point cloud data, the ground surface may fluctuate both near and far, leading to inaccurate ground plane determination or fitting. Furthermore, it cannot filter flat, unlabeled non-ground scenes (such as flower beds) and may place obstacles within unlabeled entities (such as trees, flower beds, and walls), resulting in illogical scenes in the enhanced point cloud and prone to false detections.

[0051] Furthermore, point cloud diversity can be enhanced by scaling, rotating, flipping, and performing noise transformations on the global point cloud data and the local point cloud data. However, due to the randomness of scaling, rotating, flipping, and noise transformations on the local point cloud data, the enhanced point cloud may deviate from the actual scene, resulting in irrational scenes and point cloud targets, which can lead to false detections in the trained model.

[0052] For ease of understanding, the point cloud data enhancement method provided in this application is specifically introduced below with reference to the accompanying drawings.

[0053] The point cloud data enhancement method provided in the embodiment of the present application can be applied to a point cloud data enhancement system. Figure 1 FIG. 1 is a schematic diagram showing the structure of a point cloud data enhancement system according to an exemplary embodiment. Figure 1 As shown, the point cloud data enhancement system 10 includes: a point cloud pasting box acquisition module 11, a ground threshold determination module 12, a point cloud pasting box placement position determination module 13 and a point cloud data enhancement module 14.

[0054] Among them, the point cloud paste box acquisition module 11 is used to obtain at least one paste box corresponding to any type of point cloud data from the training database, and determine multiple derivative boxes corresponding to each paste box in the at least one paste box; the ground threshold determination module 12 is used to determine the ground height standard deviation threshold and the ground height difference threshold corresponding to any type of point cloud data; the point cloud paste box placement position determination module 13 is used to determine the target derivative box corresponding to each paste box from the multiple derivative boxes corresponding to each paste box in the at least one paste box; the point cloud data enhancement module 14 is used to add the point cloud data corresponding to each paste box in the at least one paste box to the target derivative box corresponding to each paste box to obtain enhanced point cloud data.

[0055] Figure 2 is a flow chart of a point cloud data enhancement method according to an exemplary embodiment. Figure 2 As shown, the point cloud data enhancement method includes the following steps:

[0056] S201 . For any type of point cloud data among multiple types of point cloud data included in a training database corresponding to a target vehicle, obtain at least one pasting box corresponding to any type of point cloud data from the training database.

[0057] S202: Determine a plurality of derived frames corresponding to each of the at least one pasting frame.

[0058] The size range of the paste box corresponding to each type of point cloud data in the multiple types of point cloud data is different. The paste box is used to indicate the virtual obstacle corresponding to any type of point cloud data, and the derivative box is used to indicate the copy obstacle of the virtual obstacle corresponding to a paste box.

[0059] Optionally, all target frames (i.e., virtual obstacles corresponding to the point cloud data) in each frame of point cloud data in the training data set corresponding to the target vehicle can be extracted, and all target frames and the annotation information corresponding to each target frame in all target frames can be stored in the training database. When training and processing each frame of point cloud data, each type of point cloud data in the multiple types of point cloud data included in any frame of point cloud data in the training database corresponding to the target vehicle can be traversed, and a preset number of paste frames (i.e., at least one paste frame) corresponding to any type of point cloud data can be randomly extracted from the training database.

[0060] Exemplarily, the preset number may be three, and the preset number of paste boxes may be a PBoxes array, and the PBoxes array may include a paste box B1, a paste box B2, and a paste box B3.

[0061] It should be noted that the annotation information corresponding to the target frame is used to indicate the position of the target frame in the point cloud coordinate system with the target vehicle as the origin.

[0062] S203 : Determine a ground height standard deviation threshold and a ground height difference threshold corresponding to any type of point cloud data based on the size of each paste box in the at least one paste box.

[0063] Optionally, you can first set the estimated size of each paste box required for the training model and select a batch of multi-class point cloud data with multiple scenes. Based on the estimated size of each paste box, the point cloud data corresponding to the ground blocks and non-ground blocks at close range, medium range, and long range are respectively extracted from each frame of point cloud data through each paste box (the Z-axis height of the ground blocks and non-ground blocks is not limited), and based on the point cloud data corresponding to the ground blocks and non-ground blocks extracted by each paste box, the ground height standard deviation threshold and ground height difference threshold corresponding to any type of point cloud data are determined.

[0064] For example, the non-ground area may be a bush, a flower bed, a junction between a road and a curb, or a junction between a road and a vehicle.

[0065] It should be noted that both ground blocks and non-ground blocks are virtual obstacles.

[0066] S204 , based on a ground height standard deviation threshold and a ground height difference threshold corresponding to any type of point cloud data, determining a target derived frame corresponding to each paste frame from a plurality of derived frames corresponding to each paste frame in at least one paste frame.

[0067] Among them, one paste box corresponds to one target derivative box.

[0068] Optionally, the target derived box can be added to the original target box TBoxes array of the target point cloud frame and added to the FBoxes array at the same time.

[0069] Exemplarily, the target derived box may include a derived box B13, a derived box B23, and a derived box B32. The FBoxes array after adding the derived box B13, the derived box B23, and the derived box B32 may include a paste box B1, a derived box B13, a paste box B2, a derived box B23, a paste box B3, and a derived box B32.

[0070] It should be noted that the original target box TBoxes array is used to indicate all virtual obstacles included in the target point cloud frame.

[0071] S205 , adding the point cloud data corresponding to each paste box in the at least one paste box to the target derivative box corresponding to each paste box to obtain enhanced point cloud data.

[0072] Optionally, the point cloud data corresponding to each paste box in at least one paste box can be added to the target derivative box corresponding to each paste box, and then the target derivative box is added to the original target box TBoxes array, the original target box TBoxes array is updated to obtain enhanced point cloud data, and the updated original target box TBoxes array is used as the original target box TBoxes array corresponding to the next category of point cloud data in the multi-category point cloud data, and each category of point cloud data in the multi-category point cloud data is looped through in sequence until the point cloud data of the paste box corresponding to each category of point cloud data is added to the original target box TBoxes array.

[0073] It should be noted that the updated original target box TBoxes array includes the point cloud data in the target derived box. When adding the point cloud data corresponding to the paste box to the target derived box corresponding to each paste box, the original point cloud data within the vertical region of the target derived box must first be deleted, and then the point cloud data corresponding to the paste box must be added to obtain the updated original target box TBoxes array. The annotation information of the paste box in the FBoxes array can be added to the ground truth of the original point cloud data corresponding to the target derived box.

[0074] Figure 3 is a flowchart of another point cloud data enhancement method according to an exemplary embodiment. Figure 3 As shown, the method in step S202 above specifically includes the following steps:

[0075] S301 : Taking the target vehicle as the origin, rotating each paste box in at least one paste box based on the original position to obtain a plurality of derivative boxes corresponding to each paste box in the at least one paste box.

[0076] Optionally, the target vehicle can be used as the origin (i.e., the origin of the point cloud coordinate system), and based on the original position of each paste box in at least one paste box (i.e., the original annotation position P), a plurality of limited angles can be randomly selected for a first rotation to obtain a first derivative frame corresponding to each paste box in at least one paste box. Furthermore, based on the rotated position of each paste box in at least one paste box, a plurality of limited angles can be randomly selected for a second rotation to obtain a second derivative frame corresponding to each paste box in at least one paste box. Similarly, multiple derivative frames corresponding to each paste box in at least one paste box can be obtained.

[0077] Exemplarily, the multiple defined angles may be three defined angles, and the defined angles may be reasonable angles such as ±5°, ±10°, or ±15°. The multiple derived frames corresponding to the paste box B1 in the at least one paste box may include a derived frame B11, a derived frame B12, and a derived frame B13. The multiple derived frames corresponding to the paste box B2 in the at least one paste box may include a derived frame B21, a derived frame B22, and a derived frame B23. The multiple derived frames corresponding to the paste box B3 in the at least one paste box may include a derived frame B31, a derived frame B32, and a derived frame B33.

[0078] It should be noted that after the rotation, the annotation angle of each paste box in at least one paste box needs to be changed to the angle after the rotation, and the annotation position needs to be changed to the position after the rotation.

[0079] S302 : Determine the position of each derivative frame in the plurality of derivative frames corresponding to each paste frame in the at least one paste frame based on the position of each paste frame in the at least one paste frame and the rotation angle corresponding to each derivative frame.

[0080] Optionally, based on the original position of any one of the at least one pasting frames, the frame can be rotated at a first preset angle to obtain a first derivative frame corresponding to the frame. Further, based on the original position of the frame and the first preset angle (i.e., the rotation angle corresponding to the first derivative frame), the position of the first derivative frame corresponding to the frame can be determined. Based on the position of the first derivative frame, the frame can be rotated at a second preset angle to obtain a second derivative frame corresponding to the frame. Further, based on the position of the first derivative frame and the second preset angle, the position of the second derivative frame corresponding to the frame can be determined. Similarly, the position of each derivative frame in the multiple derivative frames corresponding to each of the at least one pasting frame can be determined.

[0081] For example, Figure 4 is a schematic diagram showing a paste frame rotation amplification according to an exemplary embodiment. Figure 4 As shown, assuming that the vehicle is the origin of the coordinate system, based on the original annotation position (X, Y) and original annotation angle Yaw of the pasting frame B1, a clockwise rotation is performed at a first rotation angle α to obtain a first derived frame B11 corresponding to the pasting frame B1, and based on the position of the first derived frame B11, a rotation is performed at a second preset angle θ to obtain a second derived frame B12 corresponding to the pasting frame B1. The position (X1, Y1) of the first derived frame B11 can be determined based on the following formula 1:

[0082]

[0083] The angle Yaw1 of the first derivative frame B11 can be determined based on the following formula 2:

[0084] Yaw1=Yaw+αFormula 2

[0085] It should be noted that the z coordinate value remains unchanged when calculating the position and angle of the derived box. Rotating and calculating the PBoxes array of the same category (that is, each paste box in at least one paste box) can speed up the calculation.

[0086] Figure 5 is a flow chart of another point cloud data enhancement method according to an exemplary embodiment, wherein each pasting box corresponds to multiple point clouds, such as Figure 5 As shown, the method in step S203 above specifically includes the following steps:

[0087] S401 : Determine a maximum height value, a minimum height value, and a height standard deviation corresponding to each paste box based on a height value of each point cloud data in a plurality of point cloud data corresponding to each paste box in at least one paste box.

[0088] Optionally, the maximum height value of each point cloud data among the multiple point cloud data corresponding to each paste box in at least one paste box can be determined as the maximum height value corresponding to each paste box (i.e., Hmax), the minimum height value can be determined as the minimum height value corresponding to each paste box (i.e., Hmin), and the height standard deviation (i.e., Hstd) corresponding to each paste box can be determined based on the average height value (i.e., Hmean) of each point cloud.

[0089] It should be noted that since the ground area is relatively flat, the height of the point cloud data in the ground block often does not fluctuate much, so the Hstd corresponding to the paste box of the ground block is smaller, while the height of the point cloud data in the non-ground block often has sudden fluctuations, so the Hstd corresponding to the paste box of the non-ground block is larger.

[0090] S402: Determine a ground height standard deviation threshold corresponding to any type of point cloud data based on a height standard deviation corresponding to each paste box in at least one paste box.

[0091] Optionally, based on the height standard deviation corresponding to each paste box in the at least one paste box, a ground height standard deviation threshold (ie, THstd) corresponding to any type of point cloud data may be determined.

[0092] It should be noted that THstd can be used to determine whether any type of point cloud data is point cloud data of the ground area.

[0093] S403 : Determine a ground height difference threshold corresponding to any type of point cloud data based on a maximum height value and a minimum height value corresponding to each paste box in at least one paste box.

[0094] Optionally, based on the maximum height value and the minimum height value corresponding to each paste box in the at least one paste box, a ground height difference threshold (ie, ΔHmax) corresponding to any type of point cloud data may be determined.

[0095] It should be noted that ΔHmax can be used to determine whether any type of point cloud data is point cloud data of the ground area.

[0096] Figure 6 is a flowchart of another point cloud data enhancement method according to an exemplary embodiment. Figure 6 As shown, the method in step S204 above specifically includes the following steps:

[0097] S501: Determine a point cloud height standard deviation and a point cloud height difference corresponding to each of a plurality of derivative frames corresponding to each of at least one pasting frame.

[0098] Optionally, the point cloud data within the vertical area where each derivative box of the multiple derivative boxes corresponding to each paste box in at least one paste box is located can be cut out, and based on the height value of the point cloud data within the vertical area where each derivative box is located, the point cloud height standard deviation (i.e., PBoxes_Hstd), point cloud height difference (i.e., PBoxes_ΔHmax) and average point cloud height (i.e., PBoxes_Hmean) corresponding to each derivative box can be calculated.

[0099] Optionally, when extracting point cloud data within a vertical area where each derivative box in a plurality of derivative boxes corresponding to each paste box in at least one paste box is located, the length or width of the derivative box can be expanded to expand the area to obtain more point cloud data.

[0100] S502 : Based on a ground height standard deviation threshold and a ground height difference threshold corresponding to any type of point cloud data, determine at least one derivative frame corresponding to each paste frame from a plurality of derivative frames corresponding to each paste frame in at least one paste frame.

[0101] The point cloud height standard deviation corresponding to each derivative box in at least one derivative box is smaller than the ground height standard deviation threshold, and the point cloud height difference corresponding to each derivative box is smaller than the ground height difference threshold.

[0102] Optionally, based on the ground height standard deviation threshold and the ground height difference threshold corresponding to any type of point cloud data, at least one derivative box in which PBoxes_Hstd is less than THstd and PBoxes_ΔHmax is less than ΔHmax can be determined from multiple derivative boxes corresponding to each paste box in at least one paste box, at least one derivative box is retained, and the center position (i.e., the height position) of each derivative box in at least one derivative box is adjusted to the height corresponding to PBoxes_Hmean.

[0103] Exemplarily, at least one derivative box determined from the derivative box B11, derivative box B12 and derivative box B13 corresponding to the paste box B1 may include derivative box B12 and derivative box B13; at least one derivative box determined from the derivative box B21, derivative box B22 and derivative box B23 corresponding to the paste box B2 may include derivative box B21 and derivative box B23; at least one derivative box determined from the derivative box B31, derivative box B32 and derivative box B33 corresponding to the paste box B3 may include derivative box B31 and derivative box B32.

[0104] It should be noted that by adjusting the center position of the derivative box to the height corresponding to PBoxes_Hmean, the derivative box can be placed on the ground corresponding to the vertical area.

[0105] S503: Determine the intersection over union (IOU) of each derivative frame in at least one derivative frame corresponding to each paste frame in at least one paste frame, and determine the derivative frame with zero IOU in at least one derivative frame corresponding to each paste frame as the target derivative frame.

[0106] Among them, IOU is the overlap ratio between the derived box and the original point cloud data.

[0107] Optionally, a 3D intersection over union (IOU) may be performed on each derivative box in at least one derivative box corresponding to each paste box in at least one paste box and each original target box in the original target box TBoxes array to obtain the IOU corresponding to each derivative box, retain the derivative boxes with an IOU of zero, and randomly select a derivative box corresponding to each paste box in at least one paste box from the derivative boxes with an IOU of zero, and determine the derivative box corresponding to each paste box as the target derivative box.

[0108] For example, the derived box with zero IOU in the derived boxes B12 and B13 corresponding to the pasted box B1 may be derived box B13; the derived boxes with zero IOU in the derived boxes B21 and B23 corresponding to the pasted box B2 may be derived boxes B21 and B23; the derived boxes with zero IOU in the derived boxes B31 and B32 corresponding to the pasted box B3 may be derived boxes B31 and B32. The target derived box randomly selected from the derived box B13 with zero IOU corresponding to the pasted box B1 is derived box B13; the target derived box randomly selected from the derived boxes B21 and B23 with zero IOU corresponding to the pasted box B2 may be derived box B23; the target derived box randomly selected from the derived boxes B31 and B32 with zero IOU corresponding to the pasted box B3 may be derived box B32.

[0109] Figure 7is a flowchart of another point cloud data enhancement method according to an exemplary embodiment. Figure 7 As shown, the method further includes the following steps:

[0110] S601. Based on the category of the point cloud data, the point cloud data corresponding to each paste box in at least one paste box corresponding to each category of point cloud data in the multiple categories of point cloud data is added to the target derivative box corresponding to each paste box to obtain enhanced point cloud data.

[0111] S602. For each paste box in at least one paste box corresponding to each type of point cloud data in the multiple types of point cloud data, and the target derived box corresponding to each paste box, connect the center point of each paste box or each target derived box with multiple corner points respectively, so as to divide each paste box or each target derived box into multiple vertebrae.

[0112] Exemplarily, the plurality of corner points may be 8 corner points, and the plurality of cones may be 6 cones.

[0113] S603: Perform transformation processing on multiple vertebrae corresponding to each pasted frame or each target derived frame to obtain further enhanced point cloud data.

[0114] The transformation processing includes at least one of the following: encryption and supplementation, sparsification, noise addition, exchange, mixing, and discarding.

[0115] Optionally, based on a preset probability, the point cloud data in multiple vertebrae corresponding to each pasting box or each target derivative box can be encrypted, supplemented, thinned, noisy, exchanged, mixed and discarded. Furthermore, all point cloud data included in the target point cloud frame can be globally processed (such as random rotation, translation, scaling and noise addition) within a preset range.

[0116] Figure 8 is a schematic diagram showing the exchange and mixing of vertebral point clouds of a pasting frame according to an exemplary embodiment. Figure 8 As shown, the paste box can be divided into 6 cones, and based on the preset exchange probability, the point cloud data in any cone in the paste box a is exchanged with the point cloud data in the corresponding cone in the paste box b of the same category, and based on the preset mixing probability, the point cloud data in any cone in the paste box a is mixed with the point cloud data in the corresponding cone in the paste box b of the same category.

[0117] Optionally, reasonable point cloud transformation constraints can be formulated based on the relative distance and angle between each paste box or target derived box and the point cloud coordinate system. Specifically, point cloud data from nearby unobstructed paste boxes or target derived boxes can be swapped or blended instead of discarded; distant point cloud data can be appropriately encrypted; and point cloud data within the entire cone can be noised or thinned.

[0118] For example, the preset probability may be reasonable values ​​such as 0.2, 0.3, and 0.4.

[0119] One possible implementation involves properly encrypting distant point cloud data to improve distant obstacle detection. Formulating reasonable point cloud transformation constraints for different scenarios simulates these scenarios to a certain extent, helping the detection model perform better in different scenarios and improving its robustness.

[0120] Figure 9 is a flowchart of a point cloud data enhancement method according to an exemplary embodiment. Figure 9 As shown, the ground judgment threshold corresponding to each category of point cloud data can be counted, and a paste box database can be generated based on the point cloud data in each frame of point cloud. Paste box sampling is performed for the category of point cloud data in the paste box database to generate derivative boxes of the paste box at multiple rotation positions. Then, the derivative box is ground judged based on the ground judgment threshold corresponding to each category of point cloud data. The derivative box that has been judged on the ground is subjected to collision judgment with the original target box to determine the target derivative box. The target derivative box is added to the original target box included in the target point cloud frame, and the original point cloud data at the position of the target derivative box is removed. Then, each category of point cloud data in the paste box database is looped until the target derivative box corresponding to each category of point cloud data is added to the original target box. Finally, the updated original target box is enhanced by partition block transformation, and the updated original target box is enhanced by global transformation.

[0121] An embodiment of the present application provides a point cloud data enhancement method. First, all point cloud data can be extracted from a point cloud training set, and the point cloud data and the corresponding annotations of the point cloud data can be stored in a database. Then, the threshold value related to the ground height of the point cloud in the vertical area of ​​each category is statistically set. By setting multiple rotation positions of the target pasting box and judging whether the rotation position is the ground and whether it collides with the original target, the reasonable position of the target pasting is determined. After locking the target pasting position, the target pasting box is placed in the target box update, and the center of the updated target box is connected to the 8 corner points to form 6 cones. Reasonable transformation restriction rules are set based on the actual situation, and a certain probability of random block shape transformation operation (including encryption supplementation, sparseness, noise addition, exchange, mixing, and discarding) is performed on the point cloud data in each cone.

[0122] Statistics of the height standard deviation threshold and ground height difference threshold of each category of target frame in the vertical block of the ground point cloud in multivariate scenarios can be used to subsequently determine whether each target pasting frame can be placed on the ground and whether each target pasting frame collides with other vehicles. This improves the rationality of target pasting, reduces calculation time and ensures its accuracy, and improves training speed. When placing the target pasting frame, the possibility of successful placement of the target pasting frame is increased by providing multiple rotation positions. Based on the actual situation, the corresponding block transformation enhancement of the target point cloud can increase the diversity of the point cloud and ensure its rationality, improve the robustness of the model, and improve the effect of point cloud detection (reflected in the target detection rate and position accuracy). It can balance the data categories in each frame of the point cloud and improve the detection rate of a small number of samples.

[0123] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, the point cloud data enhancement device or electronic device includes hardware structures and / or software modules corresponding to the execution of each function. It should be easy for those skilled in the art to realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0124] In the embodiment of the present application, the functional modules of the point cloud data enhancement device or the electronic device can be divided according to the above method. For example, the point cloud data enhancement device or the electronic device can include functional modules corresponding to the functional divisions, or two or more functions can be integrated into one processing module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0125] Figure 10 FIG. 1 is a block diagram of a point cloud data enhancement device according to an exemplary embodiment. Figure 10 , the point cloud data enhancement device 70 includes: an acquisition module 701, a determination module 702 and a processing module 703;

[0126] An acquisition module 701 is configured to acquire, from the training database, at least one pasting box corresponding to any one type of point cloud data among multiple types of point cloud data included in the training database corresponding to the target vehicle, wherein the pasting box corresponding to each type of point cloud data has a different size range, and the pasting box is used to indicate a virtual obstacle corresponding to any one type of point cloud data;

[0127] A determination module 702 is configured to determine a plurality of derivative frames corresponding to each of at least one pasting frame, wherein the derivative frames are used to indicate duplicate obstacles of a virtual obstacle corresponding to a pasting frame;

[0128] The determination module 702 is further configured to determine a ground height standard deviation threshold and a ground height difference threshold corresponding to any type of point cloud data based on the size of each paste box in the at least one paste box;

[0129] The determination module 702 is further configured to determine, based on a ground height standard deviation threshold and a ground height difference threshold corresponding to any type of point cloud data, a target derived frame corresponding to each paste frame from a plurality of derived frames corresponding to each paste frame in the at least one paste frame, wherein each paste frame corresponds to one target derived frame;

[0130] The processing module 703 is configured to add the point cloud data corresponding to each of the at least one pasting box to the target derivative box corresponding to each pasting box to obtain enhanced point cloud data.

[0131] In one possible embodiment, the processing module 703 is further used to rotate each paste box in at least one paste box based on the original position of each paste box in the at least one paste box with the target vehicle as the origin to obtain multiple derivative boxes corresponding to each paste box in the at least one paste box; the determination module 702 is further used to determine the position of each derivative box in the multiple derivative boxes corresponding to each paste box in the at least one paste box based on the position of each paste box in the at least one paste box and the rotation angle corresponding to each derivative box.

[0132] In one possible implementation, each paste box corresponds to multiple point clouds; the determination module 702 is further used to determine the maximum height value, minimum height value and height standard deviation corresponding to each paste box based on the height value of each point cloud data in the multiple point cloud data corresponding to each paste box in at least one paste box; the determination module 702 is further used to determine the ground height standard deviation threshold corresponding to any type of point cloud data based on the height standard deviation corresponding to each paste box in at least one paste box; the determination module 702 is further used to determine the ground height difference threshold corresponding to any type of point cloud data based on the maximum height value and minimum height value corresponding to each paste box in at least one paste box.

[0133] In one possible embodiment, the determination module 702 is further used to determine the point cloud height standard deviation and point cloud height difference corresponding to each derivative box in the multiple derivative boxes corresponding to each paste box in the at least one paste box; the determination module 702 is further used to determine, based on the ground height standard deviation threshold and the ground height difference threshold corresponding to any type of point cloud data, at least one derivative box corresponding to each paste box in the at least one paste box, from the multiple derivative boxes corresponding to each paste box in the at least one derivative box, the point cloud height standard deviation corresponding to each derivative box in the at least one derivative box is less than the ground height standard deviation threshold, and the point cloud height difference corresponding to each derivative box is less than the ground height difference threshold; the determination module 702 is further used to determine the intersection-over-union (IOU) corresponding to each derivative box in the at least one derivative box corresponding to each paste box in the at least one paste box, and determine the derivative box with an IOU of zero in the at least one derivative box corresponding to each paste box as the target derivative box, where the IOU is the overlap rate between the derivative box and the original point cloud data.

[0134] In one possible embodiment, the processing module 703 is further used to, based on the category of the point cloud data, add the point cloud data corresponding to each paste box in at least one paste box corresponding to each category of point cloud data in the multiple categories of point cloud data to the target derivative box corresponding to each paste box, to obtain enhanced point cloud data; the processing module 703 is further used to, for each paste box in at least one paste box corresponding to each category of point cloud data in the multiple categories of point cloud data, and the target derivative box corresponding to each paste box, connect the center point of each paste box or each target derivative box with multiple corner points, respectively, so as to divide each paste box or each target derivative box into multiple vertebrae; the processing module 703 is further used to perform transformation processing on the multiple vertebrae corresponding to each paste box or each target derivative box, to obtain further enhanced point cloud data, and the transformation processing includes at least one of the following: encryption supplementation, sparsification, noise addition, exchange, mixing, and discarding.

[0135] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0136] Figure 11 FIG. 1 is a block diagram of an electronic device according to an exemplary embodiment. Figure 11 As shown, the electronic device 80 includes but is not limited to: a processor 801 and a memory 802 .

[0137] The memory 802 is configured to store executable instructions of the processor 801. It is understood that the processor 801 is configured to execute instructions to implement the point cloud data enhancement method in the above embodiment.

[0138] It should be noted that those skilled in the art can understand that Figure 11 The electronic device structure shown in the figure does not limit the electronic device, and the electronic device may include Figure 11 More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.

[0139] The processor 801 is the control center of the electronic device. It uses various interfaces and lines to connect the various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 802 and calling data stored in the memory 802, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. The processor 801 may include one or more processing modules. Optionally, the processor 801 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly handles wireless communications. It is understood that the above-mentioned modem processor may not be integrated into the processor 801.

[0140] The memory 802 can be used to store software programs and various data. The memory 802 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and application programs required by at least one functional module (such as a determination unit, a processing unit, an acquisition unit, etc.). Furthermore, the memory 802 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0141] In an exemplary embodiment, a computer-readable storage medium including instructions is further provided, such as a memory 802 including instructions. The above instructions can be executed by the processor 801 of the electronic device 800 to implement the point cloud data enhancement method in the above embodiment.

[0142] In actual implementation, Figure 10 The functions of the acquisition module 701, the determination module 702 and the processing module 703 can all be accomplished by Figure 11 The processor 801 in the embodiment calls the computer program stored in the memory 802. The specific execution process can be referred to the description of the point cloud data enhancement method in the above embodiment, which will not be repeated here.

[0143] Optionally, the computer-readable storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0144] In an exemplary embodiment, a vehicle including a point cloud data enhancement device is also provided, and the vehicle can complete the point cloud data enhancement method in the above embodiment through the point cloud data enhancement device.

[0145] In an exemplary embodiment, the present application also provides a computer program product including one or more instructions, which can be executed by the processor 801 of the electronic device to complete the point cloud data enhancement method in the above embodiment.

[0146] It should be noted that when the instructions in the above-mentioned computer-readable storage medium or one or more instructions in the computer program product are executed by the processor of the electronic device, the various processes of the above-mentioned point cloud data enhancement method embodiment are implemented, and the same technical effect as the above-mentioned point cloud data enhancement method can be achieved. To avoid repetition, they will not be repeated here.

[0147] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete the full classification or partial functions described above.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0150] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0151] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the full classification part or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute the full classification part or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks or optical disks.

[0152] The above are only specific embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A point cloud data enhancement method, characterized in that: include: For any type of point cloud data among multiple types of point cloud data included in a training database corresponding to a target vehicle, obtaining at least one paste box corresponding to the any type of point cloud data from the training database, and determining multiple derived boxes corresponding to each of the at least one paste box, wherein the paste boxes corresponding to each type of point cloud data in the multiple types of point cloud data have different size ranges, the paste boxes are used to indicate virtual obstacles corresponding to the any type of point cloud data, and the derived boxes are used to indicate duplicate obstacles of the virtual obstacles corresponding to a paste box; Determine, based on the size of each paste box in the at least one paste box, a ground height standard deviation threshold and a ground height difference threshold corresponding to any type of point cloud data; Determine a point cloud height standard deviation and a point cloud height difference corresponding to each of a plurality of derivative frames corresponding to each of the at least one pasting frame; Based on a ground height standard deviation threshold and a ground height difference threshold corresponding to any type of point cloud data, determining at least one derivative frame corresponding to each paste frame from a plurality of derivative frames corresponding to each paste frame in the at least one paste frame, wherein the point cloud height standard deviation corresponding to each derivative frame in the at least one derivative frame is less than the ground height standard deviation threshold, and the point cloud height difference corresponding to each derivative frame is less than the ground height difference threshold; Determine an intersection-over-union (IOU) of each derived frame in at least one derived frame corresponding to each paste frame in the at least one paste frame, and determine a derived frame with an IOU of zero in the at least one derived frame corresponding to each paste frame as a target derived frame, where the IOU is an overlap ratio between the derived frame and the original point cloud data, and one paste frame corresponds to one target derived frame; The point cloud data corresponding to each paste box in the at least one paste box is added to the target derivative box corresponding to each paste box to obtain enhanced point cloud data.

2. The method according to claim 1, characterized in that The determining of the plurality of derived frames corresponding to each of the at least one pasting frame includes: Taking the target vehicle as an origin, rotating each paste box in the at least one paste box based on an original position to obtain a plurality of derivative boxes corresponding to each paste box in the at least one paste box; Based on the position of each paste box in the at least one paste box and the rotation angle corresponding to each derivative box, the position of each derivative box in the plurality of derivative boxes corresponding to each paste box in the at least one paste box is determined.

3. The method according to claim 1 or 2, characterized in that Each pasted box corresponds to multiple point clouds; The determining, based on the size of each paste box in the at least one paste box, a ground height standard deviation threshold and a ground height difference threshold corresponding to any type of point cloud data includes: Determining a maximum height value, a minimum height value, and a height standard deviation corresponding to each paste box based on a height value of each point cloud data in the plurality of point cloud data corresponding to each paste box in the at least one paste box; Determine a ground height standard deviation threshold corresponding to any one type of point cloud data based on a height standard deviation corresponding to each paste box in the at least one paste box; Based on the maximum height value and the minimum height value corresponding to each paste box in the at least one paste box, a ground height difference threshold corresponding to the any type of point cloud data is determined.

4. The method according to claim 1 or 2, characterized in that The method further comprises: Based on the category of the point cloud data, sequentially adding the point cloud data corresponding to each paste box in at least one paste box corresponding to each category of the point cloud data to the target derivative box corresponding to each paste box to obtain the enhanced point cloud data; For each pasting frame in at least one pasting frame corresponding to each type of point cloud data in the multiple types of point cloud data, and a target derived frame corresponding to each pasting frame, connecting a center point of each pasting frame or each target derived frame with a plurality of corner points, so as to divide each pasting frame or each target derived frame into a plurality of vertebrae; Transformation processing is performed on multiple vertebrae corresponding to each pasted frame or each target derivative frame to obtain further enhanced point cloud data, wherein the transformation processing includes at least one of the following: encryption and supplementation, sparseness, noise addition, exchange, mixing, and discarding.

5. A point cloud data enhancement device, characterized in that: It includes an acquisition module, a determination module and a processing module; The acquisition module is configured to acquire, from a training database corresponding to a target vehicle, at least one paste box corresponding to any one type of point cloud data among multiple types of point cloud data included in the training database, wherein the paste box corresponding to each type of point cloud data has a different size range, and the paste box is configured to indicate a virtual obstacle corresponding to the any one type of point cloud data; The determining module is configured to determine a plurality of derived frames corresponding to each of the at least one pasting frame, wherein the derived frames are configured to indicate duplicate obstacles of a virtual obstacle corresponding to a pasting frame; The determining module is further configured to determine a ground height standard deviation threshold and a ground height difference threshold corresponding to any type of point cloud data based on a size of each paste box in the at least one paste box; The determining module is further configured to determine a point cloud height standard deviation and a point cloud height difference corresponding to each of a plurality of derivative frames corresponding to each of the at least one pasting frame; The determining module is further configured to determine, from a plurality of derivative frames corresponding to each of the at least one pasting frame, at least one derivative frame corresponding to each pasting frame based on a ground height standard deviation threshold and a ground height difference threshold corresponding to any type of point cloud data, wherein the point cloud height standard deviation corresponding to each derivative frame in the at least one derivative frame is less than the ground height standard deviation threshold, and the point cloud height difference corresponding to each derivative frame is less than the ground height difference threshold; The determination module is further configured to determine an intersection-over-union (IOU) corresponding to each derivative frame in at least one derivative frame corresponding to each paste frame in the at least one paste frame, and determine a derivative frame having an IOU of zero in the at least one derivative frame corresponding to each paste frame as a target derivative frame, where the IOU is an overlap ratio between the derivative frame and the original point cloud data, and one paste frame corresponds to one target derivative frame; The processing module is used to add the point cloud data corresponding to each paste box in the at least one paste box to the target derivative box corresponding to each paste box to obtain enhanced point cloud data.

6. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that When the computer-executable instructions stored in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can perform the method according to any one of claims 1 to 4.

8. A vehicle, characterized in that: The vehicle comprises the point cloud data enhancement device according to claim 5 , and the vehicle is used to implement the method according to any one of claims 1 to 4 .

Images