A point cloud data processing method, device, equipment and storage medium
By merging categories in the point cloud dataset and stitching together the obstacle dataset, a dataset with strong generalization ability is generated, which solves the robustness problem of abnormal obstacle detection models in autonomous driving and achieves efficient obstacle detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA AUTOMOTIVE INNOVATION CORP
- Filing Date
- 2024-05-31
- Publication Date
- 2026-07-24
AI Technical Summary
In existing technologies, the number of road obstacle categories is far greater than the number of categories in the dataset, making it difficult to collect abnormal data. This leads to class prediction conflicts in the network model after training the model parameters, resulting in insufficient robustness of the dataset.
By merging categories in the point cloud dataset of the road scene and stitching together the point cloud dataset of obstacles, a dataset with strong generalization ability is generated. Then, by using the initial dataset and the second point cloud dataset, abnormal instance stitching and instance enhancement processing are performed to generate the target point cloud dataset.
It improves the robustness of the dataset for models that detect abnormal obstacles in autonomous driving, reduces the time and cost of manual annotation, and improves detection accuracy.
Smart Images
Figure CN118521994B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of autonomous driving technology, and in particular to a point cloud data processing method, apparatus, device, and storage medium. Background Technology
[0002] In the field of autonomous driving, the perception and localization of the vehicle's surroundings is particularly important, and road obstacle detection technology is one of the essential key technologies in autonomous driving systems. Data collected by radar can be used to create point cloud maps, thereby depicting the shape, size, and distance of obstacles. Deep learning can utilize massive amounts of perception data to recognize and learn features such as road markings and obstacles, enabling the perception of the vehicle's surroundings and generating appropriate driving strategies.
[0003] In existing technologies, the number of obstacle categories on the road is far greater than the number of categories in the dataset, making it difficult to collect abnormal data. Labeling the collected data consumes a lot of time and cost. Network models trained on existing datasets have conflicts in category prediction. Therefore, how to improve the robustness of abnormal obstacle detection models to the dataset has become an urgent problem to be solved. Summary of the Invention
[0004] To address the problems of existing technologies, this application provides a point cloud data processing method, apparatus, device, and storage medium. By merging the point cloud dataset of a road scene by category, the road point cloud data is stitched together with the point cloud dataset of obstacles to generate a dataset with strong generalization ability, thus solving the problem of improving the robustness of the dataset for detecting abnormal obstacles in autonomous driving.
[0005] According to one aspect of this application, a point cloud data processing method is provided, the method comprising:
[0006] Obtain a first point cloud dataset and a second point cloud dataset; the first point cloud dataset includes multiple category labels, and each category label corresponds to a subset of point cloud data.
[0007] Based on the multiple category labels, the first point cloud dataset is merging categories to obtain an initial dataset;
[0008] The initial dataset and the second point cloud dataset are concatenated with abnormal instances to obtain the abnormal instance concatenation result;
[0009] The abnormal instance splicing result is subjected to instance enhancement processing to obtain the target point cloud dataset; the target point cloud dataset includes multiple labeled target point cloud data.
[0010] In one possible implementation, the initial dataset includes a preset number of target category labels.
[0011] The initial dataset is obtained by merging categories in the first point cloud dataset based on the multiple category labels, including:
[0012] Based on the similarity between each category label and the preset number of target category labels, each category label is mapped to a target category label that meets the preset mapping conditions, thereby obtaining a subset of target point cloud data corresponding to each target category label;
[0013] The initial dataset is determined based on the subset of target point cloud data corresponding to each target category label.
[0014] In one possible implementation, the step of mapping each category label to a target category label that satisfies a preset mapping condition based on the similarity between each category label and the preset number of target category labels, to obtain a subset of target point cloud data corresponding to each target category label, includes:
[0015] Calculate the similarity between each category label and each target category label to obtain the similarity calculation results;
[0016] If the similarity calculation results meet the preset similarity conditions, a subset of point cloud data corresponding to each category label is determined as the target point cloud data subset corresponding to the target category label that meets the preset similarity conditions.
[0017] In one possible implementation, the second point cloud dataset includes multiple abnormal instance point cloud data, each abnormal instance point cloud data corresponding to abnormal instance information, and the target point cloud data subset includes multiple target point cloud data.
[0018] The step of concatenating the initial dataset and the second point cloud dataset to obtain the abnormal instance concatenation result includes:
[0019] The abnormal instance point cloud data is concatenated with the target point cloud data to obtain the concatenation result of each target point cloud data.
[0020] And the abnormal instance information corresponding to the abnormal instance point cloud data is fused with the target category label corresponding to the target point cloud data to obtain the fused label data of each target point cloud data;
[0021] The stitching result of each target point cloud data and the fused label data of each target point cloud data are determined as the stitching result of the abnormal instance.
[0022] In one possible implementation, the target point cloud dataset includes multiple labeled target data.
[0023] The instance augmentation process performed on the concatenated result of the abnormal instances to obtain the target point cloud dataset includes:
[0024] Instance augmentation processing is performed on the stitching result of each target point cloud data to obtain each instance-augmented target point cloud data;
[0025] Based on the enhanced target point cloud data of each instance, the corresponding fused label data is updated to obtain the label data of the enhanced target point cloud data of each instance.
[0026] The target dataset is generated by determining the target point cloud data for each instance to be augmented as target point cloud data and the label data for each instance to be augmented as target point cloud data labels.
[0027] In one possible implementation, an initial abnormal instance segmentation network is trained based on the target dataset to obtain a target abnormal instance segmentation network, including:
[0028] Each target point cloud data is input into the initial abnormal instance segmentation network to perform instance segmentation processing on each target point cloud data, thereby obtaining the predicted label for each target point cloud data.
[0029] The matching result is obtained by matching the label of each target point cloud data with the predicted label of each target point cloud data;
[0030] If the matching result does not meet the preset conditions, determine the difference information between the label and the predicted label, and update the network parameters of the initial abnormal instance segmentation network based on the difference information;
[0031] Until the matching result meets the preset condition, the initial abnormal instance segmentation network corresponding to the matching result that meets the preset condition is determined as the target abnormal instance segmentation network.
[0032] In one possible implementation, the target anomaly instance segmentation network is used to perform instance segmentation on radar point cloud data in a driving scenario.
[0033] The radar point cloud data in the driving scenario is segmented into instances based on the aforementioned target anomaly instance segmentation network, including:
[0034] Acquire radar point cloud data in a driving scenario; the radar point cloud data represents the road scene from a top-down perspective.
[0035] The radar point cloud data is input into the target anomaly instance segmentation network for instance segmentation processing to obtain instance segmentation processing results, which represent obstacle information in the road scene.
[0036] On the other hand, a point cloud data processing apparatus is provided, the apparatus comprising:
[0037] The data acquisition module is used to acquire a first point cloud dataset and a second point cloud dataset; the first point cloud dataset includes multiple category labels, and each category label corresponds to a subset of point cloud data;
[0038] The merging processing module is used to perform category merging processing on the first point cloud dataset based on the multiple category labels to obtain an initial dataset;
[0039] The instance stitching module is used to stitch abnormal instances into the initial dataset and the second point cloud dataset to obtain the abnormal instance stitching result;
[0040] The instance enhancement processing module is used to perform instance enhancement processing on the abnormal instance splicing result to obtain the target point cloud dataset; the target point cloud dataset includes multiple labeled target point cloud data.
[0041] In one possible implementation, the initial dataset includes a preset number of target category labels, and the merging processing module is used to:
[0042] Based on the similarity between each category label and the preset number of target category labels, each category label is mapped to a target category label that meets the preset mapping conditions, thereby obtaining a subset of target point cloud data corresponding to each target category label;
[0043] The initial dataset is determined based on the subset of target point cloud data corresponding to each target category label.
[0044] In one possible implementation, the merging module is further configured to:
[0045] Calculate the similarity between each category label and each target category label to obtain the similarity calculation results;
[0046] If the similarity calculation results meet the preset similarity conditions, a subset of point cloud data corresponding to each category label is determined as the target point cloud data subset corresponding to the target category label that meets the preset similarity conditions.
[0047] In one possible implementation, the second point cloud dataset includes multiple abnormal instance point cloud data, each abnormal instance point cloud data corresponding to abnormal instance information; the target point cloud data subset includes multiple target point cloud data; and the instance stitching module is used for:
[0048] The abnormal instance point cloud data is concatenated with the target point cloud data to obtain the concatenation result of each target point cloud data.
[0049] And the abnormal instance information corresponding to the abnormal instance point cloud data is fused with the target category label corresponding to the target point cloud data to obtain the fused label data of each target point cloud data;
[0050] The stitching result of each target point cloud data and the fused label data of each target point cloud data are determined as the stitching result of the abnormal instance.
[0051] In one possible implementation, the target point cloud dataset includes multiple labeled target data, and the instance enhancement processing module is used for:
[0052] Instance augmentation processing is performed on the stitching result of each target point cloud data to obtain each instance-augmented target point cloud data;
[0053] Based on the enhanced target point cloud data of each instance, the corresponding fused label data is updated to obtain the label data of the enhanced target point cloud data of each instance.
[0054] The target dataset is generated by determining the target point cloud data for each instance to be augmented as target point cloud data and the label data for each instance to be augmented as target point cloud data labels.
[0055] In one possible implementation, the point cloud data processing device further includes an abnormal instance segmentation network training module, which is used to:
[0056] Each target point cloud data is input into the initial abnormal instance segmentation network to perform instance segmentation processing on each target point cloud data, thereby obtaining the predicted label for each target point cloud data.
[0057] The matching result is obtained by matching the label of each target point cloud data with the predicted label of each target point cloud data;
[0058] If the matching result does not meet the preset conditions, determine the difference information between the label and the predicted label, and update the network parameters of the initial abnormal instance segmentation network based on the difference information;
[0059] Until the matching result meets the preset condition, the initial abnormal instance segmentation network corresponding to the matching result that meets the preset condition is determined as the target abnormal instance segmentation network.
[0060] In one possible implementation, the target anomaly instance segmentation network is used to perform instance segmentation on radar point cloud data in a driving scenario. The point cloud data processing device further includes an instance segmentation module, which is used to:
[0061] Acquire radar point cloud data in a driving scenario; the radar point cloud data represents the road scene from a top-down perspective.
[0062] The radar point cloud data is input into the target anomaly instance segmentation network for instance segmentation processing to obtain instance segmentation processing results, which represent obstacle information in the road scene.
[0063] On the other hand, an electronic device is provided, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the point cloud data processing method of any of the above aspects.
[0064] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement the point cloud data processing method as described above.
[0065] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the point cloud data processing method of any of the above aspects.
[0066] This application embodiment obtains a first point cloud dataset and a second point cloud dataset. The first point cloud dataset includes multiple category labels, each category label corresponding to a subset of point cloud data. Based on the multiple category labels, the first point cloud dataset undergoes category merging processing to obtain an initial dataset. Abnormal instances are concatenated between the initial dataset and the second point cloud dataset to obtain an abnormal instance concatenation result. Instance augmentation processing is performed on the abnormal instance concatenation result to obtain a target point cloud dataset. The target point cloud dataset includes multiple labeled target point cloud data. By merging the point cloud dataset of a road scene and concatenating the road point cloud data with the obstacle point cloud dataset, a dataset with strong generalization ability is generated, solving the problem of improving the robustness of the dataset for detecting abnormal obstacles in autonomous driving. Attached Figure Description
[0067] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0068] Figure 1 This is a flowchart illustrating a point cloud data processing method provided in an embodiment of this application;
[0069] Figure 2 This is a schematic diagram of a network structure provided in an embodiment of this application;
[0070] Figure 3 This is a schematic diagram of a segmentation prediction result provided in an embodiment of this application;
[0071] Figure 4 This is a schematic diagram of another segmentation prediction result provided in an embodiment of this application;
[0072] Figure 5 This is a structural block diagram of a point cloud data processing device provided in an embodiment of this application. Detailed Implementation
[0073] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0074] This application provides a point cloud data processing method. In the emerging field of radar point clouds, how to efficiently perform panoramic segmentation remains an unsolved research problem. This application's embodiments are based on a fast and robust LiDAR point cloud panoramic segmentation framework called Panoptic-PolarNet. This framework learns semantic segmentation and category-independent instance clustering in a single inference network, using polar coordinate bird's eye view (BEV) representation, thereby avoiding the occlusion problem between instances in urban street scenes. See also... Figure 1 The point cloud data processing method includes steps S101 to S107.
[0075] In step S101, a first point cloud dataset and a second point cloud dataset are obtained; the first point cloud dataset includes multiple category labels, and each category label corresponds to a subset of point cloud data.
[0076] In one possible implementation, the first point cloud dataset is SemanticKITTI, which is a point-by-point annotation dataset of point cloud sequences, annotated with 28 classes. The point-by-point annotation of the point cloud sequence corresponding to each class shows unprecedented detail and is suitable for various tasks.
[0077] In one possible implementation, the second point cloud dataset is the PCL (Point Cloud Library) dataset produced by Stanford University. The PCL dataset contains various sculptural point cloud scans, the shapes of which are different from the obstacles detected on the road, and is used to simulate sudden road situations.
[0078] In one possible implementation, the multiple category labels are the 28 classes of SemanticKITTI.
[0079] In one possible implementation, each category label corresponds to a subset of point cloud data, which is the point cloud sequence corresponding to each class.
[0080] In step S103, the first point cloud dataset is merging categories based on the multiple category labels to obtain an initial dataset.
[0081] In one possible implementation, the initial dataset includes a preset number of target category labels.
[0082] The initial dataset is obtained by merging categories in the first point cloud dataset based on the multiple category labels, including:
[0083] Based on the similarity between each category label and the preset number of target category labels, each category label is mapped to a target category label that meets the preset mapping conditions, thereby obtaining a subset of target point cloud data corresponding to each target category label;
[0084] The initial dataset is determined based on the subset of target point cloud data corresponding to each target category label.
[0085] In one possible implementation, based on the feature similarity between the subsets of point cloud data corresponding to each category label, multiple category labels are merged into one target category label, and each category label corresponds to one target category label, resulting in a preset number of target category labels.
[0086] In one possible implementation, the subset of point cloud data corresponding to a category label of a target category label is taken as the subset of target point cloud data corresponding to that target category label.
[0087] In one possible implementation, the initial dataset is generated based on a preset number of target category labels and a preset number of target point cloud data subsets corresponding to the target category labels.
[0088] In one possible implementation, categories in the Semantic-kitti dataset such as road and parking, sidewalk and other-ground, and vegetation and terrain are similar in features to each other, and similar categories can be merged when validating on the vehicle dataset.
[0089] In this implementation, the first point cloud dataset is categorized based on the multiple category labels to obtain an initial dataset, which is then used to classify objects that do not belong to these categories or are difficult to distinguish.
[0090] In one possible implementation, the step of mapping each category label to a target category label that satisfies a preset mapping condition based on the similarity between each category label and the preset number of target category labels, to obtain a subset of target point cloud data corresponding to each target category label, includes:
[0091] Calculate the similarity between each category label and each target category label to obtain the similarity calculation results;
[0092] If the similarity calculation results meet the preset similarity conditions, a subset of point cloud data corresponding to each category label is determined as the target point cloud data subset corresponding to the target category label that meets the preset similarity conditions.
[0093] In one possible implementation, a preset number of target category labels are set according to the scenario, and the semantic similarity between each category label and each target category label is calculated to obtain the similarity value between each category label and each target category label, which is used as the similarity calculation result.
[0094] In one possible implementation, the target category label corresponding to the largest similarity value among the similarity calculation results of each category label is taken as the target category label corresponding to the similarity calculation result that satisfies the preset similarity conditions, and the subset of point cloud data corresponding to the category label is determined as the target point cloud data subset of the target category label corresponding to the similarity calculation result that satisfies the preset similarity conditions.
[0095] In this implementation, based on the similarity between each category label and the preset number of target category labels, each category label is mapped to a target category label that meets the preset mapping conditions, thereby obtaining a subset of target point cloud data corresponding to each target category label. Mapping based on similarity improves the accuracy of category classification.
[0096] In step S105, abnormal instances are concatenated between the initial dataset and the second point cloud dataset to obtain the abnormal instance concatenation result.
[0097] In one possible implementation, the second point cloud dataset includes multiple abnormal instance point cloud data, each abnormal instance point cloud data corresponding to abnormal instance information, and the target point cloud data subset includes multiple target point cloud data.
[0098] The step of concatenating the initial dataset and the second point cloud dataset to obtain the abnormal instance concatenation result includes:
[0099] The abnormal instance point cloud data is concatenated with the target point cloud data to obtain the concatenation result of each target point cloud data.
[0100] And the abnormal instance information corresponding to the abnormal instance point cloud data is fused with the target category label corresponding to the target point cloud data to obtain the fused label data of each target point cloud data;
[0101] The stitching result of each target point cloud data and the fused label data of each target point cloud data are determined as the stitching result of the abnormal instance.
[0102] In one possible implementation, the second point cloud dataset is the PCL (Point Cloud Library) dataset produced by Stanford University, which contains various point cloud scans of sculptures.
[0103] In one possible implementation, the initial dataset and the second point cloud dataset are used to simulate sudden road conditions.
[0104] In one possible implementation, the abnormal instance point cloud data is concatenated with the target point cloud data to obtain a concatenated result for each target point cloud data. Specifically, the concatenation requires two main steps: registration and optimization.
[0105] In one possible implementation, registration is typically accomplished through feature matching or rigid transformation, based on the spatial correspondence between the data, by registering the abnormal instance point cloud data with the target point cloud data.
[0106] In one possible implementation, the point cloud data of the anomalous instances and the target point cloud data are optimized to minimize the error in these correspondences. Specifically, the Iterative ClosestPoint (ICP) algorithm or a variant thereof is used for optimization.
[0107] In one possible implementation, while concatenating the abnormal instance point cloud data with the target point cloud data, the abnormal instance information corresponding to the abnormal instance point cloud data and the target category label corresponding to the target point cloud data are semantically fused to obtain fused label data for each target point cloud data.
[0108] In this implementation, within the SemanticKITTI scenario, the registration and optimization process is more robust and accurate because the data itself already contains rich semantic information. Furthermore, the semantic information also helps to better understand and interpret the splicing results.
[0109] In step S107, the abnormal instance splicing result is subjected to instance enhancement processing to obtain the target point cloud dataset; the target point cloud dataset includes multiple labeled target point cloud data.
[0110] In one possible implementation, the target point cloud dataset includes multiple labeled target data.
[0111] The instance augmentation process performed on the concatenated result of the abnormal instances to obtain the target point cloud dataset includes:
[0112] Instance augmentation processing is performed on the stitching result of each target point cloud data to obtain each instance-augmented target point cloud data;
[0113] Based on the enhanced target point cloud data of each instance, the corresponding fused label data is updated to obtain the label data of the enhanced target point cloud data of each instance.
[0114] The target dataset is generated by determining the target point cloud data for each instance to be augmented as target point cloud data and the label data for each instance to be augmented as target point cloud data label.
[0115] In one possible implementation, the point cloud data corresponding to the fused label data of each target point cloud data is an instance.
[0116] In one possible implementation, geometric transformations (such as flipping, translating, rotating, or adding noise) or color transformations (such as brightness, contrast, saturation, or channel shuffling) are performed on each instance of the stitched result of the target point cloud data to increase the diversity of the dataset.
[0117] In one possible implementation, the concatenated result of the abnormal instances is oversampled, and a predetermined number of instances are randomly selected from the entire training set and inserted into the current data sequence. The probability of each category being selected is proportional to the inverse of its point distribution, and the imported points retain the same coordinates and reflection values as their original data. Specifically, the predetermined number is preferably 5.
[0118] In one possible implementation, the position of an instance on the horizontal plane is changed without altering its projection onto the sensor. To keep the projection invariant, the transformation is restricted to either rotation around a center or reflection around the center of a view plane. Both transformations are applied to each instance with a probability of 20%, and a small translation and rotation are performed on each instance as simulated noise. We sample the translation amounts [Δx, Δy, Δz] with a normal distribution N(0, 0.25) and the rotation angle Δθ with a uniform distribution U(-20 / π, +20 / π).
[0119] In this implementation, the abnormal instance splicing result is subjected to instance enhancement processing to obtain the target point cloud dataset, which improves the diversity and robustness of the dataset.
[0120] In one possible implementation, the initial abnormal instance segmentation network is trained based on the target dataset to obtain the target abnormal instance segmentation network, including:
[0121] Each target point cloud data is input into the initial abnormal instance segmentation network to perform instance segmentation processing on each target point cloud data, thereby obtaining the predicted label for each target point cloud data.
[0122] The matching result is obtained by matching the label of each target point cloud data with the predicted label of each target point cloud data;
[0123] If the matching result does not meet the preset conditions, determine the difference information between the label and the predicted label, and update the network parameters of the initial abnormal instance segmentation network based on the difference information;
[0124] Until the matching result meets the preset condition, the initial abnormal instance segmentation network corresponding to the matching result that meets the preset condition is determined as the target abnormal instance segmentation network.
[0125] In one possible implementation, such as Figure 2 This paper presents an overview of the network structure of the initial abnormal instance segmentation network. It is based on the Panoptic-PolarNet framework, which is a radar point cloud panoramic segmentation framework. It uses polar bird's-eye view (BEV) to represent the original point cloud. After point cloud feature extraction, Unet, and backbone, the output is sent to two heads (detection heads), which output the center point of the instance and the offset of the instance from the BEV perspective, as well as the semantic segmentation prediction result.
[0126] In one possible implementation, the Panoptic-PolarNet framework consists of the following four parts:
[0127] (1) A network Polar BEV encode encodes the raw point cloud data into a fixed-size two-dimensional polar BEV representation.
[0128] (2) Shared encoder-decoder backbone network,
[0129] (3) Two independent semantic segmentation and instance segmentation (semantic segmentation can provide a foreground mask, which is then fused with the instance feature to form a panoramic segmentation map)
[0130] (4) The fusion step merges the above predictions into the last segmentation result that displays the panorama.
[0131] In one possible implementation, Panoptic-PolarNet uses a 2D grid under BEV (Browser-Earth-Oriented) coordinate system to represent the data. This leverages 2D convolution to improve data processing efficiency, and there are virtually no overlapping objects in the height direction, thus preventing information loss. However, while typical 2D grids use Cartesian coordinates, this method uses polar coordinates, which reduces quantization errors in close-range regions and improves feature extraction capabilities. Furthermore, point feature extraction employs a Multilayer Perceptron (MLP) combined with Max Pooling, ultimately yielding a fixed-length feature vector (C = 512) for each grid cell. The entire feature extraction process is very concise, involving only quantization, MLP, and Pooling, resulting in very low computational complexity.
[0132] In one possible implementation, the backbone network also employs the very simple U-Net (Semantic Segmentation Network), with the semantic and instance branches sharing all features except for the last layer, which greatly helps reduce computation. The semantic branch outputs multiple predictions at each grid location and maps them back to a 3D grid to distinguish semantic categories at different heights (e.g., objects and road surfaces). The loss during training is also calculated in 3D grid space. The instance branch is similar to the CenterNet (single-stage object detection algorithm) structure popular in vision tasks, outputting the confidence score (heatmap) of the object center and the offset of the center point relative to the grid center position at each 2D grid.
[0133] In one possible implementation, with the above information, the panoptic segmentation module first generates Top-K accurate object center points on the heatmap using Non-Maximum Suppression (NMS), and determines their precise locations based on the offset. Then, a foreground mask is extracted from the semantic segmentation results. For each point in the foreground, the nearest object center is found, and clusters are formed based on these centers. The category of each cluster is determined by the semantic category of the majority of points within it. All these operations can be performed on the GPU, resulting in high computational efficiency. Specifically, Top-K accuracy is used to calculate the percentage of the top K most probable predictions that contain the correct label.
[0134] In one possible implementation, Panoptic-PolarNet provides 3D semantic segmentation results, with each point having a semantic label. The instance branch provides the object center location, which, combined with the foreground points output by the semantic branch, yields the segmentation result for each instance. This completes the panoptic segmentation task. Furthermore, the 2D mesh data structure, U-Net, and center point prediction included in the network are all very concise structures that can be efficiently implemented on a GPU.
[0135] In one possible implementation, to predict the center point and offset of the BEV bird's-eye view, pixels with close center points of the instance segmentation head are merged. Ultimately, the instance obtained by this method is independent of the data category, thereby avoiding conflicts between data classification and bounding box annotation functions in other detection heads.
[0136] In one possible implementation, to generate a heatmap of the center points, the centroid of each instance is encoded as a ground truth center map using a 2D Gaussian distribution during network training. For each pixel p in the BEV map, the heatmap is calculated using the following formula:
[0137]
[0138] Where p is the pixel value, c i σ is the centroid of an instance in the polar BEV coordinate system. 2 is the standard deviation of pixel values around ci, and maxi is the normalization coefficient.
[0139] In one possible implementation, the Panoptic-PolarNet framework, a radar point cloud panoramic segmentation framework, uses polar bird's-eye view (BEV) representation to learn semantic segmentation and category-independent instance clustering in a single inference network. This can circumvent the occlusion problem between instances in urban street scenes. It proposes an adaptive instance augmentation technique and a novel adversarial point cloud pruning method to improve the network's learnability.
[0140] In this implementation, the bottom-up design only provides class-independent instance grouping, avoiding conflicts between class predictions between segmentation and instance branches. Furthermore, instance heads can be trained without bounding box annotations, which can avoid the occlusion problem between instances in urban street scenes and improve the learnability of the network.
[0141] In one possible implementation, the target anomaly instance segmentation network is used to perform instance segmentation on radar point cloud data in a driving scenario.
[0142] The radar point cloud data in the driving scenario is segmented into instances based on the aforementioned target anomaly instance segmentation network, including:
[0143] Acquire radar point cloud data in a driving scenario; the radar point cloud data represents the road scene from a top-down perspective.
[0144] The radar point cloud data is input into the target anomaly instance segmentation network for instance segmentation processing to obtain instance segmentation processing results, which represent obstacle information in the road scene.
[0145] In one possible implementation, in a real-world application scenario, radar detection data of the vehicle's environment is acquired to obtain radar point cloud data.
[0146] In one possible implementation, the obtained radar point cloud data is input into the target anomaly instance segmentation network for instance segmentation processing to obtain instance segmentation processing results, which represent obstacle information of the road scene.
[0147] In one possible implementation, the obtained radar point cloud data is input into the target anomaly instance segmentation network. After point cloud feature extraction, Unet, and backbone processing, the data is output to two heads, which respectively output the center point and offset of the instance from the BEV perspective, as well as the semantic segmentation prediction result. Figure 3 and Figure 4 As shown.
[0148] In this implementation, the target abnormal instance segmentation network trained on the target point cloud dataset is used to segment radar point cloud data in driving scenarios. It can detect abnormal obstacles on the road, reduce the time and cost of manual annotation during training, and has a high detection accuracy.
[0149] Figure 5 This diagram illustrates the structure of a point cloud data processing device 500 according to an embodiment of this application. This device has the function of implementing the point cloud data processing method described in the above-described method embodiments. This function can be implemented by hardware or by hardware executing corresponding software. Figure 5 As shown, the device may include:
[0150] The data acquisition module 501 is used to acquire a first point cloud dataset and a second point cloud dataset; the first point cloud dataset includes multiple category labels, and each category label corresponds to a subset of point cloud data.
[0151] The merging processing module 502 is used to perform category merging processing on the first point cloud dataset based on the multiple category labels to obtain an initial dataset;
[0152] The instance stitching module 503 is used to stitch abnormal instances into the initial dataset and the second point cloud dataset to obtain the abnormal instance stitching result;
[0153] The instance enhancement processing module 504 is used to perform instance enhancement processing on the abnormal instance splicing result to obtain the target point cloud dataset; the target point cloud dataset includes multiple labeled target point cloud data.
[0154] In one possible implementation, the initial dataset includes a preset number of target category labels, and the merging processing module 502 is used to:
[0155] Based on the similarity between each category label and the preset number of target category labels, each category label is mapped to a target category label that meets the preset mapping conditions, thereby obtaining a subset of target point cloud data corresponding to each target category label;
[0156] The initial dataset is determined based on the subset of target point cloud data corresponding to each target category label.
[0157] In one possible implementation, the merging processing module 502 is further configured to:
[0158] Calculate the similarity between each category label and each target category label to obtain the similarity calculation results;
[0159] If the similarity calculation results meet the preset similarity conditions, a subset of point cloud data corresponding to each category label is determined as the target point cloud data subset corresponding to the target category label that meets the preset similarity conditions.
[0160] In one possible implementation, the second point cloud dataset includes multiple abnormal instance point cloud data, each abnormal instance point cloud data corresponding to abnormal instance information; the target point cloud data subset includes multiple target point cloud data; and the instance stitching module 503 is used for:
[0161] The abnormal instance point cloud data is concatenated with the target point cloud data to obtain the concatenation result of each target point cloud data.
[0162] And the abnormal instance information corresponding to the abnormal instance point cloud data is fused with the target category label corresponding to the target point cloud data to obtain the fused label data of each target point cloud data;
[0163] The stitching result of each target point cloud data and the fused label data of each target point cloud data are determined as the stitching result of the abnormal instance.
[0164] In one possible implementation, the target point cloud dataset includes multiple labeled target data, and the instance enhancement processing module 504 is used for:
[0165] Instance augmentation processing is performed on the stitching result of each target point cloud data to obtain each instance-augmented target point cloud data;
[0166] Based on the enhanced target point cloud data of each instance, the corresponding fused label data is updated to obtain the label data of the enhanced target point cloud data of each instance.
[0167] The target dataset is generated by determining the target point cloud data for each instance to be augmented as target point cloud data and the label data for each instance to be augmented as target point cloud data labels.
[0168] In one possible implementation, the point cloud data processing device further includes an abnormal instance segmentation network training module 505, which is used for:
[0169] Each target point cloud data is input into the initial abnormal instance segmentation network to perform instance segmentation processing on each target point cloud data, thereby obtaining the predicted label for each target point cloud data.
[0170] The matching result is obtained by matching the label of each target point cloud data with the predicted label of each target point cloud data;
[0171] If the matching result does not meet the preset conditions, determine the difference information between the label and the predicted label, and update the network parameters of the initial abnormal instance segmentation network based on the difference information;
[0172] Until the matching result meets the preset condition, the initial abnormal instance segmentation network corresponding to the matching result that meets the preset condition is determined as the target abnormal instance segmentation network.
[0173] In one possible implementation, the target anomaly instance segmentation network is used to perform instance segmentation on radar point cloud data in a driving scenario. The point cloud data processing device further includes an instance segmentation module 506, which is used to:
[0174] Acquire radar point cloud data in a driving scenario; the radar point cloud data represents the road scene from a top-down perspective.
[0175] The radar point cloud data is input into the target anomaly instance segmentation network for instance segmentation processing to obtain instance segmentation processing results, which represent obstacle information in the road scene.
[0176] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.
[0177] This application provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement any of the point cloud data processing methods provided in the above method embodiments.
[0178] Memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. Memory can primarily include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for the functions, etc.; the data storage area can store data created based on the use of the device, etc. Furthermore, memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory.
[0179] Embodiments of this application also provide a computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a point cloud data processing method. The at least one instruction or the at least one program is loaded and executed by the processor to implement any of the point cloud data processing methods provided in the above-described method embodiments.
[0180] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0181] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0182] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0183] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0184] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A point cloud data processing method, characterized in that, The method includes: Obtain a first point cloud dataset and a second point cloud dataset; the first point cloud dataset includes multiple category labels, each category label corresponds to a subset of point cloud data; the second point cloud dataset includes multiple abnormal instance point cloud data, each abnormal instance point cloud data corresponds to abnormal instance information; Based on the multiple category labels, the first point cloud dataset is merging categories to obtain an initial dataset; the initial dataset includes a preset number of target category labels. The point cloud data of the abnormal instance is concatenated with the target point cloud data to obtain the concatenation result of each target point cloud data. And the abnormal instance information corresponding to the abnormal instance point cloud data is fused with the target category label corresponding to the target point cloud data to obtain the fused label data of each target point cloud data; The stitching result of each target point cloud data and the fused label data of each target point cloud data are identified as the stitching result of the abnormal instance; Instance augmentation processing is performed on the stitching result of each target point cloud data to obtain each instance-augmented target point cloud data; Based on the enhanced target point cloud data of each instance, the corresponding fused label data is updated to obtain the label data of the enhanced target point cloud data of each instance. Each instance of augmented target point cloud data is identified as target point cloud data, and the label data of each instance of augmented target point cloud data is identified as target point cloud data label, thereby generating a target point cloud dataset; the target point cloud dataset includes multiple labeled target point cloud data.
2. The point cloud data processing method according to claim 1, characterized in that, The initial dataset is obtained by merging categories in the first point cloud dataset based on the multiple category labels, including: Based on the similarity between each category label and the preset number of target category labels, each category label is mapped to a target category label that meets the preset mapping conditions, thereby obtaining a subset of target point cloud data corresponding to each target category label; The initial dataset is determined based on the subset of target point cloud data corresponding to each target category label.
3. The point cloud data processing method according to claim 2, characterized in that, The step of mapping each category label to a target category label that meets a preset mapping condition based on the similarity between each category label and the preset number of target category labels, to obtain a subset of target point cloud data corresponding to each target category label, includes: Calculate the similarity between each category label and each target category label to obtain the similarity calculation results; If the similarity calculation results meet the preset similarity conditions, a subset of point cloud data corresponding to each category label is determined as the target point cloud data subset corresponding to the target category label that meets the preset similarity conditions.
4. The point cloud data processing method according to claim 1, characterized in that, After generating the target point cloud dataset, the method further includes: Each target point cloud data is input into the initial abnormal instance segmentation network to perform instance segmentation processing on each target point cloud data, and obtain the predicted label of each target point cloud data. The matching result is obtained by matching the label of each target point cloud data with the predicted label of each target point cloud data; If the matching result does not meet the preset conditions, determine the difference information between the label and the predicted label, and update the network parameters of the initial abnormal instance segmentation network based on the difference information; When the matching result meets the preset condition, the initial abnormal instance segmentation network corresponding to the matching result that meets the preset condition is determined as the target abnormal instance segmentation network.
5. The point cloud data processing method according to claim 4, characterized in that, The target anomaly instance segmentation network is used to segment radar point cloud data in driving scenarios into instances. The radar point cloud data in the driving scenario is segmented into instances based on the aforementioned target anomaly instance segmentation network, including: Acquire radar point cloud data in a driving scenario; the radar point cloud data represents the road scene from a top-down perspective. The radar point cloud data is input into the target anomaly instance segmentation network for instance segmentation processing to obtain instance segmentation processing results, which represent obstacle information in the road scene.
6. A point cloud data processing device, characterized in that, The device includes: The data acquisition module is used to acquire a first point cloud dataset and a second point cloud dataset. The first point cloud dataset includes multiple category labels, and each category label corresponds to a subset of point cloud data. The second point cloud dataset includes multiple abnormal instance point cloud data, and each abnormal instance point cloud data corresponds to abnormal instance information. The merging processing module is used to perform category merging processing on the first point cloud dataset based on the multiple category labels to obtain an initial dataset; the initial dataset includes a preset number of target category labels; The instance stitching module is used to stitch the abnormal instance point cloud data with the target point cloud data to obtain the stitching result of each target point cloud data. And the abnormal instance information corresponding to the abnormal instance point cloud data is fused with the target category label corresponding to the target point cloud data to obtain the fused label data of each target point cloud data; The stitching result of each target point cloud data and the fused label data of each target point cloud data are identified as the stitching result of the abnormal instance; The instance enhancement processing module is used to perform instance enhancement processing on the stitching result of each target point cloud data to obtain each instance-enhanced target point cloud data; based on each instance-enhanced target point cloud data, the corresponding fused label data is updated to obtain the label data of each instance-enhanced target point cloud data; each instance-enhanced target point cloud data is determined as target point cloud data, and the label data of each instance-enhanced target point cloud data is determined as target point cloud data label, generating a target point cloud dataset; the target point cloud dataset includes multiple labeled target point cloud data.
7. An electronic device, characterized in that, The method includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the point cloud data processing method as described in any one of claims 1 to 5.
8. A computer-readable storage medium storing at least one instruction or at least one program, said at least one instruction or said at least one program being loaded and executed by a processor to implement the point cloud data processing method as described in any one of claims 1 to 5.