Point cloud instance segmentation sample automatic construction method and system, and laser radar
By automatically annotating panoramic images to generate point cloud instance segmentation samples, the high cost and low efficiency problems of existing technologies are solved, realizing automated and high-precision point cloud instance segmentation, which is suitable for training on urban scene data.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- UNRE (SHANGHAI) INFORMATION TECHNOLOGY CO LTD
- Filing Date
- 2026-03-17
- Publication Date
- 2026-06-09
AI Technical Summary
Existing deep learning methods for 3D point cloud instance segmentation suffer from high costs, high human intervention, and low efficiency, especially when training on city-scale data, making it difficult to achieve high-precision segmentation.
By acquiring the spatial mapping relationship between point cloud data and panoramic image data, the panoramic images are automatically labeled to generate point cloud instance segmentation samples. An automated production line is then built, including spherical unfolding, perspective subview segmentation, back projection fusion, and data augmentation, forming an automatic closed-loop training system.
It achieves automated annotation for point cloud instance segmentation, reduces manual costs, supports training on large-scale urban scene data, improves segmentation accuracy and class discrimination ability, and is applicable to various point cloud instance segmentation network architectures.
Smart Images
Figure CN122176443A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method, system, and lidar for automatically constructing point cloud instance segmentation samples. Background Technology
[0002] With the widespread application of mobile laser scanning systems and panoramic vision acquisition devices, point cloud data and panoramic image data of large-scale urban scenes can be acquired simultaneously. Point cloud data has accurate three-dimensional geometric information, while image data has high-resolution texture and semantic expression capabilities, and the two have natural complementary advantages.
[0003] While existing deep learning methods for 3D point cloud instance segmentation have made some progress, they generally suffer from the following problems: high-quality instance annotation samples are extremely scarce, and manual point-by-point annotation is costly; the data production process is highly dependent on manual participation, making it difficult to support city-level scale data training; model testing and evaluation still require manual verification, which is inefficient and highly subjective. Summary of the Invention
[0004] The technical problem to be solved by this invention is to overcome the shortcomings of high cost, low efficiency and poor accuracy of 3D point cloud instance segmentation in the prior art. It provides a method, system and lidar for automatically constructing point cloud instance segmentation samples that can automatically generate point cloud instance annotations, eliminate manual point-by-point annotation, and automatically produce learning samples suitable for urban scene data training. This significantly reduces model development costs and improves the accuracy and class discrimination ability of point cloud instance segmentation.
[0005] The present invention solves the above-mentioned technical problems through the following technical solution:
[0006] An automatic point cloud instance segmentation sample construction method, characterized in that the automatic point cloud instance segmentation sample construction method includes:
[0007] The point cloud data of the target area and the panoramic image data collected synchronously with the point cloud data are acquired. The point cloud data and the panoramic image data have a spatial mapping relationship.
[0008] The panoramic image data is spherically unfolded, and the unfolded panoramic image is divided into multiple perspective sub-views;
[0009] For each perspective subview, target identification and segmentation are performed to obtain segmentation results with category labels;
[0010] All perspective subviews are fused together using spherical back projection to form a panoramic instance segmentation result image, which includes the segmentation result.
[0011] The panoramic instance segmentation result image is projected onto the point cloud data using the spatial mapping relationship to obtain the point cloud segmentation result;
[0012] The point cloud segmentation results are used to generate training samples for a point cloud recognition model, which are then used for target recognition and segmentation of point cloud data.
[0013] Preferably, the automatic point cloud instance segmentation sample construction method is used for target recognition and segmentation of point cloud data in urban scenes, wherein the automatic point cloud instance segmentation sample construction method includes:
[0014] External parameter calibration of LiDAR and panoramic camera is performed to establish a spatial mapping relationship between point cloud coordinate system and panoramic image pixel coordinate system, so that each three-dimensional point cloud in the point cloud can be mapped to the corresponding pixel position in the panoramic image.
[0015] Point cloud data of the target area is acquired using a lidar system, and panoramic image data is acquired synchronously with the point cloud data using a panoramic camera.
[0016] Preferably, the method for automatically constructing point cloud instance segmentation samples includes:
[0017] Perform spherical unwrapping and distortion correction on panoramic image data;
[0018] An instance segmentation neural network is used to identify and segment objects in each perspective subview, obtaining segmentation results with category labels and pixel-level instance mask information.
[0019] Preferably, the method for automatically constructing point cloud instance segmentation samples includes:
[0020] The instance segmentation results of all perspective subviews are fused into a unified panoramic instance segmentation result map through spherical back projection.
[0021] Confidence-weighted fusion is performed on the overlapping regions to obtain a complete panoramic instance segmentation mask;
[0022] The panoramic instance segmentation mask is back-projected onto the point cloud data using the aforementioned spatial mapping relationship;
[0023] Assign semantic category labels and instance numbers to point clouds to obtain 3D point cloud data with instance annotations.
[0024] Preferably, the method for automatically constructing point cloud instance segmentation samples includes:
[0025] The point cloud instance segmentation sample automatic construction method described above is used to obtain three-dimensional point cloud data with instance annotations for several target regions;
[0026] Construct a point cloud dataset using all 3D point cloud data with instance annotations;
[0027] The point cloud dataset is standardized and data augmented to generate the training samples.
[0028] Preferably, the method for automatically constructing point cloud instance segmentation samples includes:
[0029] Acquire point cloud data for a target;
[0030] The point cloud recognition model is used to identify and segment the target point cloud data to obtain the instance segmentation prediction results of the target point cloud data;
[0031] The instance segmentation prediction results are projected onto the panoramic image space using the spatial mapping relationship;
[0032] The point cloud recognition model is evaluated using panoramic image data projected from the target point cloud data.
[0033] Preferably, the method for automatically constructing point cloud instance segmentation samples includes:
[0034] The instance segmentation prediction results are projected onto the panoramic image space using the spatial mapping relationship to generate a predicted instance mask map;
[0035] The predicted instance mask image and the recognition results of the panoramic image instance segmentation network are matched for category consistency and the region overlap is calculated to obtain the evaluation results of the point cloud recognition model.
[0036] Preferably, the method for automatically constructing point cloud instance segmentation samples includes:
[0037] The point cloud recognition model is evaluated using panoramic image data projected from the target point cloud data.
[0038] The training parameters or training sample selection strategy of the point cloud instance segmentation model are adaptively adjusted based on the evaluation metrics (the point cloud recognition model is corrected using the evaluation results of the point cloud recognition model).
[0039] The present invention also provides an automatic point cloud instance segmentation sample construction system, characterized in that the automatic point cloud instance segmentation sample construction system is used in the point cloud instance segmentation sample automatic construction method described above.
[0040] The present invention also provides a lidar, characterized in that the lidar is used in the point cloud instance segmentation sample automatic construction system as described above, and the lidar includes the point cloud recognition model.
[0041] Based on common knowledge in the field, the above-mentioned preferred conditions can be combined arbitrarily to obtain various preferred embodiments of the present invention.
[0042] The positive and progressive effects of this invention are as follows:
[0043] It can automatically generate image-driven point cloud instance annotations, eliminating the need for manual point-by-point annotation; it establishes a fully automated production line for 3D point cloud deep learning samples, supporting training on massive urban scene data; it forms an automated closed-loop system of training-prediction-backprojection evaluation, significantly reducing model development costs; it fully utilizes high-resolution semantic information of images to improve point cloud instance segmentation accuracy and category discrimination ability; it has strong engineering scalability and is suitable for various point cloud instance segmentation network architectures. Attached Figure Description
[0044] Figure 1 This is a flowchart of the automatic construction method for point cloud instance segmentation samples in Embodiment 1 of the present invention.
[0045] Figure 2 This is a schematic diagram illustrating the effect of the automatic construction method for point cloud instance segmentation samples in Embodiment 1 of the present invention. Detailed Implementation
[0046] The present invention will be further illustrated by way of embodiments below, but the present invention is not limited to the scope of the embodiments described herein.
[0047] Example 1
[0048] In the specification and claims of this application, the terms “comprising” and “having”, and any variations thereof, are used to indicate a non-exclusive inclusion, for example, a process, method, product, or apparatus that includes a series of method steps or modules is not necessarily limited to those steps or modules that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to such process, method, product, or apparatus.
[0049] See Figure 1 , Figure 2 This embodiment provides an automatic point cloud instance segmentation sample construction system, namely an automatic training sample construction system. The automatic point cloud instance segmentation sample construction system includes a lidar, a panoramic camera (fisheye camera), and a computing terminal.
[0050] The computing terminal can be a PC or a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.
[0051] The computing terminal can also be a computing module integrated into the lidar.
[0052] LiDAR is used to acquire point cloud data of a target area.
[0053] The panoramic camera is used to acquire panoramic image data that is synchronously collected with the point cloud data, and the point cloud data and panoramic image data have a spatial mapping relationship.
[0054] The computing terminal is used for:
[0055] The panoramic image data is spherically unfolded, and the unfolded panoramic image is divided into multiple perspective sub-views;
[0056] For each perspective subview, target identification and segmentation are performed to obtain segmentation results with category labels;
[0057] All perspective subviews are fused together using spherical back projection to form a panoramic instance segmentation result image, which includes the segmentation result.
[0058] The panoramic instance segmentation result image is projected onto the point cloud data using the spatial mapping relationship to obtain the point cloud segmentation result;
[0059] The point cloud segmentation results are used to generate training samples for a point cloud recognition model, which are then used for target recognition and segmentation of point cloud data.
[0060] This embodiment is applicable to target recognition and segmentation of point cloud data in urban scenes, with LiDAR and panoramic camera calibrated using extrinsic parameters.
[0061] The computing terminal is used to: establish a spatial mapping relationship between the point cloud coordinate system and the panoramic image pixel coordinate system, so that each three-dimensional point cloud in the point cloud can be mapped to the corresponding pixel position in the panoramic image.
[0062] The computing terminal is also used for:
[0063] Perform spherical unwrapping and distortion correction on panoramic image data;
[0064] An instance segmentation neural network is used to identify and segment objects in each perspective subview, obtaining segmentation results with category labels and pixel-level instance mask information.
[0065] The computing terminal is also used for:
[0066] The instance segmentation results of all perspective subviews are fused into a unified panoramic instance segmentation result map through spherical back projection.
[0067] Confidence-weighted fusion is performed on the overlapping regions to obtain a complete panoramic instance segmentation mask;
[0068] The panoramic instance segmentation mask is back-projected onto the point cloud data using the aforementioned spatial mapping relationship;
[0069] Assign semantic category labels and instance numbers to point clouds to obtain 3D point cloud data with instance annotations.
[0070] The computing terminal is also used for:
[0071] The point cloud instance segmentation sample automatic construction method described above is used to obtain three-dimensional point cloud data with instance annotations for several target regions;
[0072] Construct a point cloud dataset using all 3D point cloud data with instance annotations;
[0073] The point cloud dataset is standardized and data augmented to generate the training samples.
[0074] The computing terminal is also used for:
[0075] Acquire point cloud data for a target;
[0076] The point cloud recognition model is used to identify and segment the target point cloud data to obtain the instance segmentation prediction results of the target point cloud data;
[0077] The instance segmentation prediction results are projected onto the panoramic image space using the spatial mapping relationship;
[0078] The point cloud recognition model is evaluated using panoramic image data projected from the target point cloud data.
[0079] The computing terminal is also used for:
[0080] The instance segmentation prediction results are projected onto the panoramic image space using the spatial mapping relationship to generate a predicted instance mask map;
[0081] The predicted instance mask image and the recognition results of the panoramic image instance segmentation network are matched for category consistency and the region overlap is calculated to obtain the evaluation results of the point cloud recognition model.
[0082] The computing terminal is also used for:
[0083] The point cloud recognition model is evaluated using panoramic image data projected from the target point cloud data.
[0084] The training parameters or training sample selection strategy of the point cloud instance segmentation model are adaptively adjusted based on the evaluation metrics (the point cloud recognition model is corrected using the evaluation results of the point cloud recognition model).
[0085] Specifically, the workflow of the training sample construction system in this application is as follows:
[0086] Acquire panoramic image data that is synchronously collected with point cloud data, and establish a spatial mapping relationship between the point cloud spatial coordinate system and the panoramic image pixel coordinate system based on the extrinsic parameter calibration parameters between the lidar and the panoramic camera.
[0087] The panoramic image is subjected to spherical unfolding and distortion correction processing, the panoramic image is divided into multiple perspective sub-views, and each sub-view is subjected to instance segmentation processing to obtain the category label and pixel-level instance mask information of each target in the image (such as trees, streets, walls, streetlights, trash cans, etc.).
[0088] The instance segmentation results of each subview are fused into a unified panoramic instance segmentation result map through spherical back projection, and a confidence weighting strategy is used to optimize the fusion of overlapping areas.
[0089] Based on the mapping relationship between pixel coordinates and point cloud spatial coordinates, the instance mask information in the panoramic image is back-projected to the point cloud space, and each three-dimensional point in the point cloud is automatically assigned a corresponding semantic category label and instance number to generate three-dimensional point cloud data with instance annotations.
[0090] The above automatic point cloud annotation process is repeated for multiple different scenes and multiple target regions to construct a large-scale 3D point cloud instance segmentation training sample set.
[0091] The sample set is standardized, including coordinate normalization, point cloud block cropping, density equalization sampling, and instance class equalization enhancement, to form a point cloud instance segmentation dataset suitable for deep learning training.
[0092] A deep learning model for 3D point cloud instance segmentation (point cloud recognition model) is constructed. The model includes a point feature encoding network and an instance clustering decoding network. The point feature encoding network is used to extract point-level geometric and semantic features, and the instance clustering decoding network is used to learn the instance affiliation relationship between points.
[0093] The deep learning model is trained using the automatically generated point cloud instance segmentation sample set. By jointly optimizing the semantic classification loss, instance embedding vector similarity loss, and instance boundary consistency loss, the model's ability to segment multi-target instances in complex scenes is improved.
[0094] The trained deep learning model is applied to test point cloud data (such as 3D scan data of a new scene) to obtain point cloud instance segmentation prediction results;
[0095] Based on the spatial mapping relationship between point cloud and panoramic image, the predicted point cloud instance segmentation prediction result is back-projected into the panoramic image space to generate a predicted instance mask map.
[0096] The predicted instance mask image is matched with the recognition result obtained based on the image instance segmentation network for category consistency matching and region overlap evaluation. Evaluation indicators such as instance-level intersection-union ratio, category recognition consistency rate and false detection rate are automatically calculated.
[0097] The training parameters or sample selection strategy of the point cloud deep learning model are adaptively adjusted according to the evaluation indicators to achieve automated closed-loop optimization of point cloud instance segmentation model training and testing.
[0098] The system for automatically constructing segmented samples using the above point cloud instances is described in [reference]. Figure 1 This embodiment provides a method for automatically constructing point cloud instance segmentation samples, including:
[0099] Step 100: Perform extrinsic parameter calibration on the LiDAR and panoramic camera, and establish a spatial mapping relationship between the point cloud coordinate system and the panoramic image pixel coordinate system, so that each 3D point cloud in the point cloud can be mapped to the corresponding pixel position in the panoramic image.
[0100] Step 101: Use a lidar to acquire point cloud data of the target area, and use a panoramic camera to acquire panoramic image data that is synchronously acquired with the point cloud data. The point cloud data and the panoramic image data have a spatial mapping relationship.
[0101] Step 102: Perform spherical unfolding and distortion correction on the panoramic image data, and divide the unfolded panoramic image into multiple perspective sub-views;
[0102] Step 103: Use an instance segmentation neural network to perform target recognition and segmentation on each perspective subview to obtain segmentation results with category labels and pixel-level instance mask information;
[0103] Step 104: Fuse all perspective subviews into a panoramic instance segmentation result image by spherical back projection, wherein the panoramic instance segmentation result image includes the segmentation result;
[0104] Step 105: Project the panoramic instance segmentation result image onto the point cloud data using the spatial mapping relationship to obtain the point cloud segmentation result;
[0105] Step 106: Use the point cloud segmentation results to generate training samples for the point cloud recognition model. The training samples are used for target recognition and segmentation of point cloud data.
[0106] Step 104 includes:
[0107] The instance segmentation results of all perspective subviews are fused into a unified panoramic instance segmentation result map through spherical back projection.
[0108] The overlapping regions are subjected to confidence-weighted fusion to obtain a complete panoramic instance segmentation mask.
[0109] Step 105 includes:
[0110] The panoramic instance segmentation mask is back-projected onto the point cloud data using the aforementioned spatial mapping relationship;
[0111] Assign semantic category labels and instance numbers to point clouds to obtain 3D point cloud data with instance annotations.
[0112] Step 106 includes:
[0113] The point cloud instance segmentation sample automatic construction method described above is used to obtain three-dimensional point cloud data with instance annotations for several target regions;
[0114] Construct a point cloud dataset using all 3D point cloud data with instance annotations;
[0115] The point cloud dataset is standardized and data augmented to generate the training samples.
[0116] Furthermore, the method for automatically constructing point cloud instance segmentation samples also includes:
[0117] Step 107: Obtain point cloud data for a target;
[0118] Step 108: Use the point cloud recognition model to identify and segment the target point cloud data to obtain the instance segmentation prediction results of the target point cloud data;
[0119] Step 109: Project the instance segmentation prediction results onto the panoramic image space using the spatial mapping relationship;
[0120] Step 110: Evaluate the point cloud recognition model using panoramic image data projected from the target point cloud data.
[0121] Step 109 includes: projecting the instance segmentation prediction result onto the panoramic image space using the spatial mapping relationship and generating a prediction instance mask map;
[0122] Step 110 includes: performing category consistency matching and region overlap calculation on the predicted instance mask map and the recognition results of the panoramic image instance segmentation network to obtain the evaluation results of the point cloud recognition model.
[0123] Step 111: Adaptively adjust the training parameters or training sample selection strategy of the point cloud instance segmentation model according to the evaluation index (correct the point cloud recognition model using the evaluation results of the point cloud recognition model).
[0124] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.
Claims
1. A method for automatically constructing point cloud instance segmentation samples, characterized in that, The method for automatically constructing point cloud instance segmentation samples includes: The point cloud data of the target area and the panoramic image data collected synchronously with the point cloud data are acquired. The point cloud data and the panoramic image data have a spatial mapping relationship. The panoramic image data is spherically unfolded, and the unfolded panoramic image is divided into multiple perspective sub-views; For each perspective subview, target identification and segmentation are performed to obtain segmentation results with category labels; All perspective subviews are fused together using spherical back projection to form a panoramic instance segmentation result image, which includes the segmentation result. The panoramic instance segmentation result image is projected onto the point cloud data using the spatial mapping relationship to obtain the point cloud segmentation result; The point cloud segmentation results are used to generate training samples for a point cloud recognition model, which are then used for target recognition and segmentation of point cloud data.
2. The method for automatically constructing point cloud instance segmentation samples as described in claim 1, characterized in that, An automatic point cloud instance segmentation sample construction method is used for target recognition and segmentation of point cloud data in urban scenes. The automatic point cloud instance segmentation sample construction method includes: External parameter calibration of LiDAR and panoramic camera is performed to establish a spatial mapping relationship between point cloud coordinate system and panoramic image pixel coordinate system, so that each three-dimensional point cloud in the point cloud can be mapped to the corresponding pixel position in the panoramic image. Point cloud data of the target area is acquired using LiDAR, and panoramic image data is acquired synchronously with the point cloud data using a panoramic camera.
3. The method for automatically constructing point cloud instance segmentation samples as described in claim 1, characterized in that, The method for automatically constructing point cloud instance segmentation samples includes: Perform spherical unwrapping and distortion correction on panoramic image data; An instance segmentation neural network is used to identify and segment objects in each perspective subview, obtaining segmentation results with category labels and pixel-level instance mask information.
4. The method for automatically constructing point cloud instance segmentation samples as described in claim 3, characterized in that, The method for automatically constructing point cloud instance segmentation samples includes: The instance segmentation results of all perspective subviews are fused into a unified panoramic instance segmentation result map through spherical back projection. Confidence-weighted fusion is performed on the overlapping regions to obtain a complete panoramic instance segmentation mask; The panoramic instance segmentation mask is back-projected onto the point cloud data using the aforementioned spatial mapping relationship; Assign semantic category labels and instance numbers to point clouds to obtain 3D point cloud data with instance annotations.
5. The method for automatically constructing point cloud instance segmentation samples as described in claim 4, characterized in that, The method for automatically constructing point cloud instance segmentation samples includes: The point cloud instance segmentation sample automatic construction method described above is used to obtain three-dimensional point cloud data with instance annotations for several target regions; Construct a point cloud dataset using all 3D point cloud data with instance annotations; The point cloud dataset is standardized and data augmented to generate the training samples.
6. The method for automatically constructing point cloud instance segmentation samples as described in claim 1, characterized in that, The method for automatically constructing point cloud instance segmentation samples includes: Acquire point cloud data for a target; The point cloud recognition model is used to identify and segment the target point cloud data to obtain the instance segmentation prediction results of the target point cloud data; The instance segmentation prediction results are projected onto the panoramic image space using the spatial mapping relationship; The point cloud recognition model is evaluated using panoramic image data projected from the target point cloud data.
7. The method for automatically constructing point cloud instance segmentation samples as described in claim 6, characterized in that, The method for automatically constructing point cloud instance segmentation samples includes: The instance segmentation prediction results are projected onto the panoramic image space using the spatial mapping relationship to generate a predicted instance mask map; The predicted instance mask image and the recognition results of the panoramic image instance segmentation network are matched for category consistency and the region overlap is calculated to obtain the evaluation results of the point cloud recognition model.
8. The method for automatically constructing point cloud instance segmentation samples as described in claim 6, characterized in that, The method for automatically constructing point cloud instance segmentation samples includes: The point cloud recognition model is evaluated using panoramic image data projected from the target point cloud data. The training parameters or training sample selection strategy of the point cloud recognition model are adaptively adjusted based on the evaluation metrics.
9. An automatic point cloud instance segmentation sample construction system, characterized in that, The point cloud instance segmentation sample automatic construction system is used in the point cloud instance segmentation sample automatic construction method as described in any one of claims 1 to 8.
10. A lidar, characterized in that, The lidar is used in the point cloud instance segmentation sample automatic construction system as described in claim 9, wherein the lidar includes the point cloud recognition model.