A 3D point cloud annotation method and system based on scene reconstruction

Abstract

The present invention relates to the technical field of point cloud data annotation, and in particular to a method and system for three-dimensional point cloud annotation based on scene reconstruction. The method comprises: a user terminal inputs pre-collected road scene image data into a three-dimensional scene reconstruction platform, processes the road scene image data, and obtains a three-dimensional reconstructed scene image of the road scene; inputs the point cloud data into a point cloud semantic segmentation platform to obtain a semantic segmentation pre-annotation result; inputs the point cloud data into a 3D bounding box annotation platform to obtain a 3D bounding box pre-annotation result; the user terminal manually corrects the pre-annotation result based on the three-dimensional reconstructed scene image to obtain a semantic segmentation annotation result and a 3D bounding box annotation result; and intersects the semantic segmentation annotation result with the 3D bounding box annotation result to obtain an instance label. The present invention can save manual annotation time and improve annotation quality.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud data annotation, and in particular to a three-dimensional point cloud annotation method and system based on scene reconstruction. Background Art

[0002] Deep learning has experienced rapid development in recent years, and its widespread application in fields such as computer vision and natural language processing has achieved tremendous success. However, the associated labeling task remains a critical and complex one, directly impacting model performance and application success. Creating labeled data remains a challenging task, requiring solutions to address time-consuming and labor-intensive issues, inconsistent quality, and a shortage of professional labelers to improve the efficiency and effectiveness of deep learning applications.

[0003] Currently, creating large-scale annotated datasets requires significant human and financial resources. The cost of annotating data can be a limiting factor for many research and applications. Furthermore, the annotation process involves subjective judgment, which can lead to inconsistencies between annotators, resulting in varying label quality. Furthermore, due to the complexity of the scene, the collected point cloud data may contain incomplete object shapes due to occlusions, or object categories that are difficult to discern with the naked eye due to angular angles. Summary of the Invention

[0004] In order to solve the technical problems of the existing technology, such as the cumbersome process of creating labels, inconsistent label quality, and low labeling efficiency, the embodiment of the present invention provides a 3D point cloud labeling method and system based on scene reconstruction. The technical solution is as follows:

[0005] In one aspect, a method for annotating a 3D point cloud based on scene reconstruction is provided. The method is implemented by a 3D point cloud annotating device based on scene reconstruction, and the method comprises:

[0006] S1. A user terminal inputs pre-collected road scene image data into the 3D scene reconstruction platform, and the 3D scene reconstruction platform processes the road scene image data to obtain a 3D reconstructed scene image of the road scene;

[0007] S2. The user terminal divides the road scene point cloud data obtained in the same scene into a first batch of point cloud data, a second batch of point cloud data, and a third batch of point cloud data;

[0008] S3. Input the first batch of point cloud data into the semantic segmentation and annotation platform, perform manual annotation directly, and obtain the annotation results of the first batch of data. The annotation results of the first batch of data are used to train the point cloud semantic segmentation model integrated in the training platform. After updating the model, input the second batch of point cloud data into the semantic segmentation and annotation platform, turn on the pre-annotation function in the platform, and pass the data through the point cloud semantic segmentation model to obtain the point cloud semantic segmentation pre-annotation results; the user end manually corrects the point cloud semantic segmentation pre-annotation results according to the 3D reconstructed scene image to obtain the final semantic segmentation annotation results of the second batch of data; input the final semantic segmentation annotation results of the second batch of data into the point cloud semantic segmentation model, and after training the model, obtain the updated semantic segmentation model; the user end inputs the third batch of point cloud data into the semantic segmentation and annotation platform, and passes through the updated semantic segmentation model to obtain the pre-annotation results of the third batch of data. The user end manually corrects the pre-annotation results of the third batch of data according to the 3D reconstructed scene image to obtain the final semantic segmentation annotation results of the third batch of data;

[0009] S4. Input the first batch of point cloud data into the 3D bounding box annotation platform and perform manual annotation directly to obtain the annotation results of the first batch of data. The annotation results of the first batch of data are used to train the 3D target detection model integrated in the training platform. After updating the model, input the second batch of point cloud data into the 3D bounding box annotation platform, turn on the pre-annotation function in the platform, and pass the data through the 3D target detection model to obtain the 3D bounding box pre-annotation results. The user end manually corrects the 3D bounding box pre-annotation results based on the 3D reconstructed scene image to obtain the final 3D bounding box annotation results of the second batch of data. Input the final 3D bounding box annotation results of the second batch of data into the 3D target detection model, and after training the model, obtain the updated 3D target detection model. The user end inputs the third batch of point cloud data into the 3D bounding box annotation platform, passes through the updated 3D target detection model to obtain the pre-annotation results of the third batch of data. The user end manually corrects the pre-annotation results of the third batch of data based on the 3D reconstructed scene image to obtain the final 3D bounding box annotation results of the third batch of data.

[0010] S5. Combine the final semantic segmentation annotation result with the final 3D bounding box annotation result, take the overlapping point cloud of the two regions as an instance, and generate an instance label.

[0011] Optionally, the user terminal of S1 inputs pre-collected road scene image data into the 3D scene reconstruction platform, and the 3D scene reconstruction platform processes the road scene image data to obtain a 3D reconstructed scene image of the road scene, including:

[0012] S11. The user terminal inputs the road scene image data into the 3D scene reconstruction platform. The 3D scene reconstruction platform extracts features from the input road scene image data to obtain feature points of the road scene image data; and performs feature matching based on the feature points to determine the correspondence between different images.

[0013] S12, performing sparse 3D reconstruction based on the input road scene image data to obtain sparsely reconstructed scene sparse 3D points, camera intrinsic parameters, and camera pose data;

[0014] S13, estimating a depth map of the road scene based on the sparsely reconstructed scene sparse 3D points, camera intrinsic parameters, and camera pose data to obtain a depth map and a normal vector map;

[0015] S14, performing dense reconstruction on the road scene according to the input road scene image data, the correspondence between different images, the camera intrinsic parameters, the depth map, and the normal vector map to obtain a dense reconstruction result;

[0016] S15. Visualize the dense reconstruction results to obtain dense three-dimensional point cloud data of the object surfaces in the road scene; visualize the dense three-dimensional point cloud data of the object surfaces in the road scene to obtain a three-dimensional reconstructed scene image of the road scene.

[0017] Optionally, the semantic segmentation and annotation platform is used to visualize three-dimensional point cloud data and model prediction results; the semantic segmentation and annotation platform integrates a pre-built semantic segmentation model.

[0018] Optionally, the 3D bounding box annotation platform is used to annotate bounding boxes on three-dimensional point cloud data, and the user terminal selects, modifies and adds labels to the three-dimensional point cloud data through the interactive tools provided by the 3D bounding box annotation platform; the 3D bounding box annotation platform integrates a pre-built 3D target detection model.

[0019] Optionally, the manual correction includes correcting missing, redundant and erroneous parts of the pre-labeling results.

[0020] On the other hand, a 3D point cloud annotation system based on scene reconstruction is provided, which is applied to a 3D point cloud annotation method based on scene reconstruction, and includes:

[0021] The user terminal divides the road scene point cloud data acquired in the same scene into a first batch of point cloud data, a second batch of point cloud data and a third batch of point cloud data;

[0022] The semantic segmentation and annotation platform is used to input the first batch of point cloud data into the semantic segmentation and annotation platform, directly perform manual annotation, and obtain the annotation results of the first batch of data; input the second batch of point cloud data into the semantic segmentation and annotation platform, enable the pre-annotation function in the platform, and pass the data through the point cloud semantic segmentation model to obtain the point cloud semantic segmentation pre-annotation results; the user end inputs the third batch of point cloud data into the semantic segmentation and annotation platform, and passes it through the updated semantic segmentation model to obtain the pre-annotation results of the third batch of data;

[0023] The 3D bounding box annotation platform is used to input the first batch of point cloud data into the 3D bounding box annotation platform and directly perform manual annotation to obtain the annotation results of the first batch of data; input the second batch of point cloud data into the 3D bounding box annotation platform, enable the pre-annotation function in the platform, and pass the data through the 3D object detection model to obtain the 3D bounding box pre-annotation results; the user end inputs the third batch of point cloud data into the 3D bounding box annotation platform and passes it through the updated 3D object detection model to obtain the pre-annotation results of the third batch of data;

[0024] The three-dimensional scene reconstruction platform is used to process road scene image data to obtain a three-dimensional reconstruction model of the road scene.

[0025] Optionally, the user terminal inputs pre-collected road scene image data into the three-dimensional scene reconstruction platform, and the three-dimensional scene reconstruction platform processes the road scene image data to obtain a three-dimensional reconstructed scene image of the road scene, including:

[0026] The user terminal inputs the road scene image data into the 3D scene reconstruction platform, which extracts features from the input road scene image data to obtain feature points of the road scene image data; performs feature matching based on the feature points to determine the correspondence between different images;

[0027] Perform sparse 3D reconstruction based on the input road scene image data to obtain sparsely reconstructed scene sparse 3D points, camera intrinsic parameters, and camera pose data;

[0028] Estimate the depth map of the road scene based on the sparsely reconstructed sparse 3D points of the scene, the camera intrinsic parameters, and the camera pose data to obtain the depth map and normal vector map;

[0029] Based on the input road scene image data, the correspondence between different images, the camera internal parameters, the depth map and the normal vector map, the road scene is densely reconstructed to obtain a dense reconstruction result;

[0030] The dense reconstruction results are visualized to obtain dense three-dimensional point cloud data of the object surface in the road scene; the dense three-dimensional point cloud data of the object surface in the road scene is visualized to obtain a three-dimensional reconstructed scene image of the road scene.

[0031] Optionally, the semantic segmentation and annotation platform is used to visualize three-dimensional point cloud data and model prediction results; the semantic segmentation and annotation platform integrates a pre-built semantic segmentation model.

[0032] Optionally, the 3D bounding box annotation platform is used to annotate bounding boxes on three-dimensional point cloud data, and the user terminal selects, modifies and adds labels to the three-dimensional point cloud data through the interactive tools provided by the 3D bounding box annotation platform; the 3D bounding box annotation platform integrates a pre-built 3D target detection model.

[0033] Optionally, the manual correction includes correcting missing, redundant and erroneous parts of the pre-labeling results.

[0034] On the other hand, a three-dimensional point cloud annotation device based on scene reconstruction is provided, and the three-dimensional point cloud annotation device based on scene reconstruction includes: a processor; a memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, any one of the three-dimensional point cloud annotation methods based on scene reconstruction is implemented.

[0035] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned three-dimensional point cloud annotation methods based on scene reconstruction.

[0036] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0037] In the embodiment of the present invention, the user terminal first inputs the pre-collected road scene image data into the 3D scene reconstruction platform, and the 3D scene reconstruction platform processes the road scene image data to obtain a 3D reconstructed scene image of the road scene; the user terminal divides the road scene point cloud data acquired under the same scene into the first batch of point cloud data, the second batch of point cloud data and the third batch of point cloud data; the first batch of point cloud data is input into the semantic segmentation and annotation platform, and manual annotation is directly performed to obtain the annotation results of the first batch of data, and the annotation results of the first batch of data are used to train the point cloud semantic segmentation model integrated in the platform. After the model is updated, the second batch of point cloud data is input into the semantic segmentation and annotation platform. The semantic segmentation and annotation platform is used to turn on the pre-annotation function in the platform. The data passes through the point cloud semantic segmentation model to obtain the point cloud semantic segmentation pre-annotation results; the user end manually corrects the point cloud semantic segmentation pre-annotation results based on the 3D reconstructed scene image to obtain the final semantic segmentation and annotation results of the second batch of data; the final semantic segmentation and annotation results of the second batch of data are input into the point cloud semantic segmentation model, and after training the model, an updated semantic segmentation model is obtained; the user end inputs the third batch of point cloud data into the semantic segmentation and annotation platform, and obtains the pre-annotation results of the third batch of data through the updated semantic segmentation model. The user end corrects the third batch of data based on the 3D reconstructed scene image. The pre-labeling results of the first batch of point cloud data are manually corrected to obtain the final semantic segmentation labeling results of the third batch of data; the first batch of point cloud data is input into the 3D bounding box labeling platform and manually labeled directly to obtain the labeling results of the first batch of data. The labeling results of the first batch of data are used to train the 3D target detection model integrated in the platform. After updating the model, the second batch of point cloud data is input into the 3D bounding box labeling platform, the pre-labeling function in the platform is turned on, and the data passes through the 3D target detection model to obtain the 3D bounding box pre-labeling results; the user end manually corrects the 3D bounding box pre-labeling results according to the 3D reconstructed scene image to obtain the final 3D bounding box of the second batch of data. The final 3D bounding box annotation results of the second batch of data are input into the 3D target detection model. After training the model, an updated 3D target detection model is obtained. The user end inputs the third batch of point cloud data into the 3D bounding box annotation platform. After the updated 3D target detection model, the pre-annotation results of the third batch of data are obtained. The user end manually corrects the pre-annotation results of the third batch of data based on the 3D reconstructed scene image to obtain the final 3D bounding box annotation results of the third batch of data. The final semantic segmentation annotation results are combined with the final 3D bounding box annotation results, and the point cloud of the overlapping part of the two areas is taken as an instance to generate instance labels.

[0038] The embodiment of the present invention uses three-dimensional modeling for auxiliary labeling, which avoids the situation where the object shape is incomplete due to occlusion or the object category is difficult to distinguish with the naked eye due to biased angles, thereby improving the accuracy and efficiency of segmentation labeling. The embodiment of the present invention combines the final semantic segmentation labeling result with the overlapping part of the final 3D bounding box labeling result to generate instance labels; wherein, the instance label can automatically distinguish each object in the point cloud, saving the time and workload of manually creating instance labels; the instance label can provide key information for the segmentation algorithm, improving the recognition and segmentation accuracy of object instances; compared with the existing technology, the embodiment of the present invention saves manual labeling time, can quickly obtain the approximate distribution of objects in a frame of point cloud data, reduces the time for labelers to perform object category and shape analysis, and can improve the quality of labeling through a combination of model prediction and manual correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 This is a schematic diagram of the process structure of a three-dimensional point cloud annotation based on scene reconstruction provided by an embodiment of the present invention;

[0041] Figure 2 This is a flow chart of a three-dimensional point cloud annotation method based on scene reconstruction provided by an embodiment of the present invention;

[0042] Figure 3 This is a block diagram of a 3D point cloud annotation system based on scene reconstruction provided by an embodiment of the present invention;

[0043] Figure 4 It is a structural schematic diagram of a three-dimensional point cloud annotation device based on scene reconstruction provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0045] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0046] In the embodiments of the present invention, the terms "image" and "picture" may be used interchangeably. It should be noted that, when the distinction between them is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may be used interchangeably. It should be noted that, when the distinction between them is not emphasized, the meanings they convey are the same.

[0047] In the embodiments of the present invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0048] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0049] The embodiment of the present invention provides a three-dimensional point cloud annotation method based on scene reconstruction, which can be implemented by a three-dimensional point cloud annotation device based on scene reconstruction, and the three-dimensional point cloud annotation device based on scene reconstruction can be a terminal or a server. Figure 13D point cloud annotation is a process structure diagram of a scene reconstruction-based 3D point cloud annotation provided by an embodiment of the present invention; wherein the annotation platform includes a semantic segmentation annotation platform and a 3D bounding box annotation platform; a feasible implementation method is to input the collected scene image into the 3D scene reconstruction platform to obtain the 3D reconstruction result; input the first batch of point cloud data into the semantic segmentation annotation platform, directly perform manual annotation to obtain the annotation results of the first batch of data, use the annotation results of the first batch of data to train the point cloud semantic segmentation model integrated in the training platform, and after updating the model, input the second batch of point cloud data into the semantic segmentation annotation platform to start the pre-labeling in the platform. The annotation function is used. The data passes through the point cloud semantic segmentation model to obtain the point cloud semantic segmentation pre-labeling results; the user end manually corrects the point cloud semantic segmentation pre-labeling results according to the 3D reconstructed scene image to obtain the final semantic segmentation labeling results of the second batch of data; the final semantic segmentation labeling results of the second batch of data are input into the point cloud semantic segmentation model, and after training the model, an updated semantic segmentation model is obtained; the user end inputs the third batch of point cloud data into the semantic segmentation labeling platform, and obtains the pre-labeling results of the third batch of data after the updated semantic segmentation model. The user end manually corrects the pre-labeling results of the third batch of data according to the 3D reconstructed scene image. The final semantic segmentation annotation results of the third batch of data are obtained; the first batch of point cloud data is input into the 3D bounding box annotation platform, and manual annotation is performed directly to obtain the annotation results of the first batch of data. The annotation results of the first batch of data are used to train the 3D target detection model integrated in the platform. After updating the model, the second batch of point cloud data is input into the 3D bounding box annotation platform, and the pre-annotation function in the platform is turned on. The data passes through the 3D target detection model to obtain the 3D bounding box pre-annotation results; the user end manually corrects the 3D bounding box pre-annotation results based on the 3D reconstructed scene image to obtain the final 3D bounding box annotation results of the second batch of data. ; Input the final 3D bounding box annotation results of the second batch of data into the 3D target detection model, and after training the model, obtain the updated 3D target detection model; the user end inputs the third batch of point cloud data into the 3D bounding box annotation platform, and obtains the pre-annotation results of the third batch of data through the updated 3D target detection model. The user end manually corrects the pre-annotation results of the third batch of data based on the three-dimensional reconstructed scene image to obtain the final 3D bounding box annotation results of the third batch of data; combine the final semantic segmentation annotation results with the final 3D bounding box annotation results, take the overlapping point clouds of the two areas as an instance, and generate instance labels. The 3D bounding box labels are used for 3D detection algorithm training tasks; wherein, the 3D bounding box labels are projected to generate 2D bounding box labels for 2D detection algorithm training tasks. wherein, the 3D bounding box labels are combined with the overlapping parts of the semantic segmentation labels to obtain instance labels; wherein, the instance labels are used for 3D detection algorithm training tasks. As Figure 2The flowchart of the 3D point cloud annotation method based on scene reconstruction is shown. The processing flow of the method may include the following steps:

[0050] S1. The user terminal inputs the pre-collected road scene image data into the three-dimensional scene reconstruction platform. The three-dimensional scene reconstruction platform processes the road scene image data to obtain a three-dimensional reconstructed scene image of the road scene.

[0051] Among them, the road scene image data can be obtained by collecting it through a camera mounted on a movable vehicle.

[0052] Among them, the acquisition of multi-view images by a camera mounted on a movable cart can improve the accuracy and stability of reconstruction.

[0053] Among them, the 3D scene reconstruction platform integrates the graphical interface software colmap.

[0054] Optionally, the specific implementation steps of S1 include S11-S15:

[0055] S11. The user terminal inputs the road scene image data into the 3D scene reconstruction platform. The 3D scene reconstruction platform extracts features from the input road scene image data to obtain feature points of the road scene image data; and performs feature matching based on the feature points to determine the correspondence between different images.

[0056] S12, performing sparse 3D reconstruction based on the input road scene image data to obtain sparsely reconstructed scene sparse 3D points, camera intrinsic parameters, and camera pose data;

[0057] S13, estimating a depth map of the road scene based on the sparsely reconstructed scene sparse 3D points, camera intrinsic parameters, and camera pose data to obtain a depth map and a normal vector map;

[0058] S14, performing dense reconstruction on the road scene according to the input road scene image data, the correspondence between different images, the camera intrinsic parameters, the depth map, and the normal vector map to obtain a dense reconstruction result;

[0059] S15. Visualize the dense reconstruction results to obtain dense three-dimensional point cloud data of the object surfaces in the road scene; visualize the dense three-dimensional point cloud data of the object surfaces in the road scene to obtain a three-dimensional reconstructed scene image of the road scene.

[0060] S2. The user end divides the road scene point cloud data obtained in the same scene into a first batch of point cloud data, a second batch of point cloud data, and a third batch of point cloud data.

[0061] Among them, the road scene image data and the road scene point cloud data are both collected in the same scene.

[0062] Among them, road scene point cloud data can be obtained through HESAI XT32 lidar sensor collection.

[0063] A feasible implementation method is to use the HESAI XT32 lidar sensor to collect data at different locations in the same scene, obtaining continuous point cloud data frames at each location; and extracting some frames from the point cloud data sequence as the point cloud data to be processed.

[0064] S3. Input the first batch of point cloud data into the semantic segmentation and annotation platform, perform manual annotation directly, and obtain the annotation results of the first batch of data. Use the annotation results of the first batch of data to train the point cloud semantic segmentation model integrated in the platform. After updating the model, input the second batch of point cloud data into the semantic segmentation and annotation platform, turn on the pre-annotation function in the platform, and pass the data through the point cloud semantic segmentation model to obtain the point cloud semantic segmentation pre-annotation results; the user end manually corrects the point cloud semantic segmentation pre-annotation results based on the 3D reconstructed scene image to obtain the final semantic segmentation annotation results of the second batch of data; input the final semantic segmentation annotation results of the second batch of data into the point cloud semantic segmentation model, and after training the model, obtain the updated semantic segmentation model; the user end inputs the third batch of point cloud data into the semantic segmentation and annotation platform, and passes through the updated semantic segmentation model to obtain the pre-annotation results of the third batch of data. The user end manually corrects the pre-annotation results of the third batch of data based on the 3D reconstructed scene image to obtain the final semantic segmentation annotation results of the third batch of data.

[0065] A feasible implementation method is that during the manual correction process, when occlusion, abnormal shapes, and angles that are difficult to judge occur, the details of each object are confirmed through three-dimensional scene images. Using three-dimensional scene images to assist manual labeling can avoid situations where occlusion or angle deviation causes difficulty in distinguishing.

[0066] Among them, the user end can select, modify and add labels to point cloud data through the interactive tools provided by the semantic annotation platform, and correct errors in the model prediction results through the interactive tools.

[0067] Among them, the semantic segmentation model can be a 2dpass (2D PriorsAssisted Semantic Segmentation, semantic segmentation of lidar point cloud based on two-dimensional image prior assistance) model or a U-Net model, which is not limited in the present invention.

[0068] In a feasible implementation, the embodiment of the present invention selects the 3D network part of the 2dpass model as the 3D semantic segmentation model. In a feasible implementation, the 3D semantic segmentation model is trained using supervised learning. The training process includes:

[0069] (1) Input the point cloud data into the 3D semantic segmentation model, and the point cloud data is voxelized to obtain a point-to-voxel mapping. The voxel mapping is sparsely convolved to obtain a voxel-level feature map.

[0070] (2) Obtain the feature set of each point based on the nearest neighbor interpolation method and point voxel mapping relationship;

[0071] (3) Finally, the 3D segmentation prediction result is obtained based on the classifier;

[0072] Among them, the evaluation of 3D semantic segmentation model can use IoU to quantify the segmentation quality.

[0073] Optionally, a semantic segmentation annotation platform is used to visualize 3D point cloud data and model prediction results; the semantic segmentation annotation platform integrates a pre-built semantic segmentation model.

[0074] Among them, the preliminary prediction results of semantic segmentation are integrated into the data display interface of the semantic segmentation annotation platform, including: matching the label corresponding color information of the preliminary prediction results of semantic segmentation with the point cloud data, so that users can see the segmentation label of each point.

[0075] S4. Input the first batch of point cloud data into the 3D bounding box annotation platform and perform manual annotation directly to obtain the annotation results of the first batch of data. The annotation results of the first batch of data are used to train the 3D target detection model integrated in the training platform. After updating the model, input the second batch of point cloud data into the 3D bounding box annotation platform, turn on the pre-annotation function in the platform, and pass the data through the 3D target detection model to obtain the 3D bounding box pre-annotation results; the user end manually corrects the 3D bounding box pre-annotation results based on the 3D reconstructed scene image to obtain the final 3D bounding box annotation results of the second batch of data; input the final 3D bounding box annotation results of the second batch of data into the 3D target detection model, and after training the model, obtain the updated 3D target detection model; the user end inputs the third batch of point cloud data into the 3D bounding box annotation platform, passes through the updated 3D target detection model, and obtains the pre-annotation results of the third batch of data. The user end manually corrects the pre-annotation results of the third batch of data based on the 3D reconstructed scene image to obtain the final 3D bounding box annotation results of the third batch of data.

[0076] Among them, the 3D target detection model can be a PV-RCNN (Poi ntVoxel-RCNN, a high-performance 3D object detection framework) model or a Voxel Net model, which is not limited in the present invention.

[0077] The 3D object detection model used in the embodiment of the present invention is the PV-RCNN model. In one feasible implementation, the training process of the PV-RCNN model uses supervised learning, where the weights of the PV-RCNN model can be optimized by minimizing the loss function. The specific training process includes:

[0078] (1) Input the point cloud data into the PV-RCNN model, and perform feature extraction on the point cloud data using the 3D convolution layer to extract preliminary feature representations; compress the extracted features onto a two-dimensional plane; and use the 2D convolution layer to perform feature extraction to obtain high-level features;

[0079] (2) On a two-dimensional plane, a convolutional layer is used to generate candidate boxes; for each candidate box, a classification score is calculated to determine whether the box contains an object of a specific category; finally, a set of bounding boxes is output, each of which has an associated classification label and confidence score;

[0080] (3) Use non-maximum suppression method to remove overlapping detection results.

[0081] The weights of the PV-RCNN model are optimized by minimizing the loss function. The loss function includes classification loss and bounding box regression loss. The PV-RCNN model is evaluated using evaluation metrics including average precision and recall.

[0082] Optionally, the 3D bounding box annotation platform is used to annotate bounding boxes on 3D point cloud data. The user terminal selects, modifies and adds labels to the 3D point cloud data through the interactive tools provided by the 3D bounding box annotation platform; the 3D bounding box annotation platform integrates a pre-built 3D target detection model.

[0083] Among them, the 3D bounding box annotation platform is used to annotate the bounding boxes of three-dimensional point cloud data.

[0084] Among them, the 3D point cloud data is annotated with bounding boxes through the 3D bounding box annotation platform to generate a 3D bounding box surrounding the object. The generated 3D bounding box can accurately locate and describe each object in the scene.

[0085] Optionally, manual correction includes correcting missing, redundant, and erroneous parts of the pre-annotated results.

[0086] Among them, the results obtained after manual correction are used to train and update the model, so that the labeling results of the data collected next time in the pre-labeling stage are accurate.

[0087] S5. Combine the final semantic segmentation annotation result with the final 3D bounding box annotation result, take the overlapping point cloud of the two regions as an instance, and generate an instance label.

[0088] Among them, the final semantic segmentation annotation results can be used for 3D segmentation algorithm training tasks.

[0089] Among them, the final 3D bounding box annotation results can be used for training tasks of 2D target detection algorithms and 3D detection algorithms.

[0090] In one feasible implementation, the 3D object detection model used in the embodiment of the present invention is PV-RCNN. Point cloud data is input into the PV-RCNN model, which outputs a 3D bounding box. Cropping is performed around the target bounding box, and occlusions are randomly introduced within the ROI for data augmentation. This can improve the model's ability to handle occlusions.

[0091] A feasible implementation method is to project the final 3D bounding box annotation result onto the corresponding 2D image through the projection matrix to obtain the 2D bounding box of each object in the image. The specific form of the projection matrix depends on the internal and external parameters of the camera, including: the focal length, optical center position, rotation matrix, and translation vector information of the camera. The following formula (1) represents the projection matrix:

[0092] P=K[R|t] (1)

[0093] Where P is the projection matrix, K is the camera's intrinsic parameter matrix, which includes the focal length and optical center information. [R|t] is the extrinsic parameter matrix, which includes the camera's rotation matrix R and translation vector t.

[0094] The obtained 2D bounding box can be used for the 2D part detection task.

[0095] Instance labels can be used for training 3D segmentation algorithms. They can be used to augment instance-level data. When training with instance labels, instance labels can be copied and pasted to generate more object point clouds, improving data richness for underrepresented categories. Instance labels enrich the information content of semantic segmentation tasks, enabling the model to fully and accurately understand and interpret object instances in images.

[0096] In the embodiment of the present invention, the user terminal first inputs the pre-collected road scene image data into the 3D scene reconstruction platform, and the 3D scene reconstruction platform processes the road scene image data to obtain a 3D reconstructed scene image of the road scene; the user terminal divides the road scene point cloud data acquired under the same scene into the first batch of point cloud data, the second batch of point cloud data and the third batch of point cloud data; the first batch of point cloud data is input into the semantic segmentation and annotation platform, and manual annotation is directly performed to obtain the annotation results of the first batch of data, and the annotation results of the first batch of data are used to train the point cloud semantic segmentation model integrated in the platform. After the model is updated, the second batch of point cloud data is input into the semantic segmentation and annotation platform. The semantic segmentation and annotation platform is used to turn on the pre-annotation function in the platform. The data passes through the point cloud semantic segmentation model to obtain the point cloud semantic segmentation pre-annotation results; the user end manually corrects the point cloud semantic segmentation pre-annotation results based on the 3D reconstructed scene image to obtain the final semantic segmentation and annotation results of the second batch of data; the final semantic segmentation and annotation results of the second batch of data are input into the point cloud semantic segmentation model, and after training the model, an updated semantic segmentation model is obtained; the user end inputs the third batch of point cloud data into the semantic segmentation and annotation platform, and obtains the pre-annotation results of the third batch of data through the updated semantic segmentation model. The user end corrects the third batch of data based on the 3D reconstructed scene image. The pre-labeling results of the first batch of point cloud data are manually corrected to obtain the final semantic segmentation labeling results of the third batch of data; the first batch of point cloud data is input into the 3D bounding box labeling platform and manually labeled directly to obtain the labeling results of the first batch of data. The labeling results of the first batch of data are used to train the 3D target detection model integrated in the platform. After updating the model, the second batch of point cloud data is input into the 3D bounding box labeling platform, the pre-labeling function in the platform is turned on, and the data passes through the 3D target detection model to obtain the 3D bounding box pre-labeling results; the user end manually corrects the 3D bounding box pre-labeling results according to the 3D reconstructed scene image to obtain the final 3D bounding box of the second batch of data. The final 3D bounding box annotation results of the second batch of data are input into the 3D target detection model. After training the model, an updated 3D target detection model is obtained. The user end inputs the third batch of point cloud data into the 3D bounding box annotation platform. After the updated 3D target detection model, the pre-annotation results of the third batch of data are obtained. The user end manually corrects the pre-annotation results of the third batch of data based on the 3D reconstructed scene image to obtain the final 3D bounding box annotation results of the third batch of data. The final semantic segmentation annotation results are combined with the final 3D bounding box annotation results, and the point cloud of the overlapping part of the two areas is taken as an instance to generate instance labels.

[0097] The embodiment of the present invention uses three-dimensional modeling for auxiliary labeling, which avoids the situation where the object shape is incomplete due to occlusion or the object category is difficult to distinguish with the naked eye due to biased angles, thereby improving the accuracy and efficiency of segmentation labeling. The embodiment of the present invention combines the final semantic segmentation labeling result with the overlapping part of the final 3D bounding box labeling result to generate instance labels; wherein, the instance label can automatically distinguish each object in the point cloud, saving the time and workload of manually creating instance labels; the instance label can provide key information for the segmentation algorithm, improving the recognition and segmentation accuracy of object instances; compared with the existing technology, the embodiment of the present invention saves manual labeling time, can quickly obtain the approximate distribution of objects in a frame of point cloud data, reduces the time for labelers to perform object category and shape analysis, and can improve the quality of labeling through a combination of model prediction and manual correction.

[0098] Figure 3 This is a block diagram of a 3D point cloud annotation system based on scene reconstruction according to an exemplary embodiment. The system is used for a 3D point cloud annotation method based on scene reconstruction. Figure 3 The system includes a user terminal 310, a semantic segmentation and annotation platform 320, a 3D bounding box annotation platform 330, and a 3D scene reconstruction platform 340.

[0099] The user terminal 310 divides the road scene point cloud data acquired in the same scene into a first batch of point cloud data, a second batch of point cloud data, and a third batch of point cloud data;

[0100] The semantic segmentation and annotation platform 320 is used to input the first batch of point cloud data into the semantic segmentation and annotation platform, directly perform manual annotation, and obtain the annotation results of the first batch of data; input the second batch of point cloud data into the semantic segmentation and annotation platform, enable the pre-annotation function in the platform, and pass the data through the point cloud semantic segmentation model to obtain the point cloud semantic segmentation pre-annotation results; the user end inputs the third batch of point cloud data into the semantic segmentation and annotation platform, and passes it through the updated semantic segmentation model to obtain the pre-annotation results of the third batch of data;

[0101] The 3D bounding box annotation platform 330 is used to input the first batch of point cloud data into the 3D bounding box annotation platform and perform manual annotation directly to obtain the annotation results of the first batch of data; input the second batch of point cloud data into the 3D bounding box annotation platform, enable the pre-annotation function in the platform, and pass the data through the 3D object detection model to obtain the 3D bounding box pre-annotation results; the user terminal inputs the third batch of point cloud data into the 3D bounding box annotation platform and passes it through the updated 3D object detection model to obtain the pre-annotation results of the third batch of data;

[0102] The three-dimensional scene reconstruction platform 340 is used to process the road scene image data to obtain a three-dimensional reconstructed model of the road scene.

[0103] Optionally, the user terminal inputs pre-collected road scene image data into the 3D scene reconstruction platform, and the 3D scene reconstruction platform processes the road scene image data to obtain a 3D reconstructed scene image of the road scene, including:

[0104] The user terminal inputs the road scene image data into the 3D scene reconstruction platform, which extracts features from the input road scene image data to obtain feature points of the road scene image data; performs feature matching based on the feature points to determine the correspondence between different images;

[0105] Perform sparse 3D reconstruction based on the input road scene image data to obtain sparsely reconstructed scene sparse 3D points, camera intrinsic parameters, and camera pose data;

[0106] Estimate the depth map of the road scene based on the sparsely reconstructed sparse 3D points of the scene, the camera intrinsic parameters, and the camera pose data to obtain the depth map and normal vector map;

[0107] Based on the input road scene image data, the correspondence between different images, the camera internal parameters, the depth map and the normal vector map, the road scene is densely reconstructed to obtain a dense reconstruction result;

[0108] The dense reconstruction results are visualized to obtain dense three-dimensional point cloud data of the object surface in the road scene; the dense three-dimensional point cloud data of the object surface in the road scene is visualized to obtain a three-dimensional reconstructed scene image of the road scene.

[0109] Optionally, the semantic segmentation annotation platform 320 is used to visualize the three-dimensional point cloud data and model prediction results; the semantic segmentation annotation platform integrates a pre-built semantic segmentation model.

[0110] Optionally, the 3D bounding box annotation platform 330 is used to annotate the bounding boxes of the three-dimensional point cloud data. The user terminal selects, modifies and adds labels to the three-dimensional point cloud data through the interactive tools provided by the 3D bounding box annotation platform; the 3D bounding box annotation platform integrates a pre-built 3D target detection model.

[0111] Optionally, manual correction includes correcting missing, redundant, and erroneous parts of the pre-annotated results.

[0112] In the embodiment of the present invention, the user terminal first inputs the pre-collected road scene image data into the 3D scene reconstruction platform, and the 3D scene reconstruction platform processes the road scene image data to obtain a 3D reconstructed scene image of the road scene; the user terminal divides the road scene point cloud data acquired under the same scene into the first batch of point cloud data, the second batch of point cloud data and the third batch of point cloud data; the first batch of point cloud data is input into the semantic segmentation and annotation platform, and manual annotation is directly performed to obtain the annotation results of the first batch of data, and the annotation results of the first batch of data are used to train the point cloud semantic segmentation model integrated in the platform. After the model is updated, the second batch of point cloud data is input into the semantic segmentation and annotation platform. The semantic segmentation and annotation platform is used to turn on the pre-annotation function in the platform. The data passes through the point cloud semantic segmentation model to obtain the point cloud semantic segmentation pre-annotation results; the user end manually corrects the point cloud semantic segmentation pre-annotation results based on the 3D reconstructed scene image to obtain the final semantic segmentation and annotation results of the second batch of data; the final semantic segmentation and annotation results of the second batch of data are input into the point cloud semantic segmentation model, and after training the model, an updated semantic segmentation model is obtained; the user end inputs the third batch of point cloud data into the semantic segmentation and annotation platform, and obtains the pre-annotation results of the third batch of data through the updated semantic segmentation model. The user end corrects the third batch of data based on the 3D reconstructed scene image. The pre-labeling results of the first batch of point cloud data are manually corrected to obtain the final semantic segmentation labeling results of the third batch of data; the first batch of point cloud data is input into the 3D bounding box labeling platform and manually labeled directly to obtain the labeling results of the first batch of data. The labeling results of the first batch of data are used to train the 3D target detection model integrated in the platform. After updating the model, the second batch of point cloud data is input into the 3D bounding box labeling platform, the pre-labeling function in the platform is turned on, and the data passes through the 3D target detection model to obtain the 3D bounding box pre-labeling results; the user end manually corrects the 3D bounding box pre-labeling results according to the 3D reconstructed scene image to obtain the final 3D bounding box of the second batch of data. The final 3D bounding box annotation results of the second batch of data are input into the 3D target detection model. After training the model, an updated 3D target detection model is obtained. The user end inputs the third batch of point cloud data into the 3D bounding box annotation platform. After the updated 3D target detection model, the pre-annotation results of the third batch of data are obtained. The user end manually corrects the pre-annotation results of the third batch of data based on the 3D reconstructed scene image to obtain the final 3D bounding box annotation results of the third batch of data. The final semantic segmentation annotation results are combined with the final 3D bounding box annotation results, and the point cloud of the overlapping part of the two areas is taken as an instance to generate instance labels.

[0113] The embodiment of the present invention uses three-dimensional modeling for auxiliary labeling, which avoids the situation where the object shape is incomplete due to occlusion or the object category is difficult to distinguish with the naked eye due to biased angles, thereby improving the accuracy and efficiency of segmentation labeling. The embodiment of the present invention combines the final semantic segmentation labeling result with the overlapping part of the final 3D bounding box labeling result to generate instance labels; wherein, the instance label can automatically distinguish each object in the point cloud, saving the time and workload of manually creating instance labels; the instance label can provide key information for the segmentation algorithm, improving the recognition and segmentation accuracy of object instances; compared with the existing technology, the embodiment of the present invention saves manual labeling time, can quickly obtain the approximate distribution of objects in a frame of point cloud data, reduces the time for labelers to perform object category and shape analysis, and can improve the quality of labeling through a combination of model prediction and manual correction.

[0114] Figure 4 is a structural diagram of a three-dimensional point cloud annotation device based on scene reconstruction provided by an embodiment of the present invention, such as Figure 4 As shown, the 3D point cloud annotation device based on scene reconstruction may include the above Figure 3 The 3D point cloud annotation system based on scene reconstruction is shown. Optionally, the 3D point cloud annotation device 410 based on scene reconstruction may include a first processor 2001 .

[0115] Optionally, the three-dimensional point cloud annotation device 410 based on scene reconstruction may further include a memory 2002 and a transceiver 2003 .

[0116] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0117] The following combination Figure 4 The components of the scene reconstruction-based 3D point cloud annotation device 410 are described in detail:

[0118] The first processor 2001 is the control center of the scene reconstruction-based 3D point cloud annotation device 410, and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement an embodiment of the present invention, such as one or more microprocessors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0119] Optionally, the first processor 2001 can perform various functions of the three-dimensional point cloud annotation device 410 based on scene reconstruction by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0120] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0121] In a specific implementation, as an embodiment, the 3D point cloud annotation device 410 based on scene reconstruction may also include multiple processors, such as Figure 4 , a first processor 2001 and a second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0122] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0123] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-on-ly memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and access the memory through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0124] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0125] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 (not shown separately in the figure). The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0126] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or can exist independently and be connected to the first processor 2001 through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0127] It should be noted that Figure 4 The structure of the scene reconstruction-based three-dimensional point cloud annotation device 410 shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0128] In addition, the technical effects of the three-dimensional point cloud annotation device 410 based on scene reconstruction can refer to the technical effects of the three-dimensional point cloud annotation method based on scene reconstruction described in the above method embodiment, and will not be repeated here.

[0129] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may also be any conventional processor, etc.

[0130] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0131] The above embodiments can be implemented in whole or in part through software, hardware (such as circuits), firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired method (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, or magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0132] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0133] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0134] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0135] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0136] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0137] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0138] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0139] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0140] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0141] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A 3D point cloud annotation method based on scene reconstruction, characterized in that: The three-dimensional point cloud annotation method based on scene reconstruction is implemented by a three-dimensional point cloud annotation system based on scene reconstruction, and the three-dimensional point cloud annotation system based on scene reconstruction includes: a user terminal, a semantic segmentation annotation platform, a 3D bounding box annotation platform and a three-dimensional scene reconstruction platform; S1. A user terminal inputs pre-collected road scene image data into the 3D scene reconstruction platform, and the 3D scene reconstruction platform processes the road scene image data to obtain a 3D reconstructed scene image of the road scene; S2. The user terminal divides the road scene point cloud data obtained in the same scene into a first batch of point cloud data, a second batch of point cloud data, and a third batch of point cloud data; S3. Input the first batch of point cloud data into the semantic segmentation and annotation platform, perform manual annotation directly, and obtain the annotation results of the first batch of data. The annotation results of the first batch of data are used to train the point cloud semantic segmentation model integrated in the training platform. After updating the model, input the second batch of point cloud data into the semantic segmentation and annotation platform, turn on the pre-annotation function in the platform, and pass the data through the point cloud semantic segmentation model to obtain the point cloud semantic segmentation pre-annotation results; the user end manually corrects the point cloud semantic segmentation pre-annotation results according to the 3D reconstructed scene image to obtain the final semantic segmentation annotation results of the second batch of data; input the final semantic segmentation annotation results of the second batch of data into the point cloud semantic segmentation model, and after training the model, obtain the updated semantic segmentation model; the user end inputs the third batch of point cloud data into the semantic segmentation and annotation platform, and passes through the updated semantic segmentation model to obtain the pre-annotation results of the third batch of data. The user end manually corrects the pre-annotation results of the third batch of data according to the 3D reconstructed scene image to obtain the final semantic segmentation annotation results of the third batch of data; S4. Input the first batch of point cloud data into the 3D bounding box annotation platform and perform manual annotation directly to obtain the annotation results of the first batch of data. The annotation results of the first batch of data are used to train the 3D target detection model integrated in the training platform. After updating the model, input the second batch of point cloud data into the 3D bounding box annotation platform, turn on the pre-annotation function in the platform, and pass the data through the 3D target detection model to obtain the 3D bounding box pre-annotation results. The user end manually corrects the 3D bounding box pre-annotation results based on the 3D reconstructed scene image to obtain the final 3D bounding box annotation results of the second batch of data. Input the final 3D bounding box annotation results of the second batch of data into the 3D target detection model, and after training the model, obtain the updated 3D target detection model. The user end inputs the third batch of point cloud data into the 3D bounding box annotation platform, passes through the updated 3D target detection model to obtain the pre-annotation results of the third batch of data. The user end manually corrects the pre-annotation results of the third batch of data based on the 3D reconstructed scene image to obtain the final 3D bounding box annotation results of the third batch of data. S5. Combine the final semantic segmentation annotation result with the final 3D bounding box annotation result, take the overlapping point cloud of the two regions as an instance, and generate an instance label.

2. The three-dimensional point cloud annotation method based on scene reconstruction according to claim 1, characterized in that: The user terminal of S1 inputs the pre-collected road scene image data into the three-dimensional scene reconstruction platform, and the three-dimensional scene reconstruction platform processes the road scene image data to obtain a three-dimensional reconstructed scene image of the road scene, including: S11. The user terminal inputs the road scene image data into the 3D scene reconstruction platform. The 3D scene reconstruction platform extracts features from the input road scene image data to obtain feature points of the road scene image data; and performs feature matching based on the feature points to determine the correspondence between different images. S12, performing sparse 3D reconstruction based on the input road scene image data to obtain sparsely reconstructed scene sparse 3D points, camera intrinsic parameters, and camera pose data; S13, estimating a depth map of the road scene based on the sparsely reconstructed scene sparse 3D points, camera intrinsic parameters, and camera pose data to obtain a depth map and a normal vector map; S14, performing dense reconstruction on the road scene according to the input road scene image data, the correspondence between different images, the camera intrinsic parameters, the depth map, and the normal vector map to obtain a dense reconstruction result; S15. Visualize the dense reconstruction results to obtain dense three-dimensional point cloud data of the object surfaces in the road scene; visualize the dense three-dimensional point cloud data of the object surfaces in the road scene to obtain a three-dimensional reconstructed scene image of the road scene.

3. The three-dimensional point cloud annotation method based on scene reconstruction according to claim 1, characterized in that: The semantic segmentation and annotation platform is used to visualize three-dimensional point cloud data and model prediction results; the semantic segmentation and annotation platform integrates a pre-built semantic segmentation model.

4. The three-dimensional point cloud annotation method based on scene reconstruction according to claim 1, characterized in that: The 3D bounding box annotation platform is used to annotate bounding boxes on three-dimensional point cloud data. The user end selects, modifies and adds labels to the three-dimensional point cloud data through the interactive tools provided by the 3D bounding box annotation platform; the 3D bounding box annotation platform integrates a pre-built 3D target detection model.

5. The three-dimensional point cloud annotation method based on scene reconstruction according to claim 1, characterized in that: The manual correction includes correcting missing, redundant and erroneous parts of the pre-labeling results.

6. A 3D point cloud annotation system based on scene reconstruction, the system being configured to implement the 3D point cloud annotation method based on scene reconstruction according to any one of claims 1 to 5, the system comprising a user terminal, a semantic segmentation annotation platform, a 3D bounding box annotation platform, and a 3D scene reconstruction platform; wherein: The user terminal divides the road scene point cloud data acquired in the same scene into a first batch of point cloud data, a second batch of point cloud data and a third batch of point cloud data; The semantic segmentation and annotation platform is used to input the first batch of point cloud data into the semantic segmentation and annotation platform, directly perform manual annotation, and obtain the annotation results of the first batch of data; input the second batch of point cloud data into the semantic segmentation and annotation platform, enable the pre-annotation function in the platform, and pass the data through the point cloud semantic segmentation model to obtain the point cloud semantic segmentation pre-annotation results; the user end inputs the third batch of point cloud data into the semantic segmentation and annotation platform, and passes it through the updated semantic segmentation model to obtain the pre-annotation results of the third batch of data; The 3D bounding box annotation platform is used to input the first batch of point cloud data into the 3D bounding box annotation platform and directly perform manual annotation to obtain the annotation results of the first batch of data; input the second batch of point cloud data into the 3D bounding box annotation platform, enable the pre-annotation function in the platform, and pass the data through the 3D object detection model to obtain the 3D bounding box pre-annotation results; the user end inputs the third batch of point cloud data into the 3D bounding box annotation platform and passes it through the updated 3D object detection model to obtain the pre-annotation results of the third batch of data; The three-dimensional scene reconstruction platform is used to process road scene image data to obtain a three-dimensional reconstruction model of the road scene.

7. The three-dimensional point cloud annotation system based on scene reconstruction according to claim 6, characterized in that: The user terminal inputs pre-collected road scene image data into the three-dimensional scene reconstruction platform, and the three-dimensional scene reconstruction platform processes the road scene image data to obtain a three-dimensional reconstructed scene image of the road scene, including: The user terminal inputs the road scene image data into the 3D scene reconstruction platform, which extracts features from the input road scene image data to obtain feature points of the road scene image data; performs feature matching based on the feature points to determine the correspondence between different images; Perform sparse 3D reconstruction based on the input road scene image data to obtain sparsely reconstructed scene sparse 3D points, camera intrinsic parameters, and camera pose data; Estimate the depth map of the road scene based on the sparsely reconstructed sparse 3D points of the scene, the camera intrinsic parameters, and the camera pose data to obtain the depth map and normal vector map; Based on the input road scene image data, the correspondence between different images, the camera internal parameters, the depth map and the normal vector map, the road scene is densely reconstructed to obtain a dense reconstruction result; The dense reconstruction results are visualized to obtain dense three-dimensional point cloud data of the object surface in the road scene; the dense three-dimensional point cloud data of the object surface in the road scene is visualized to obtain a three-dimensional reconstructed scene image of the road scene.

8. The three-dimensional point cloud annotation system based on scene reconstruction according to claim 6, characterized in that: The semantic segmentation and annotation platform is used to visualize three-dimensional point cloud data and model prediction results; the semantic segmentation and annotation platform integrates a pre-built semantic segmentation model.

9. A 3D point cloud annotation device based on scene reconstruction, characterized in that: The three-dimensional point cloud annotation device based on scene reconstruction includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 5.

Images