Efficient labeling method, device and system for large-scale point cloud data, medium and program product

Abstract

According to the efficient labeling method, device and system for the large-scale point cloud data, the medium and the program product, the large-scale point cloud data can be efficiently processed through the octree segmentation and intelligent segmentation technology, and memory consumption is remarkably reduced; an efficient visualization module is utilized, the calculation and storage requirements of a client side are reduced, and a user can carry out point cloud labeling without high-performance hardware; in addition, various intelligent labeling tools are integrated, so that the labeling process is more user-friendly, efficient and accurate. Therefore, multiple defects in the prior art are overcome, an efficient, intelligent and resource-friendly point cloud labeling solution is provided, and a new solution is provided for large-scale point cloud data processing and labeling.

Description

Technical Field

[0001] The present application relates to the field of computer vision and image processing technology, and in particular to an efficient annotation method, device, system, medium and program product for large-scale point cloud data. Background Art

[0002] In current point cloud data processing, point cloud annotation technology is the key to achieving high-precision 3D scene understanding. However, existing technologies face many challenges in practical applications, mainly in the following aspects:

[0003] (1) Challenges of large-scale point cloud instance segmentation: Existing large-scale point cloud instance segmentation technologies are highly dependent on high-performance graphics cards and large video memory, which not only limits its application scope but also affects the accuracy and efficiency of segmentation. When the video memory is insufficient, the segmentation results are often unsatisfactory and difficult to meet the needs of practical applications.

[0004] (2) Annotation visualization performance bottleneck: Traditional point cloud annotation tools often encounter performance bottlenecks when processing large-scale data due to loading and rendering large amounts of point cloud data. This not only causes freezes and delays in the annotation process, but may even cause program crashes, seriously affecting the continuity and accuracy of the annotation work.

[0005] (3) High demand for hardware resources: Most existing point cloud annotation platforms have high requirements for hardware resources, especially the need for high-performance graphics cards and large-capacity RAM, which makes it unaffordable for ordinary users and limits the popularization and application of point cloud annotation technology.

[0006] (4) Complex annotation process: Existing point cloud annotation tools are complex to operate and inefficient, especially not friendly to novice users. The complex operation process not only increases the user's learning cost, but also reduces the efficiency and accuracy of annotation.

[0007] (5) Excessive deployment requirements: Existing annotation tools need to meet complex conditions during deployment and have poor scalability. This not only increases the difficulty and cost of deployment, but also limits the flexibility and scope of application of annotation tools.

[0008] In view of this, this field urgently needs to use innovative technical means to optimize the point cloud annotation process, improve annotation efficiency and accuracy, reduce hardware resource requirements, simplify operational procedures, and improve deployment flexibility and scalability, so as to promote the application of point cloud annotation technology in a wider range of fields. Summary of the invention

[0009] In view of the shortcomings of the prior art mentioned above, the purpose of this application is to provide an efficient annotation method, device, system, medium and program product for large-scale point cloud data, which is used to solve the technical problem of how to optimize the point cloud annotation process, improve annotation efficiency and accuracy, reduce hardware resource requirements, simplify operating procedures, and improve deployment flexibility and scalability.

[0010] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides an efficient annotation system for large-scale point cloud data, including: a visualization module, a deep learning model iterative pre-segmentation module, a point cloud processing module and a data storage module; the data storage module receives and stores the original point cloud data uploaded by one or more user terminals, and sends the original point cloud data to the deep learning model iterative pre-segmentation module; the deep learning model iterative pre-segmentation module responds to the pre-segmentation instruction sent by the visualization module, uses the octree segmentation technology or the slider segmentation technology to segment the original point cloud data into multiple sub-regions, and inputs each segmented sub-region into the deep learning model for reasoning, and fuses the reasoning results of each sub-region; the point cloud data processed by the deep learning model iterative pre-segmentation module is sent to the visualization module for visualization, and the visualization module responds to the user's annotation operation and changes the rendering effect of the point cloud label in real time through the perspective-based annotation operation technology; the visualization module generates a annotation instruction log in response to the user's operation, and sends the annotation instruction log to the point cloud processing module; the point cloud processing module performs label update processing on the original point cloud data according to the annotation instruction log, and sends the processed point cloud data to the data storage module for storage.

[0011] In some embodiments of the first aspect of the present application, the visualization process of the visualization module includes: preprocessing the point cloud data based on multi-resolution level of detail technology; transmitting and rendering annotation information of the preprocessed point cloud data; and saving the annotation data.

[0012] In some embodiments of the first aspect of the present application, the process of preprocessing point cloud data based on multi-resolution level of detail technology includes: dividing the point cloud data into multiple levels and blocks to form a corresponding hierarchical structure; on the basis of the block division, generating point cloud data of different resolutions for each block to form a multi-level detail representation; dynamically selecting the corresponding resolution level for rendering according to the user's viewpoint and viewing distance, which includes using lower resolution data at a farther distance and using higher resolution data at a closer distance; using a spatial division data structure to organize and store point cloud data; based on the spatial division data structure, generating a corresponding visual data representation for each block and saving it to a visualization file, so that the visualization file corresponding to the point cloud can be loaded in the rendering window for rendering.

[0013] In some embodiments of the first aspect of the present application, the deep learning model iterative pre-segmentation module uses octree segmentation technology or slider segmentation technology to segment the original point cloud data into multiple sub-areas, and the process includes: determining the boundaries of the entire point cloud data and calculating the midpoint for segmentation; judging whether the number of points in the entire point cloud exceeds the set threshold, if not, directly sending the entire point cloud into the model, if exceeded, performing octree segmentation or slider segmentation; saving the tuples or dictionaries of the sub-areas obtained by octree segmentation or slider segmentation in the form of a list.

[0014] In some embodiments of the first aspect of the present application, the deep learning model iterative pre-segmentation module inputs each segmented sub-region into the deep learning model for reasoning, which includes: traversing the sub-region list obtained by octree segmentation or slider segmentation, and inputting each sub-region into the deep learning model in turn for reasoning; using the deep learning model to reason for each sub-region and generate a corresponding reasoning result, combining each of the reasoning results with the original position code of the sub-region to form a new tuple or dictionary; combining the reasoning results and position codes of all sub-regions to form a final list; each element in the final list is a tuple or dictionary containing the sub-region reasoning result and position information.

[0015] In some embodiments of the first aspect of the present application, the deep learning model iterative pre-segmentation module cross-fuses the model inference results of each sub-region, including: the cross-fusion method for octree segmentation includes: starting from the sub-region of the lowest level, cross-fusion is performed layer by layer, and the position relationship of each sub-region is determined through level and position coding, and data is merged in sequence; the cross-fusion method for slider segmentation includes: incremental cross-fusion along the coordinate axis, and data is merged in sequence.

[0016] In some embodiments of the first aspect of the present application, the data merging method includes: extracting the reasoning results of each sub-region to be merged, the reasoning results including the classification information and instance labels of each point; selecting one of the sub-regions to be merged, and adjusting all the instance labels of the selected sub-region so that all the instance labels in the scene constituted by the sub-regions to be merged are unique; merging the instance labels located at the boundary based on the common area of ​​adjacent sub-regions, and the merging method includes: if the two instance labels in the common area have the same position and the same category, then changing the instance labels of all points of the instance in one of the sub-regions to the label value of the instance in the other area; when duplicate points appear in the point cloud data of the stacking area of ​​each sub-region, deleting the points located in the common area of ​​one sub-region; stacking the point cloud data of each sub-region to form a new point cloud data set; reordering the instance labels so that the label value of each instance label starts from 0 and is continuous; repeating the above steps until all sub-regions in the list are merged to form a unified point cloud data set without duplicate instance labels.

[0017] In some embodiments of the first aspect of the present application, the process of the point cloud processing module performing label update processing on the original point cloud data includes: receiving the labeling instruction log file from the visualization module and extracting the labeling information, parsing multiple configuration parameters in the log file; reading the original point cloud data file in the data storage module, and creating a new point cloud data file for storing the processed data; performing coordinate conversion on the read original point cloud data, and updating the label of each point in the point cloud according to the labeling information; repeating the above steps until all the labeling information has been traversed, and storing all the processed data in the newly created point cloud data file.

[0018] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides an efficient annotation method for large-scale point cloud data, including: receiving and storing original point cloud data uploaded by one or more user terminals; in response to a pre-segmentation instruction, using an octree segmentation technology or a slider segmentation technology to segment the original point cloud data into multiple sub-regions, and inputting each segmented sub-region into a deep learning model for reasoning, and fusing the reasoning results of each sub-region; visualizing the preprocessed point cloud data, and in response to user annotation operations, changing the rendering effect of the point cloud label in real time through a perspective-based annotation operation technology; generating an annotation instruction log in response to user operations, and performing label update processing on the original point cloud data according to the annotation instruction log, and storing the processed point cloud data.

[0019] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the efficient labeling method for the large-scale point cloud data.

[0020] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer implements an efficient labeling method for large-scale point cloud data.

[0021] To achieve the above-mentioned purpose and other related purposes, the fifth aspect of the present application provides a computer device, including a memory, a processor and a computer program stored in the memory; the processor executes the computer program to implement an efficient labeling method for the large-scale point cloud data.

[0022] As described above, the efficient annotation method, device, system, medium and program product of large-scale point cloud data of the present application have the following beneficial effects:

[0023] (1) Efficient point cloud processing: Through octree segmentation and intelligent segmentation technology, the platform can efficiently process large-scale point cloud data and significantly reduce memory consumption.

[0024] (2) Low hardware requirements: By utilizing efficient visualization modules, the client’s computing and storage requirements are reduced, and users can perform point cloud annotation without high-performance hardware.

[0025] (3) Enhanced annotation efficiency: Integrate multiple intelligent annotation tools to make the annotation process more user-friendly, efficient and accurate

[0026] Therefore, the present invention not only overcomes multiple defects in the prior art, but also provides an efficient, intelligent, and resource-friendly point cloud annotation solution, providing a new solution for large-scale point cloud data processing and annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Shown is a structural schematic diagram of an efficient annotation system for large-scale point cloud data in one embodiment of the present application.

[0028] Figure 2 Shown is a schematic diagram of the processing flow of the visualization module in one embodiment of the present application.

[0029] Figure 3 Shown is a schematic diagram of an application scenario of a visualization module in one embodiment of the present application.

[0030] Figure 4Shown is a schematic diagram of the processing flow of the deep learning model iterative pre-segmentation module in one embodiment of the present application.

[0031] Figure 5 A diagram showing the difference between a traditional labeling process and a labeling process of the present invention in one embodiment of the present application.

[0032] Figure 6 Shown is a schematic diagram of the growth cycle of the iterative pre-segmentation module of the deep learning model in one embodiment of the present application.

[0033] Figure 7 Shown is a schematic diagram of the processing flow of the point cloud processing module in one embodiment of the present application.

[0034] Figure 8 Shown is a flow chart of an efficient annotation method for large-scale point cloud data in one embodiment of the present application.

[0035] Fig. 9 Shown is a schematic diagram of the structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0036] The technical solution in the embodiment of the present application will be described below in conjunction with the drawings in the embodiment of the present application. In the description of the embodiment of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0037] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0038] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0039] In order to facilitate the understanding of the present application, some professional terms involved in the embodiments of the present application are first introduced below.

[0040] Point cloud is a collection of points in three-dimensional space, usually acquired by laser scanners, cameras or other three-dimensional sensors. Each point has coordinate information (X, Y, Z), and sometimes also contains additional information such as color, intensity, normal, etc. Point cloud data is often used in 3D reconstruction, object recognition, robot navigation and other fields.

[0041] Octree segmentation is a tree-like data structure for three-dimensional space, commonly used for space segmentation and space search. In point cloud processing, octree segmentation refers to recursively dividing point cloud data into eight sub-regions (each sub-region is one eighth of the original region) until the number of points in each sub-region meets certain conditions or reaches a certain threshold. This method helps to reduce the amount of calculation and improve processing efficiency.

[0042] Sliding Window segmentation is a method for processing point cloud data. It segments the data by sliding a fixed-size window on the point cloud. The window can be moved along one or more axes. After each movement, the points in the window are selected for processing. This method can limit the number of points processed each time and is suitable for the step-by-step processing of large-scale point cloud data.

[0043] Deep learning models are a type of machine learning algorithm with a multi-layer structure. They solve tasks such as classification, regression, and detection by learning complex representations of data. In point cloud processing, deep learning models can be used for feature extraction, point cloud classification, object detection, etc.

[0044] Multi-resolution level of detail (LOD) technology refers to a method for effectively managing and rendering large-scale 3D data (such as point cloud data). This technology optimizes the rendering process by creating different resolution levels of data, thereby improving rendering efficiency and visual effects.

[0045] Semantic labels refer to the category information given to each point or object in the point cloud, such as "vehicle", "pedestrian", "building", etc. Semantic labels are used to describe the category of objects and are the basis for understanding point cloud data.

[0046] Instance labels are labels that distinguish different instances of the same type of object in a point cloud. For example, in a group of cars, instance labels can help distinguish each car, even if they are all members of the semantic class "vehicle". Instance labels are often used for more detailed object recognition and segmentation tasks.

[0047] Figure 1 The schematic diagram of the structure of an efficient annotation system for large-scale point cloud data in an embodiment of the present invention is shown. The efficient annotation system for large-scale point cloud data in this embodiment includes: a visualization module 11, a deep learning model iterative pre-segmentation module 12, a point cloud processing module 13, and a data storage module 14.

[0048] The data storage module 14 receives and stores the original point cloud data uploaded by one or more user terminals, and sends the original point cloud data to the deep learning model iteration pre-segmentation module 12. In response to the pre-segmentation instruction sent by the visualization module 11, the deep learning model iteration pre-segmentation module 12 uses the octree segmentation technology or the slider segmentation technology to segment the original point cloud data into multiple sub-regions, and inputs each segmented sub-region into the deep learning model for reasoning, and merges the reasoning results of each sub-region. The point cloud data processed by the deep learning model iteration pre-segmentation module 12 is sent to the visualization module 11. The visualization module 11 responds to the user's annotation operation and changes the rendering effect of the point cloud label in real time through the perspective-based annotation operation technology; the visualization module 11 generates a annotation instruction log in response to the user's operation, and sends the annotation instruction log to the point cloud processing module 13. The point cloud processing module 13 performs label update processing on the original point cloud data according to the annotation instruction log, and sends the processed point cloud data to the data storage module 14 for storage.

[0049] The specific facts of the above-mentioned system for efficient annotation of large-scale point cloud data provided by the present invention are described in detail below through a specific embodiment. Obviously, the described implementation is only for illustration and is not intended to limit the present invention.

[0050] The user uploads the original point cloud data through different user terminals, such as desktop computers, tablet computers, smart phones, laptop computers, VR devices, etc. The uploaded original point cloud data is transmitted to the data storage module 14. After receiving the uploaded original point cloud data, the data storage module 14 saves the data to an appropriate storage location, such as a public cloud, a private cloud or a local storage.

[0051] The data storage module 14 sends the received raw point cloud data to the deep learning model iteration pre-segmentation module 12 for pre-processing. After receiving the segmentation instruction, the module performs pre-segmentation processing on the raw point cloud data, and uses the octree segmentation technology or the slider segmentation technology to segment the point cloud data into multiple sub-areas. The segmented sub-areas are sent to the deep learning model for reasoning in turn, and the reasoning results of each area are integrated.

[0052] The point cloud data processed by the deep learning model iterative pre-segmentation module 12 is sent to the visualization module for visualization, and the user performs point cloud annotation operations in the visualization module 11. This module uses multi-resolution level of detail (LOD) technology to process and render point cloud data. Through the perspective-based annotation operation technology, efficient point cloud data rendering and real-time visualization of annotation results are achieved. Each annotation operation of the user will generate a corresponding annotation instruction log.

[0053] The user's annotation instruction log will be transmitted to the point cloud processing module 13, which will perform label update processing on the original point cloud data according to the annotation instruction log. The point cloud processing module will perform the following steps: Parse the annotation information from the log file, including the new instance label, new semantic label, old instance label, old semantic label, polygon vertex position matrix, etc. of each annotation. Update and modify the labels of the original point cloud data accordingly. Generate a new point cloud file from the processed point cloud data and store it.

[0054] Finally, the processed point cloud data will be returned to the data storage module 14 for storage. The user can access the latest annotation data at any time and perform new annotation operations based on it.

[0055] In the above, the structure of the efficient annotation system for large-scale point cloud data proposed in the embodiment of the present invention is explained, and its principle and implementation process are described accordingly through specific embodiments. In the following, each module in the system will be explained in detail.

[0056] Figure 2 The processing flow diagram of the visualization module in the embodiment of the present invention is shown, that is, the specific implementation process of the visualization module changing the rendering effect of the point cloud label in real time through the viewpoint-based annotation operation technology.

[0057] It should be understood that the visualization module 11 is equipped with a variety of manual annotation tools, allowing users to intuitively view, select, mark and edit point cloud data through a graphical user interface (GUI). The visualization module 11 not only provides annotation functions, but may also include auxiliary functions such as data loading, saving, rendering, and view switching. The viewpoint-based annotation operation technology is a method used in three-dimensional point cloud data processing and annotation, which allows users to observe and annotate point cloud data from different perspectives. The core advantage of this technology is that it provides a more intuitive and flexible interaction method, allowing users to more accurately identify and mark specific objects or features in three-dimensional space.

[0058] Exemplarily, the annotation tools preset in the visualization module 11 may be a point cloud information extraction tool, a selective label annotation tool, a point cloud cropping tool, an undo-redo tool, a sub-rendering window, an annotation tool, a multi-mode display tool, and the like.

[0059] The point cloud information extraction tool responds to user clicks, selects specific points in the 3D point cloud data and extracts the location coordinates, original semantics, and instance information of these points. The point cloud information extraction tool not only supports users to interactively obtain detailed information about points, but also has intelligent judgment functions. It can automatically detect and update the semantic and instance labels of points after the user performs annotation operations to ensure the accuracy and consistency of annotations. The point cloud information extraction tool also serves selective label annotation. When modifying specific label points, you need to know the information of these points, which can be extracted by the point cloud information extraction tool.

[0060] The Selective Label Annotation Tool responds to user selection and editing operations, accurately selecting and modifying points of specific labels when point cloud data is annotated. By using the Selective Label Annotation Tool, users can selectively annotate and modify eligible point cloud data based on preset semantic or instance label conditions, while ensuring that data points that do not meet the conditions remain intact and are not affected by the annotation operation. This not only improves the accuracy of annotation, but also makes the annotation process more efficient and targeted.

[0061] The Point Cloud Crop tool selects and crops specific sub-regions in the point cloud data in response to user cropping operations. With the Point Cloud Crop tool, users can accurately select and isolate areas of interest by drawing rectangles, polygons, or closed curves, or defining crop boxes and crop spheres. This feature allows users to focus on specific parts of the point cloud for more detailed annotation and analysis, thereby improving work efficiency and ensuring the accuracy of annotation. The cropped sub-regions can also be used for further detailed processing or studied as independent objects, enhancing the operability and practicality of point cloud data.

[0062] The undo-redo tool responds to the user's undo operation and performs undo based on the operation process log to ensure the flexibility and reversibility of the user's operation.

[0063] The sub-rendering window responds to user creation operations, creates multiple sub-windows, and each sub-window displays a different perspective of the point cloud data, so that the data can be observed and annotated from multiple directions. These sub-windows are closely synchronized with the main rendering window, and are updated in real time according to the perspective coordinate parameters and affine transformation matrix, ensuring that the annotations and observations made by the user in any sub-window can be consistently reflected in other windows, greatly improving the accuracy and efficiency of annotations.

[0064] The annotation tool annotates the point cloud data in response to the user's annotation operation. The annotation tools in this embodiment include but are not limited to a bounding box annotation tool, a polygon annotation tool, a bounding box annotation tool based on target detection, and a mouse alteration annotation tool. The bounding box annotation tool allows users to create bounding boxes, select and annotate specific areas in the point cloud, and is suitable for annotating regular cube or rectangular areas, such as the outline of a building or objects in a specific area. The polygon annotation tool allows users to select points to be annotated by drawing polygons by clicking on multiple vertices, and is suitable for annotating complex shape areas, such as road boundaries, tree groups, etc. The bounding box annotation tool based on target detection allows users to automatically generate annotation boxes using target detection algorithms, quickly locate target areas in point clouds, and is suitable for quickly annotating specific targets, such as vehicles, pedestrians, etc. The mouse alteration annotation tool allows users to alter labels directly on the point cloud with a mouse, supports adding, deleting or modifying point cloud labels, and is suitable for correcting annotation errors or refining annotation areas, such as adjusting building details, correcting tree labels, etc.

[0065] The multi-mode display tool has preset multiple modes to display and understand point cloud data, including but not limited to RGB mode, semantic mode, instance mode, altitude mode, intensity mode, and time mode. Among them, RGB mode is a color mode, representing three colors: red, green, and blue; RGB mode displays the original color of the point cloud, which is convenient for users to view the true appearance of the point cloud. Semantic mode assigns different colors to point clouds according to semantic labels, and each semantic corresponds to a color. Semantic colors are automatically generated when creating semantic labels and passed to the shader. Users can also change the color manually. Instance mode assigns different colors to point clouds according to instance labels. Each instance corresponds to a color. Instance colors are automatically generated when creating instance labels and passed to the shader. Users can also change the color manually. Altitude mode assigns different colors to point clouds according to the altitude value of point cloud data. Usually, gradient colors are used from low to high to help users understand the height changes of terrain or structures. Intensity mode assigns different colors to point clouds according to the intensity information in point cloud data, such as the reflection intensity of lidar point clouds. Different reflection intensities are displayed as different colors, which is convenient for analyzing the reflection characteristics of the object surface. Time mode uses color gradients to represent changes in point cloud data at different time points. Earlier time points use one color, and as time goes by, the color gradually transitions to another color. Users can intuitively see the time evolution of point cloud data through color changes. Alternatively, use the timeline to display the point cloud, and users can freely select time nodes or time windows to display the point cloud.

[0066] Combination Figure 2 , the processing of the visualization module is as follows:

[0067] Step S11a: pre-processing the point cloud data based on multi-resolution level of detail technology.

[0068] It should be understood that multi-resolution level of detail (LOD) technology is a method for effectively managing and rendering large-scale 3D data (such as point cloud data). This technology optimizes the rendering process by creating different resolution levels of data, thereby improving rendering efficiency and visual effects. The specific process of processing and rendering point cloud data based on multi-resolution level of detail technology is as follows:

[0069] (1) The point cloud data is divided into multiple levels and blocks to form a corresponding hierarchical structure.

[0070] The hierarchical structure referred to here can be implemented by a unified grid, voxel structure, octree structure, quadtree structure, Kd-Trees or multi-layer data structure. The purpose of splitting point cloud data into multiple levels and blocks is to organize large-scale point cloud data into small blocks that are easy to manage and process, laying the foundation for subsequent multi-resolution processing and view-related selection.

[0071] (2) Based on the block division, point cloud data of different resolutions are generated for each block to form a multi-level detail representation.

[0072] The formation of a multi-level detail representation referred to here means that each data block can have multiple versions, corresponding to different detail levels, so as to provide appropriate detail levels at different viewing distances.

[0073] (3) Dynamically select the corresponding resolution level for rendering based on the user's viewpoint and viewing distance, which includes using lower resolution data at farther distances and higher resolution data at closer distances.

[0074] Combination Figure 3 From the displayed scene diagram, it can be seen that "farther away" and "closer", "lower resolution" and "higher resolution" are relative concepts that appear in pairs. Distant objects use low-resolution data to improve rendering efficiency, while nearby objects use high-resolution data to ensure the clarity of details, thereby ensuring the optimization of the rendering process and balancing rendering quality and performance.

[0075] (4) Use spatial partitioning data structure to organize and store point cloud data.

[0076] The space partitioning data structure referred to here is mainly used to divide the three-dimensional space into multiple areas to facilitate spatial query, rendering optimization, collision detection and other operations. It can be an octree structure, a quadtree structure, a Kd tree structure, a binary space partitioning tree structure, etc.

[0077] Exemplarily, an octree structure is used to organize and store point cloud data, and the space is recursively divided into eight child nodes until the minimum unit requirement or data density requirement is met. The advantage of the octree structure is that it can efficiently perform spatial queries and data management, and is particularly suitable for block and multi-resolution processing of point cloud data. Therefore, using a spatial partitioning data structure to organize and store point cloud data can improve the efficiency of data access through spatial indexing, and provide support for dynamic rendering and view-related selection.

[0078] (5) According to the spatial division data structure, a corresponding visual data representation is generated for each block and saved in a visualization file, so that the visualization file corresponding to the point cloud can be loaded in the rendering window for rendering.

[0079] For example, according to the organizational structure of the octree, corresponding visual data representations are generated for each data block. These data representations can be point cloud models or meshes of different resolutions. These data representations are saved in visualization files. These files can be in specific formats, such as PCD, PLY or other custom formats. They contain all the point cloud information and attributes required for rendering. In the rendering window, the corresponding visualization files are dynamically loaded and rendered according to user interactions (such as translation, zooming, etc.). Due to the high efficiency of the octree structure, fast data retrieval and real-time rendering can be achieved, providing a smooth user experience.

[0080] Step S11b: transferring annotation information and rendering the preprocessed point cloud data.

[0081] (1) Respond to user annotation operations in real time, update annotation data, and effectively pass this annotation data to the shader; inside the shader, use the data structure of multi-resolution level of detail technology to filter the point cloud to determine the point cloud data that needs to be rendered under the current view and detail level, which can optimize rendering performance and ensure that only relevant points are processed and displayed.

[0082] (2) Determine and process the annotation status of each filtered rendering point. Specifically, check whether each point is selected by the annotation tool. For selected points, update the semantic and instance labels according to the user-defined labels; for unselected points, keep their original labels unchanged. This step ensures the accuracy and flexibility of annotation and allows users to edit point cloud data in detail.

[0083] (3) Render the point cloud according to the colors corresponding to the semantic and instance labels.

[0084] Specifically, semantic and instance colors are automatically generated when labels are created to ensure that each label has a unique color identity. Users can manually change the colors of semantics and instances to meet specific visual or identification needs. Subsequently, the point cloud is rendered into corresponding colors based on the semantic and instance labels, ensuring that users can immediately see the updated visual effects when performing annotation operations, thereby achieving a dynamic and interactive annotation experience.

[0085] Step S11c: Save the labeled data.

[0086] The user's annotation data can be saved as a log, which not only helps to track the user's operation history, but also provides a basis for subsequent data review and analysis. The log can be in any format that is easy to process and analyze, such as JSON, XML, or CSV. The selected format should be able to record the detailed information of the annotation, including timestamp, user ID, operation type (such as adding, modifying, deleting tags), etc. For example, when the user performs an annotation operation, the visualization module captures and records all relevant data in real time, and writes this data to the log file in a predetermined format to ensure the integrity and consistency of the data. In addition, you can consider adding error detection and data verification mechanisms to the log to ensure that the recorded data is accurate.

[0087] Figure 4 A schematic diagram of the processing flow of the iterative pre-segmentation module of the deep learning model in an embodiment of the present invention is shown.

[0088] The deep learning model iterative pre-segmentation module 12 is used to segment large-scale point clouds, use the deep learning model to perform model inference on each sub-region obtained by segmentation, and fuse the inference results of each sub-region to obtain a complete scene segmentation result.

[0089] It should be understood that when processing large-scale point cloud data (e.g., tens of millions or even hundreds of millions of points), the traditional annotation process is quite different from that of the present invention. Figure 5 shown.

[0090] In the traditional data annotation process, the entire point cloud scene usually needs to be fed into the model for semantic segmentation (category segmentation) and instance segmentation (object segmentation). Each data scene goes through data import, visual response, annotation operation, waiting for data processing, visual response, annotation operation, waiting for data processing... waiting for output processing, data export, etc. Since the model's video memory usage usually grows exponentially with the number of input points, traditional methods often occupy a large amount of video memory, resulting in video memory overflow and program interruption.

[0091] In the labeling process of the present invention, the visualization module sends the labeling instruction log to the deep learning model iterative pre-segmentation module. The deep learning model iterative pre-segmentation module performs point cloud segmentation, model reasoning and result fusion, aiming to decompose a large point cloud scene into smaller and more manageable parts, and only process one part of it at a time, thereby reducing the resources required for a single reasoning and solving the problem of video memory limitation of large-scale point cloud data in deep learning model reasoning.

[0092] Specifically, the deep learning model iterative pre-segmentation module 12 is based on the octree segmentation technology or the slider segmentation technology to limit the number of points input into the model during each reasoning process, thereby achieving efficient processing of large-scale point cloud data. However, whether it is the octree segmentation or the slider segmentation technology, after completing the deep learning model reasoning for each sub-area, these segmentation results need to be fused to obtain a complete scene segmentation result. However, the fusion of instance labels is challenging because the instance labels of each separate reasoning are random and disordered. Therefore, the deep learning model iterative pre-segmentation module proposes a point cloud fusion technology for adjacent sub-areas to ensure that the fused instance labels are consistent, thereby achieving complete and accurate scene segmentation.

[0093] Combination Figure 4 , the processing process of the deep learning model iterative pre-segmentation module is as follows:

[0094] Step S12a: Segment the point cloud based on octree segmentation or slider segmentation technology to form multiple sub-regions and limit the number of points in each sub-region. The segmentation steps are as follows:

[0095] Step 1.1: Determine the boundaries of the entire point cloud data and calculate the midpoint for segmentation.

[0096] Determining the boundaries of the entire point cloud data means first traversing the entire point cloud data set to find the minimum and maximum values ​​in each dimension (X, Y, Z). These values ​​define the overall boundaries of the point cloud data in three-dimensional space. Calculating the midpoint means that once the boundary is determined, the key coordinates of each dimension can be calculated. The midpoint is the middle value of the boundary and can be used as a reference point for point cloud segmentation. For example, the minimum value of the point cloud data on the X-axis is X1 and the maximum value is X2, then the midpoint is (X1+X2) / 2. Using these boundaries and end points, the point cloud data can be evenly divided in space, dividing the point cloud into multiple sub-areas, each of which is positioned with reference to the midpoint.

[0097] Step 1.2: Determine whether the number of points in the entire point cloud exceeds the set threshold. If not, the entire point cloud is directly sent to the model. If it exceeds, octree segmentation or slider segmentation is performed.

[0098] Step 1.3: For octree segmentation: treat the entire point cloud dataset as a cube or a cuboid, and recursively divide its space into eight smaller sub-regions, similar to dividing a cube into eight small cubes along the midpoints of the X, Y, and Z axes. In order to ensure that the segmented sub-regions can be seamlessly connected when spliced, each sub-region will retain the common parts with the adjacent sub-regions (for example, 5% of the common parts can be retained) during segmentation for subsequent splicing. Subsequently, the above segmentation process is repeated for each sub-region until the number of points in each sub-region is less than the preset threshold. In order to identify and locate each sub-region in subsequent processing, each sub-region is assigned a unique code according to its level and position in the octree structure. The code can be, for example, a string or a sequence of numbers that can clearly represent the path of the sub-region in the tree. The code of each sub-region is associated with its point data and stored in a data structure, such as a tuple or a dictionary, so that the code of each sub-region directly points to its corresponding point number and other related information, which is convenient for indexing and access. Finally, the encoding and point data arrays of all sub-areas correspond one to one and are saved in the form of tuples or dictionaries.

[0099] Step 1.4: For slider segmentation: set a cropping window (for example, a three-dimensional cropping box that can be a cube or a cuboid), which is used to perform slicing operations in the point cloud data. The initial position of the cropping window is at the minimum coordinate value of the point cloud data. The cropping window is translated from the minimum value to the maximum value along the X, Y, and Z axes in sequence. Each time it is translated, the point cloud area covered by the cropping window will be extracted for processing, and the distance of each translation depends on the size of the window and the density of the point cloud data. In order to ensure that the point cloud areas processed by adjacent cropping windows can be seamlessly spliced, the points in the common area where the current window and the previous window intersect are retained each time it is translated. Therefore, in each direction, the movement distance of the cropping window will be less than its total size to ensure overlap.

[0100] It is understandable that the size of the cropping window can be adjusted according to the density and distribution of the point cloud data. In areas with denser point clouds, smaller windows are required to control the number of points processed each time; in areas with sparser point clouds, larger windows can be used. In order to be able to identify and locate the sub-region processed by each cropping window in subsequent processing, a unique code is assigned to each window. This code can be based on the position, size or processing order of the window to ensure that each sub-region can be uniquely identified. The code and point cloud data of each sub-region processed by the cropping window are saved in a data structure, such as a tuple or dictionary. The representation of each sub-region can include information such as its code, point cloud data array, window position and size.

[0101] Therefore, in the embodiments of the present invention, large-scale point cloud data is effectively decomposed into smaller, more manageable parts through octree segmentation or slider segmentation, and encoding is used to ensure that these parts can be correctly reassembled to form a complete point cloud scene. This method not only improves the efficiency of processing large-scale point cloud data, but also ensures the accuracy and integrity of the processing results.

[0102] Step 1.5: Save the tuples or dictionaries of the sub-regions obtained by octree partitioning or slider partitioning in the form of a list.

[0103] Specifically, by using octree or slider segmentation techniques to decompose large-scale point cloud datasets into smaller sub-regions, and generating tuples or dictionaries containing unique codes and point cloud data for each sub-region, and then adding these data structures one by one to the list, a structured, easy-to-access and process dataset can be created. This approach not only optimizes the use of video memory, but also ensures the integrity and manageability of point cloud data, making efficient reasoning and subsequent analysis in deep learning models possible.

[0104] Step S12b: Use the trained deep learning model to perform model inference on each sub-region. The specific steps include:

[0105] Step 2.1: Traverse the list of sub-regions obtained by octree segmentation or slider segmentation, and input each sub-region into the deep learning model for inference in turn.

[0106] It should be understood that in the model inference stage, it is first necessary to traverse the sub-region list obtained by octree segmentation or slider segmentation, that is, to take out each sub-region in the list one by one and send it as input to the pre-trained model for inference.

[0107] Step 2.2: Use the deep learning model to infer each sub-region and generate corresponding inference results, and combine each of the inference results with the original position code of the sub-region to form a new tuple or dictionary.

[0108] After the deep learning model infers each sub-region, it will generate a corresponding inference result, which usually includes an analysis of the content of the sub-region, such as the category, location, and attributes of the object. Subsequently, this inference result needs to be combined with the original position encoding of the sub-region (i.e., its coordinates in the original image) to form a new tuple or dictionary, so that the analysis result of each sub-region is associated with its position in the original image.

[0109] Step 2.3: Combine the inference results and position codes of all sub-regions to form a final list; each element in the final list is a tuple or dictionary containing the sub-region inference result and position information.

[0110] For example, if the inference result is an N*8 array, that is, an array of N rows and 8 columns, where each row represents a point in a point cloud, and each column has a different meaning, the first 6 columns are (X, Y, Z, R, G, B), and the last two columns are semantic labels and instance labels. X is the X coordinate of the point in three-dimensional space, indicating the horizontal position; Y is the Y coordinate of the point in three-dimensional space, indicating the vertical position; Z is the Z coordinate of the point in three-dimensional space, indicating the depth position; R is the red value in the color component of the point; G is the green value in the color component of the point; B is the blue value in the color component of the point; the semantic label is the category to which the point belongs (such as people, cars, etc.); the instance label is the different instances in the same category of the point.

[0111] Therefore, the purpose of step S12b is to use the model to perform in-depth analysis of different regions in the image and preserve the contextual information of these regions in the original image. This is very critical for many visual tasks, such as object recognition in autonomous driving, lesion detection in medical image analysis, etc. In this way, the image content can be understood more accurately and provide support for further processing and decision-making.

[0112] Step S12c: cross-fusing the model inference results of each sub-region.

[0113] The cross-fusion method of octree segmentation includes: starting from the lowest level sub-region, cross-fusion is performed layer by layer, and the position relationship of each sub-region is determined by level and position coding, and data is merged in sequence.

[0114] Specifically, in octree segmentation, the image is recursively divided into smaller regions to form a tree structure. The fusion process starts from the lowest level of the tree, that is, from the smallest sub-region. At the lowest level, the reasoning results of each sub-region are fused with other adjacent sub-regions. The fusion process includes merging features, labels, or performing some form of voting mechanism to determine the final semantic and instance labels. By using hierarchical information and the position encoding of each sub-region, it is possible to determine which sub-regions are adjacent, so as to know which results need to be fused. The position encoding usually includes the coordinates and size of the sub-region in the original image. Once the adjacent relationship is determined, the reasoning results of the sub-regions can be merged. This may involve simple data superposition, weighted averaging, or more complex fusion algorithms, depending on the requirements of the task and the design of the model.

[0115] The cross-fusion method of slider segmentation includes: incremental cross-fusion along the coordinate axis, and data merging in sequence. Taking 3D data as an example, slider segmentation is segmented along the x, y, and z axes in order, and the fusion process will be performed incrementally along these axes. Adjacent sub-regions will be paired and fused. For example, if segmented along the x-axis, the first and second sub-regions will be fused first, then the second and third sub-regions, and so on.

[0116] Furthermore, the data merging involved in the octree segmentation or slider segmentation technology, the specific steps of merging are as follows:

[0117] Step 3.1: Extract the inference results in each sub-region to be merged, including the classification information and instance labels of each point. It should be understood that the instance label refers to an identifier used to distinguish different instances.

[0118] Step 3.2: Select one of the sub-regions from the sub-regions to be merged, and adjust all instance labels of the selected sub-region so that all instance labels in the scene constituted by the sub-regions to be merged are unique.

[0119] It should be understood that since instance labels in different sub-regions may be repeated, the instance labels need to be adjusted to ensure that the same instance label has a unique label in the entire scene. The specific adjustment method can be to add all instance labels of one sub-region to the maximum value of the instance labels in another sub-region plus 1. For example, assuming that the instance label range of sub-region A is from 0 to N-1, and the instance label range of sub-region B is from 0 to M-1, and M and N are different, then adding N to all instance labels of sub-region B (that is, the maximum value of the instance label of sub-region A plus 1) can ensure that the label in B will not conflict with any label in A. Therefore, when two sub-regions overlap, the same instance may be marked in both regions. By increasing the label value of one sub-region to above the maximum value of the label of the other sub-region, label conflicts of these overlapping instances can be avoided. This method can also maintain the continuity of the labels.

[0120] Step 3.3: Merge instance labels at the boundary based on the common area of ​​adjacent sub-areas. The merging method includes: if two instance labels in the common area have the same position and the same category, then change the instance label of all points of the instance in one of the sub-areas to the label value of the instance in the other area.

[0121] Step 3.4: When duplicate points appear in the point cloud data of the stacked area of ​​each sub-area, delete the points located in the common area of ​​one sub-area, so as to avoid duplicate points appearing when stacking the point cloud data of two sub-areas.

[0122] Step 3.5: Stack the point cloud data of each sub-area to form a new point cloud dataset; reorder the instance labels so that the label value of each instance label starts from 0 and is continuous.

[0123] Step 3.6: Repeat the above steps until all sub-regions in the list are merged to form a unified point cloud dataset without duplicate instance labels.

[0124] In the embodiment of the present application, the deep learning model in the above step S12b is continuously cycled and iterated through a plurality of preset closed-loop links to gradually improve the performance and adaptability of the model. The growth cycle formed is as follows: Figure 6 As shown, the specific process includes:

[0125] (1) Establish and optimize the deep learning model to enable it to have point cloud segmentation capabilities.

[0126] In this stage, a deep learning model with point cloud segmentation capabilities is first established. Methods for optimizing deep learning models include adjusting hyperparameters, improving network structure, and introducing more diverse training data to improve the generalization ability and performance of the model.

[0127] The process of building and optimizing includes: First, select a deep learning network structure suitable for the point cloud segmentation task, such as PointNet, PointCNN or U-Net 3D. Then, determine the hyperparameters of the network, such as learning rate, batch size, number of training rounds, etc. In order to make the model generalize better, enhance the training data, scale, translate the point cloud data, or add noise to simulate different acquisition environments. During the training process, continuously adjust the loss function (such as cross entropy loss function, Dice loss function, etc.) and regularization parameters to optimize the performance of the model. Through these steps, a powerful point cloud segmentation model can be established.

[0128] (2) Use a deep learning model with point cloud segmentation capabilities to infer point cloud data of unfamiliar scenes to obtain pre-labeled point cloud data.

[0129] The point cloud pre-segmentation model is used to infer point cloud data of unfamiliar scenes, so as to obtain pre-labeled point cloud data with preliminary labels. This step realizes automatic labeling and provides new coarse label data based on different scenes for the iterative update of the model.

[0130] (3) Manage and maintain the quality of the pre-annotated point cloud data.

[0131] The management and maintenance of data quality referred to here is to ensure the accuracy and consistency of training data, including correcting annotation errors and upgrading rough labeled data of new scenarios to high-quality labeled data, in preparation for improving model training effects and prediction accuracy.

[0132] (4) After the pre-annotated point cloud data is continuously obtained, managed and maintained through the deep learning model, the data set is incremented and reclassified to form a training set.

[0133] It should be understood that data increment and data classification are two important aspects in the iterative process of deep learning models. They jointly promote the improvement of model performance and the expansion of application scope, thereby realizing the continuous updating and expansion of data to ensure that the model can always adapt to new data and changing environments.

[0134] Data increment includes: increase in data volume, data diversity, improvement in data quality, data resampling, data enhancement, etc. Increase in data volume can collect more data to train the model, which helps the model learn more features and patterns and improve the generalization ability of the model. Increasing data diversity, including different scenes, lighting conditions, angles, etc., helps the model maintain stable performance in the face of various complex situations. Improving data quality means improving the quality of training data through data cleaning and labeling, ensuring that the model can learn from accurate data. Data resampling can ensure that the model does not overfit certain specific data, and can also better learn the distribution of data. Data enhancement, such as rotation, scaling, color change, etc., can improve the robustness of the model.

[0135] Data augmentation includes: introducing new categories, multi-task learning, zero-shot learning, etc. Introducing new categories helps the model recognize new categories. For example, initially the model can only recognize vehicles and pedestrians, but with data augmentation, it can recognize animals. Multi-task learning means that the model performs multiple tasks at the same time, such as object detection and classification at the same time. Zero-shot learning refers to the research and implementation of zero-shot learning or few-shot learning technology, which enables the model to recognize new categories with no or only a small amount of labeled data.

[0136] (5) Using the training set to train and retrain the deep learning model.

[0137] It should be understood that the above steps are an iterative process, so the deep learning model is trained with high-quality data that has been improved and retrained after the data is updated. During the training process, the model learns the features in the data and gradually improves its accuracy and performance on the point cloud segmentation task. The entire iterative module is a cyclical process, through continuous data collection, quality control, model training and optimization, the deep learning model can continue to learn and adapt, and ultimately achieve higher segmentation accuracy and a wider range of application scenarios. This iterative approach helps the model remain efficient and accurate in practical applications.

[0138] In addition, deep learning models can be deployed to servers or terminals through different strategies such as local deployment, personalized API deployment and cloud deployment. Local deployment means deploying the model on the user's own server or local environment, providing fast response and data privacy protection. Personalized deployment API deployment means providing an API interface, allowing users to customize functions and integration according to their needs. Users can adjust API parameters according to their needs to achieve personalized function customization, and the API interface is convenient for integration with other systems or applications to improve system scalability. Cloud deployment is to deploy the model on a cloud server, such as a public cloud or private cloud environment, and use cloud resources for large-scale computing and storage.

[0139] Figure 7 A schematic diagram of the processing flow of the point cloud processing module in an embodiment of the present invention is shown.

[0140] The point cloud processing module 13 receives the labeling instruction log file from the visualization module 11 , and performs label update processing on the original point cloud data in the data storage module 14 based on the labeling instruction log file.

[0141] Combination Figure 7 , the processing process of the point cloud processing module 13 is as follows:

[0142] Step S13a: receiving the annotation instruction log file from the visualization module 11 and extracting the annotation information, and parsing multiple configuration parameters in the log file.

[0143] The multiple configuration parameters in the parsed log file include, but are not limited to: new instance labels for each annotation, new semantic labels, old instance labels, old semantic labels, polygon vertex position matrices, world coordinate system to projection coordinate system conversion matrices, inverse matrices and indices of clipping matrices of all annotation tools, etc.

[0144] It should be understood that the annotation information extracted from the annotation instruction log is generated by the user when annotating point cloud data, and records all the details related to the annotation. The polygon vertex position matrix refers to the vertex coordinates that define the polygon in two-dimensional or three-dimensional space, which is used to determine the boundary of the annotation area. The world coordinate system to projection coordinate system conversion matrix is ​​used to convert point cloud data from the world coordinate system to the projection coordinate system to meet the needs of two-dimensional display or annotation tools. The inverse matrix of the clipping matrix of all annotation tools refers to the use of the clipping matrix to limit the scope of the annotation tool, and the inverse matrix is ​​used to restore the original space from the cropped space. The index is used to identify specific annotation tools or configuration parameters to ensure that the correct parameters are used when processing data.

[0145] Step S13b: Read the original point cloud data file in the data storage module and create a new point cloud data file for storing the processed data. Afterwards, the new point cloud data file will be retained and the original point cloud file will be deleted.

[0146] Step S13c: perform coordinate conversion on the read original point cloud data, and update the label of each point in the point cloud according to the annotation information. Repeat the above steps until all the annotation information is traversed, and store all the processed data in the newly created point cloud data file.

[0147] It should be understood that polygon tools are usually used to mark on two-dimensional planes, so points in three-dimensional space need to be converted to two-dimensional space in order to determine whether the points are correctly marked. The BOX tool is specifically used for marking in three-dimensional space, so points can be operated and labels updated directly in the three-dimensional coordinate system without additional coordinate conversion.

[0148] Taking the polygon tool as an example, the following steps are performed for each annotation: First, all points in the point cloud are multiplied by the transformation matrix from the world coordinate system to the projection coordinate system. The purpose is to convert the three-dimensional point cloud data into two-dimensional data for easy annotation and processing in the two-dimensional interface. Secondly, use the polygon point cloud clipping tool to filter out the points that are both inside the polygon and inside the clipping tool. When annotating in a two-dimensional interface, you can use the polygon tool to select an area, and use the clipping tool to further limit the scope of the annotation. Therefore, the point cloud processing module needs to be able to identify points that meet both conditions at the same time, that is, points that are both inside the polygon and inside the clipping area. The point cloud clipping tool has been explained in the visualization module above and will not be repeated here. Finally, the labels of the filtered points are updated. Specifically, this can be achieved using the selective label annotation tool mentioned in the visualization module above, which will not be repeated here.

[0149] After processing all the annotations, the point cloud processing module 13 saves the updated point cloud data as a new file and generates some visualization files for the next manual annotation or inspection.

[0150] In an embodiment of the present application, the data storage module 14 is used to store the original data uploaded by the user, the data processed by the deep learning model iterative pre-segmentation module 12, and the data processed by the point cloud processing module 13. It is combined with the deep learning model iterative pre-segmentation module 12 and the point cloud processing module 13 to cooperate with their instructions to complete the processing of the point cloud data.

[0151] It should be understood that the storage mode of the data storage module 14 can be private cloud storage, public cloud storage, local storage or server storage, etc. Private cloud storage supports data storage in a private cloud environment to ensure data security and privacy protection. Public cloud storage supports data storage in a public cloud environment, utilizing the elasticity and scalability of cloud resources. Local storage supports data storage in a local environment, providing efficient data access and processing capabilities. Server storage supports data storage in a server environment, ensuring centralized management and efficient processing of data.

[0152] It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0153] Figure 8 A schematic diagram of a process of an efficient annotation method for large-scale point cloud data in an embodiment of the present invention is shown, which specifically includes:

[0154] Step S81: receiving and storing original point cloud data uploaded by one or more user terminals.

[0155] Step S82: In response to the pre-segmentation instruction, the original point cloud data is segmented into multiple sub-regions using the octree segmentation technology or the slider segmentation technology, and each segmented sub-region is input into the deep learning model for reasoning, and the reasoning results of each sub-region are fused.

[0156] Step S83: Visualize the preprocessed point cloud data, and in response to the user's annotation operation, change the rendering effect of the point cloud label in real time through the perspective-based annotation operation technology.

[0157] Step S84: generating a labeling instruction log in response to the user operation, performing label update processing on the original point cloud data according to the labeling instruction log, and storing the processed point cloud data.

[0158] It should be noted that the principle and implementation method of the efficient annotation method for large-scale point cloud data in the embodiment of the present invention are similar to those of the efficient annotation system for large-scale point cloud data mentioned above, and will not be repeated here.

[0159] Fig. 9 The schematic block diagram of the computer device provided by the embodiment of the present application is shown. The computer device includes: at least one processor 901, a memory 902, at least one network interface 903 and a user interface 905. The various components in the device are coupled together through a bus system 904. It can be understood that the bus system 904 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 904 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Fig. 9 In the specification, various buses are labeled as bus systems.

[0160] The user interface 905 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0161] It is understood that the memory 902 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0162] The memory 902 in the embodiment of the present invention is used to store various categories of data to support the operation of the electronic terminal 900. Examples of these data include: any executable program for operating on the electronic terminal 900, such as an operating system 9021 and an application 9022; the operating system 9021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 9022 can include various applications, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. The XX method provided by the embodiment of the present invention can be included in the application 9022.

[0163] The method disclosed in the above embodiment of the present invention can be applied to the processor 901, or implemented by the processor 901. The processor 901 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 901 or the instruction in the form of software. The above processor 901 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 901 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 901 may be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0164] In an exemplary embodiment, the electronic terminal 900 may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0165] According to the method provided in the embodiment of the present application, the present application also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes an efficient labeling method for large-scale point cloud data.

[0166] According to the method provided in the embodiment of the present application, the present application also provides a computer-readable storage medium, which stores a program code. When the program code runs on a computer, the computer executes an efficient labeling method for large-scale point cloud data.

[0167] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program and / or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may, for example, communicate through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0168] Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0169] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0170] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0171] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0173] In the above embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When loading and executing computer program instructions (programs) on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integrations. Available media may be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., high-density digital video discs (DVDs), or semiconductor media (e.g., solid state disks (SSDs), etc.).

[0174] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0175] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0176] In summary, the present application provides an efficient annotation method, device, system, medium and program product for large-scale point cloud data. Through octree segmentation and intelligent segmentation technology, it can efficiently process large-scale point cloud data and significantly reduce memory consumption; using an efficient visualization module, it reduces the computing and storage requirements of the client, and users can perform point cloud annotation without high-performance hardware; in addition, it integrates a variety of intelligent annotation tools to make the annotation process more user-friendly, efficient and accurate. Therefore, the present invention not only overcomes many defects in the prior art, but also provides an efficient, intelligent, and resource-friendly point cloud annotation solution, providing a new solution for large-scale point cloud data processing and annotation. Therefore, the present application effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0177] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. An efficient annotation system for large-scale point cloud data, characterized in that: include: Visualization module, deep learning model iterative pre-segmentation module, point cloud processing module and data storage module; The data storage module receives and stores the original point cloud data uploaded by one or more user terminals, and sends the original point cloud data to the deep learning model iterative pre-segmentation module; The deep learning model iterative pre-segmentation module responds to the pre-segmentation instruction sent by the visualization module, uses the octree segmentation technology or the slider segmentation technology to segment the original point cloud data into multiple sub-regions, and inputs each segmented sub-region into the deep learning model for reasoning, and fuses the reasoning results of each sub-region; The point cloud data processed by the deep learning model iterative pre-segmentation module is sent to the visualization module for visualization. The visualization module responds to the user's annotation operation and changes the rendering effect of the point cloud label in real time through the perspective-based annotation operation technology. The visualization module generates an annotation instruction log in response to the user's operation and sends the annotation instruction log to the point cloud processing module. The point cloud processing module updates the labels of the original point cloud data according to the labeling instruction log, and sends the processed point cloud data to the data storage module for storage.

2. The efficient annotation system for large-scale point cloud data according to claim 1, characterized in that: The visualization process of the visualization module includes: preprocessing the point cloud data based on the multi-resolution level of detail technology; transmitting and rendering the annotation information of the preprocessed point cloud data; and saving the annotation data.

3. The efficient annotation system for large-scale point cloud data according to claim 2, characterized in that: The process of preprocessing point cloud data by the visualization module based on multi-resolution level of detail technology includes: Split the point cloud data into multiple levels and blocks to form a corresponding hierarchical structure; On the basis of block division, point cloud data of different resolutions are generated for each block to form a multi-level detail representation; Dynamically select the corresponding resolution level for rendering based on the user's viewpoint and viewing distance, which includes using lower resolution data at farther distances and higher resolution data at closer distances; Use spatial partitioning data structures to organize and store point cloud data; According to the spatial division data structure, a corresponding visualization data representation is generated for each block and saved in a visualization file, so that the visualization file corresponding to the point cloud can be loaded in the rendering window for rendering.

4. The efficient annotation system for large-scale point cloud data according to claim 1, characterized in that: The deep learning model iterative pre-segmentation module uses the octree segmentation technology or the slider segmentation technology to segment the original point cloud data into multiple sub-areas, and the process includes: Determine the boundaries of the entire point cloud data and calculate the midpoint for segmentation; Determine whether the number of points in the entire point cloud exceeds the set threshold. If not, the entire point cloud is directly sent to the model. If it exceeds, octree segmentation or slider segmentation is performed; Save the tuples or dictionaries of sub-regions obtained by octree partitioning or slider partitioning in the form of a list.

5. The efficient annotation system for large-scale point cloud data according to claim 1, characterized in that: The deep learning model iterative pre-segmentation module inputs each segmented sub-region into the deep learning model for reasoning, which includes: Traverse the list of sub-regions obtained by octree segmentation or slider segmentation, and input each sub-region into the deep learning model for inference in turn; After inferring each sub-region using a deep learning model, a corresponding inference result is generated, and each of the inference results is combined with the original position code of the sub-region to form a new tuple or dictionary; The reasoning results and position codes of all sub-regions are combined to form a final list; each element in the final list includes a tuple or dictionary containing the sub-region reasoning result and position information.

6. The efficient annotation system for large-scale point cloud data according to claim 1, characterized in that: The deep learning model iterative pre-segmentation module cross-integrates the model inference results of each sub-region, which includes: The cross-fusion method of octree segmentation includes: starting from the lowest level sub-region, cross-fusion is performed layer by layer, and the position relationship of each sub-region is determined by level and position coding, and data is merged in sequence; The cross-fusion method for the slider segmentation includes: increasing cross-fusion along the coordinate axis and merging data in sequence.

7. The efficient annotation system for large-scale point cloud data according to claim 6, characterized in that: The data merging methods include: Extract the inference results of each sub-region to be merged, which include the classification information and instance labels of each point; Selecting one sub-region from the sub-regions to be merged, and adjusting all instance labels of the selected sub-region so that all instance labels in the scene constituted by the sub-regions to be merged are unique; The instance labels at the boundary are merged based on the common area of ​​adjacent sub-areas. The merging method includes: if two instance labels in the common area have the same position and the same category, the instance labels of all points of the instance in one of the sub-areas are changed to the label value of the instance in the other area; When duplicate points appear in the point cloud data of the stacked areas of each sub-area, the points in the common area of ​​one sub-area are deleted; The point cloud data of each sub-area are stacked to form a new point cloud dataset; the instance labels are reordered so that the label value of each instance label starts from 0 and is continuous; Repeat the above steps until all sub-regions in the list are merged to form a unified point cloud dataset without duplicate instance labels.

8. The efficient annotation system for large-scale point cloud data according to claim 1, characterized in that: The process of the point cloud processing module performing label update processing on the original point cloud data includes: Receive the annotation instruction log file from the visualization module and extract the annotation information, and parse multiple configuration parameters in the log file; Read the original point cloud data file in the data storage module and create a new point cloud data file for storing the processed data; The original point cloud data is read and the coordinates are converted, and the labels of each point in the point cloud are updated according to the annotation information; Repeat the above steps until all the annotation information is traversed, and store all processed data in the newly created point cloud data file.

9. An efficient annotation method for large-scale point cloud data, characterized in that: include: Receive and store raw point cloud data uploaded by one or more user terminals; In response to the pre-segmentation instruction, the original point cloud data is segmented into a plurality of sub-regions using an octree segmentation technique or a slider segmentation technique, and each of the segmented sub-regions is input into a deep learning model for reasoning, and the reasoning results of each sub-region are fused; Visualize the pre-processed point cloud data, respond to user annotation operations, and change the rendering effect of the point cloud labels in real time through the perspective-based annotation operation technology; A labeling instruction log is generated in response to a user operation, and label update processing is performed on the original point cloud data according to the labeling instruction log, and the processed point cloud data is stored.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the efficient labeling method for large-scale point cloud data as claimed in claim 9 is implemented.

11. A computer program product, characterized in that The computer program product includes computer program code, and when the computer program code is executed on a computer, the computer implements the efficient labeling method for large-scale point cloud data as claimed in claim 9.

12. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the efficient labeling method for large-scale point cloud data as claimed in claim 9.

Images