Multi-frame point cloud target detection and segmentation fusion labeling method, device and terminal

Abstract

The invention belongs to the technical field of point cloud data processing, and particularly relates to a multi-frame point cloud target detection and segmentation fusion labeling method and device and a terminal. The method comprises the following steps: performing target detection, and converting the target detection into segmentation data; performing overlapped frame rendering processing to obtain rendered 4D point cloud data; frame stacking segmentation processing is carried out on the 4D point cloud data; and storing and rendering a segmentation labeling result. According to the method, segmentation and labeling of the point cloud data are realized at one time, repeated adjustment steps are avoided, the segmentation efficiency is improved, high-quality input is provided for subsequent data analysis and processing, and the accuracy and smoothness of target extraction and analysis in a dynamic scene are ensured. In addition, it is ensured that the rendering process can still maintain high efficiency and low latency even under a large amount of data.

Description

Technical Field

[0001] The present invention belongs to the technical field of point cloud data processing, and specifically relates to a multi-frame point cloud target detection and segmentation fusion labeling method, device and terminal. Background Art

[0002] Object detection and point cloud segmentation are two key technologies in the field of computer vision, especially in the fields of unmanned and autonomous driving. Object detection aims to identify specific objects and their locations in images or point cloud data. By detecting and tracking surrounding vehicles, pedestrians and other obstacles in real time, the system can make more accurate decisions, thereby avoiding collisions and improving driving efficiency.

[0003] Point cloud segmentation further subdivides specific objects identified by target detection into smaller parts or instances, which in turn helps the system identify different areas on the road, such as lane lines, traffic signs, etc., thereby assisting navigation and control.

[0004] In traditional point cloud annotation scenarios, object detection (boxing) and point cloud segmentation are usually performed in steps, each step depends on the output of the previous step and requires multiple fine-tuning. The specific process includes: first identifying and boxing dynamic objects, then converting the detection results into segmentation data and making preliminary adjustments, and finally segmenting static objects and further fine-tuning the segmentation of dynamic objects frame by frame. Although this process can complete the annotation task to a certain extent, it also faces several challenges.

[0005] First, each step in the process requires fine-tuning the results of the previous step, which increases the workload and time consumption of annotation. In particular, in the third step, single-frame segmentation may lead to repeated annotations, while overlapping frame segmentation may affect the accuracy of segmentation due to the interference of dynamic object afterimages. When removing these interferences, it is often necessary to sacrifice the fine-tuning effect of dynamic segmentation.

[0006] Therefore, it is necessary to provide a method for fusion labeling of target detection and point cloud segmentation to reduce unnecessary repeated fine-tuning, especially in the processing of dynamic objects. By effectively integrating the steps of target detection, segmentation adjustment and static object segmentation, we strive to complete the labeling in one go, thereby improving the labeling efficiency and reducing unnecessary complexity. Summary of the invention

[0007] In order to solve the above technical problems in the prior art, the present invention provides a multi-frame point cloud target detection and segmentation fusion labeling method, device and terminal.

[0008] To achieve the above object, the technical solution of the present invention is as follows: In a first aspect, a multi-frame point cloud target detection and segmentation fusion annotation method includes: S1, perform target detection and convert target detection into segmentation data; S2. Perform frame-overlap rendering processing on the segmented data to obtain rendered 4D point cloud data; specifically, the processing includes: S201, performing position alignment on multiple frames of point cloud data, storing the adjusted point cloud data as full data, and performing octree segmentation on the full data; S202, downsampling the full data to generate downsampled data, and performing octree segmentation on the downsampled data; S203, merging the downsampled data into a 4D point cloud data set, and rendering the 4D point cloud data set to obtain rendered 4D point cloud data; S3. Perform overlapping frame segmentation processing on the 4D point cloud data.

[0009] Furthermore, before performing target detection in step S1, the step further includes: configuring a target tag.

[0010] Furthermore, when configuring the target label, it is necessary to ensure that the same target label for target detection and point cloud segmentation is consistent.

[0011] Furthermore, the target detection is converted into segmentation data, specifically: a box is drawn for each frame of dynamic targets, and the box drawing result is converted into segmentation data; Furthermore, in step S201, the positions of the multiple frames of point cloud data are aligned, and the specific method is: taking the coordinate system of the first frame as a reference, the positions of the point cloud data of the subsequent frames are adjusted according to the translation and rotation parameters.

[0012] Furthermore, during the downsampling process of the full data, the system will record the association relationship between the downsampled point cloud and the original full data.

[0013] Furthermore, in step S3, overlapping frame segmentation processing is performed on the 4D point cloud data, specifically including: S301, using the mouse track to obtain the target position, and comparing the target position with the slice of the downsampled data to find the slice area that intersects with the target position; S302, traverse all points in the intersection slice, and if the point is within the mouse track, find the point of each frame of the full data according to the association relationship between the downsampled point cloud and the full data.

[0014] Furthermore, the method further comprises: S4. The points of the full data found in step S302 are recorded as segmentation and annotation results and stored. The points recorded in the downsampling within the mouse track in step S302 are recorded as rendering data and rendered in the interface.

[0015] In a second aspect, the present invention further provides a multi-frame point cloud target detection and segmentation fusion annotation device, which executes the above-mentioned multi-frame point cloud target detection and segmentation fusion annotation method, and the device comprises: Startup module for object detection and converting object detection into segmentation data; The overlapping frame rendering module is used for overlapping frame rendering processing to obtain the rendered 4D point cloud data; The overlapping frame segmentation module is used for overlapping frame segmentation processing.

[0016] Furthermore, the device also includes: The storage and rendering module is used to store the segmentation and annotation results and render them on the user interface.

[0017] Furthermore, the startup module is also used to configure the target tag.

[0018] In a third aspect, the present invention further provides a terminal, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned multi-frame point cloud target detection and segmentation fusion labeling method is implemented.

[0019] Compared with the prior art, the present invention has the following beneficial effects: The multi-frame point cloud target detection and segmentation fusion labeling method provided by the present invention aligns the positions of multi-frame dynamic targets after overlapping frames, then performs downsampling processing, renders the downsampled data, and then segments and labels the downsampled data to obtain segmentation and labeling results. The present invention realizes the segmentation and labeling of point cloud data at one time, avoids repeated adjustment steps, and improves the segmentation efficiency by downsampling processing, and provides high-quality input for subsequent data analysis and processing, ensuring the accuracy and fluency of target extraction and analysis in dynamic scenes. In addition, the present invention also performs octree segmentation on the full amount of data to quickly locate and process by area, ensuring that even with a large amount of data, the rendering process can still maintain high efficiency and low latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention will be clearly described below in conjunction with the accompanying drawings. Obviously, the described embodiments are not all embodiments of the present invention, and all other embodiments obtained by ordinary technicians in the field without making creative work are within the protection scope of the present invention.

[0022] It should be noted that, unless otherwise specifically stated, the relative arrangements of components and steps, and numerical expressions set forth in these embodiments should not be construed as limiting the scope of the present invention.

[0023] The following description of the exemplary embodiments is merely illustrative and is not intended to limit the present invention and its application or use in any sense. Techniques, methods and devices known to ordinary technicians in the relevant field may not be discussed in detail here, but where applicable, these techniques, methods and devices should be considered as part of this specification.

[0024] Embodiment 1 This embodiment provides a multi-frame point cloud target detection and segmentation fusion annotation method, such as Figure 1 As shown, including: S1. Configure target tags.

[0025] Among them, the targets include static targets and dynamic targets. This process requires that the classification of targets be clarified in the initial stage of data processing and unique labels be assigned to each type of target. In particular, static targets and dynamic targets need to be labeled separately in order to distinguish different categories of objects in subsequent processing.

[0026] At the same time, in order to ensure the consistency of annotation, it is necessary to ensure that the labels of the same object in the object detection and object segmentation processes are consistent. In other words, the detection label and segmentation label of the dynamic object are unified, avoiding labeling errors or confusion caused by inconsistent labels in different processing stages.

[0027] Then, target detection is performed and converted into segmentation data: a box is drawn to annotate each frame of dynamic targets and the box drawing result is converted into segmentation data.

[0028] This process is a bridge between target detection and point cloud segmentation. During the process of drawing boxes to mark dynamic objects, the system will automatically identify the position of the dynamic object and select the box, and the box selection results will be used as input data for the subsequent segmentation steps. In order to improve the marking efficiency, the system converts these box selection results into segmentation data, which facilitates the precise segmentation of the target area in the subsequent steps. In this way, the marking of dynamic objects can not only be completed efficiently, but also ensure seamless connection from target detection to segmentation, thereby improving the accuracy and processing efficiency of the overall marking.

[0029] In general, this step solves the common problems of unclear target classification and inconsistent processing steps in point cloud annotation by reasonably configuring target labels and achieving smooth transition from target detection to segmentation, laying the foundation for subsequent efficient annotation.

[0030] S2. Perform frame-overlap rendering processing on the segmented data to obtain rendered 4D point cloud data; specifically, the processing includes: S201, performing position alignment on multiple frames of point cloud data, storing the adjusted point cloud data as full data, and performing octree segmentation on the full data; First, the coordinate system of the first frame is used as a reference, and the position of the point cloud data of subsequent frames is adjusted according to the translation and rotation parameters. This process involves coordinate transformation of the point cloud of each frame so that the data of each frame can be processed in a unified coordinate system. The adjusted point cloud data is recalculated and stored as full data to ensure that all point cloud information is retained.

[0031] Next, the entire data is divided into octrees to efficiently store and retrieve point cloud data hierarchically and by region. This division method helps improve the efficiency of subsequent processing and rendering.

[0032] S202, downsampling the full data to generate downsampled data, and performing octree segmentation on the downsampled data; Downsampling is to optimize the amount of data and improve rendering efficiency. At this stage, the system downsamples the full point cloud data to generate a simplified point cloud data set. The downsampling process selects representative points according to a specific sampling strategy and removes redundant points, thereby reducing the amount of data. At the same time, the system records the association between the downsampled point cloud and the original full data, which is used to restore details or perform comparative analysis in subsequent steps. The downsampled data is also divided into octrees for rapid positioning and processing by area. The application of the octree structure here ensures that the rendering process remains efficient and low-latency even with large amounts of data.

[0033] S203, merging the downsampled data into a 4D point cloud data set, and rendering the 4D point cloud data set to obtain rendered 4D point cloud data; The downsampled point cloud data will be merged into a 4D point cloud dataset, which not only contains spatial location data, but also introduces the time dimension, that is, the time information of each frame. This 4D point cloud dataset can more accurately express the dynamic changes of objects by fusing multi-frame point cloud data. When displaying on the page, the system synthesizes the point cloud data at multiple times by stacking frames and dynamically renders them, allowing users to view the dynamic effects of objects changing over time in a visual interface. Through the stacked frame display, the changes of dynamic targets in the time dimension can be intuitively presented. At the same time, the system supports user interaction and allows operations such as adjusting the viewing angle and zooming to obtain a clearer and more comprehensive point cloud data view.

[0034] Through the efficient coordination of the above three steps, the frame stacking rendering process can not only quickly locate, store and render target objects in large-scale point cloud data, but also effectively handle the changes in spatiotemporal data in dynamic scenes. The combination of frame stacking and downsampling technology makes point cloud data processing efficient and accurate, while 4D point cloud rendering provides users with an interactive and immersive display platform, providing strong technical support for the in-depth analysis and application of point cloud data.

[0035] S3. Perform overlapping frame segmentation processing on 4D point cloud data. Specifically including: S301, using the mouse track to obtain the target position, and comparing the target position with the slice of the downsampled data to find the slice area that intersects with the target position; In the process of segmenting point cloud data, the first step is to obtain the user's interactive information on the interface through the mouse trajectory. The user marks the target area of ​​interest on the visual interface with the mouse, and the system determines the spatial position of the target based on the mouse trajectory. Specifically, the system maps the mouse trajectory to the spatial position of the downsampled point cloud. This process relies on the octree slice structure of the downsampled point cloud. During the downsampling process, the point cloud data is divided into multiple regions or slices for fast access and processing.

[0036] Next, the position corresponding to the mouse track is compared with the slices in the downsampled octree. The goal of this comparison is to find the slice areas that intersect with the mouse track. Through this process, the spatial range of the target can be accurately determined, and the slices that may contain the target point cloud data can be further filtered out. Since the downsampled data has fewer points than the full data, the use of octree slices makes the process more efficient, and can quickly locate the area containing the target, reducing the amount of subsequent calculations.

[0037] S302, traverse all points in the intersection slice, and if the point is within the mouse track, find the point of each frame of the full data according to the association relationship between the downsampled point cloud and the full data.

[0038] In step S301, the slice area containing the target has been determined, and a more refined point cloud segmentation is required. Specifically, all points in the intersection slice are traversed, and it is determined whether these points are within the area marked by the user's mouse track. The core of this operation is to determine whether the point is within the target area defined by the mouse track based on its spatial coordinates, thereby filtering out the target point.

[0039] For each eligible point, the position of the point in the full data is further accurately located based on the association between the downsampled data and the full data. Since the downsampled data is simplified and compressed based on the full data, the positional relationship of each point in the full data is recorded during the downsampling process, which makes it possible to map the points in the downsampled data back to the original full data. This step ensures that not only can the target be identified efficiently in the downsampled data, but also that the target points finally extracted accurately correspond to the original points in the full data.

[0040] Through this process, the present invention can accurately extract the target point cloud data in each frame. This extraction not only retains the spatial and temporal information of each frame, but also ensures that in dynamic scenes, even if the target object changes in multiple frames, the target points can be accurately extracted from the full amount of data and consistently labeled and processed.

[0041] Through steps S301 and S302, the present invention can quickly and accurately extract and segment targets in large-scale point cloud data. First, by comparing the mouse trajectory and the slices of the downsampled data, the spatial range of the target can be quickly located, and the slices that may contain the target can be screened out; then, by using the association between the downsampled data and the full data, the target point cloud of each frame can be accurately extracted from the full data to ensure the accuracy and consistency of the segmentation. The efficiency of point cloud segmentation is greatly improved, and high-quality input is provided for subsequent data analysis and processing.

[0042] S4. The points of the full data found in step S302 are recorded as segmentation and annotation results and stored. The points recorded in the downsampling within the mouse track in step S302 are recorded as rendering data and rendered in the interface.

[0043] Embodiment 2 This embodiment provides a multi-frame point cloud target detection and segmentation fusion labeling device, which executes the multi-frame point cloud target detection and segmentation fusion labeling method provided in the first embodiment, and the device includes: Startup module,The startup module is responsible for configuring the labels of static and dynamic targets, and converting the detection results of dynamic targets into segmentation data, laying the foundation for subsequent processing.

[0044] The stacked frame rendering module is used for stacked frame rendering to obtain rendered 4D point cloud data; the stacked frame rendering module optimizes data storage and rendering efficiency by stacking and downsampling multiple frames of point cloud data, and generates 4D point cloud data for real-time display, which corresponds to step S2 in the first embodiment.

[0045] The overlapping frame segmentation module is used for overlapping frame segmentation processing. The overlapping frame segmentation module extracts the target area based on the mouse trajectory and uses the correlation between the downsampled data and the full data to perform accurate point cloud segmentation and extraction. This corresponds to step S3 in the first embodiment.

[0046] The storage and rendering module is used to store the segmentation and annotation results and render them on the user interface.

[0047] The main task of the storage and rendering module in point cloud data processing is to ensure that the processed data can be stored efficiently and rendered and displayed on the user interface.

[0048] During the point cloud data segmentation process, the full amount of annotated point cloud data needs to be saved, including the segmentation label and related attributes (such as category, location, etc.) of each point. The storage rendering module will record this data in the database or file system to ensure the integrity and subsequent availability of the data. This process ensures that the data will not be lost and can be retrieved or reprocessed in the future.

[0049] The downsampled data is used for rendering in the user interface. Since the downsampled data has fewer points and faster rendering speed, it is suitable for real-time display. The rendering process includes displaying the downsampled point cloud data on the interface with appropriate visual effects (such as point size and color) and supporting user interaction operations such as rotating and zooming the view. This allows users to intuitively view the point cloud data and quickly identify the target object.

[0050] The storage and rendering module ensures efficient management and visualization of point cloud data by effectively storing segmented and annotated data and rendering downsampled data in real time.

[0051] Embodiment 3 This embodiment provides a terminal, including: A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the multi-frame point cloud target detection and segmentation fusion labeling method provided in the first embodiment is implemented.

[0052] The above specific implementation methods are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to examples, a person skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A multi-frame point cloud target detection and segmentation fusion annotation method, characterized in that: include: S1, perform target detection and convert target detection into segmentation data; S2. Perform frame-overlap rendering processing on the segmented data to obtain rendered 4D point cloud data; specifically, the processing includes: S201, performing position alignment on multiple frames of point cloud data, storing the adjusted point cloud data as full data, and performing octree segmentation on the full data; S202, downsampling the full data to generate downsampled data, and performing octree segmentation on the downsampled data; S203, merging the downsampled data into a 4D point cloud data set, and rendering the 4D point cloud data set to obtain rendered 4D point cloud data; S3. Perform overlapping frame segmentation processing on the 4D point cloud data.

2. The multi-frame point cloud target detection and segmentation fusion annotation method according to claim 1 is characterized in that: In step S1, before performing target detection, the following steps are also included: configuring target labels; when configuring target labels, it is necessary to ensure that the labels of the same target in target detection and point cloud segmentation are consistent.

3. The multi-frame point cloud target detection and segmentation fusion annotation method according to claim 1 is characterized in that: Convert target detection into segmentation data, specifically: draw a box to mark the dynamic target in each frame, and convert the box drawing result into segmentation data.

4. The multi-frame point cloud target detection and segmentation fusion annotation method according to claim 1, characterized in that: The specific method for aligning the positions of the multi-frame point cloud data in step S201 is: taking the coordinate system of the first frame as a reference, adjusting the positions of the point cloud data of the subsequent frames according to the translation and rotation parameters.

5. The multi-frame point cloud target detection and segmentation fusion annotation method according to claim 1, characterized in that: During the downsampling process of the full data, the system will record the association between the downsampled point cloud and the original full data.

6. The multi-frame point cloud target detection and segmentation fusion annotation method according to claim 5, characterized in that: In step S3, overlapping frame segmentation processing is performed on the 4D point cloud data, specifically including: S301, using the mouse track to obtain the target position, and comparing the target position with the slice of the downsampled data to find the slice area that intersects with the target position; S302, traverse all points in the intersection slice, and if the point is within the mouse track, find the point of each frame of the full data according to the association relationship between the downsampled point cloud and the full data.

7. The multi-frame point cloud target detection and segmentation fusion annotation method according to claim 6, characterized in that: Also includes: S4. The points of the full data found in step S302 are recorded as segmentation and annotation results and stored. The points recorded in the downsampling within the mouse track in step S302 are recorded as rendering data and rendered in the interface.

8. A multi-frame point cloud target detection and segmentation fusion labeling device, which executes the multi-frame point cloud target detection and segmentation fusion labeling method according to any one of claims 1 to 7, characterized in that: The multi-frame point cloud target detection and segmentation fusion annotation device comprises: Startup module for object detection and converting object detection into segmentation data; The overlapping frame rendering module is used for overlapping frame rendering processing to obtain the rendered 4D point cloud data; The overlapping frame segmentation module is used for overlapping frame segmentation processing.

9. The multi-frame point cloud target detection and segmentation fusion annotation device according to claim 8, characterized in that: Also includes: The storage and rendering module is used to store the segmentation and annotation results and render them on the user interface.

10. The multi-frame point cloud target detection and segmentation fusion annotation device according to claim 8, characterized in that: The startup module is also used to configure the target tag.

11. A terminal, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multi-frame point cloud target detection and segmentation fusion labeling method as described in any one of claims 1 to 7 is implemented.

Images