Dense occupancy truth value generation method based on point cloud detection and image segmentation

Abstract

The invention relates to a dense occupancy truth value generation method and device based on point cloud detection and image segmentation, electronic equipment and a storage medium. The method comprises the following steps: generating a high-precision point cloud 3D label of a frame-by-frame instance dynamic object through an offline four-dimensional detection algorithm; generating a pixel-level semantic classification result through an image semantic segmentation algorithm, and projecting the classification result to a point cloud space; based on a high-precision point cloud 3D label, performing multi-frame splicing after separating dynamic and static objects in a time sequence scene, and reconstructing a static scene based on a Poisson reconstruction algorithm to realize point cloud densification; and carrying out dynamic and static point cloud fusion splicing, carrying out voxelization processing, and realizing semantic association between voxels through a nearest neighbor algorithm. According to the method and the device, a full-automatic process from original data to dense occupation true value generation is constructed, the accuracy of three-dimensional occupation label generation is improved, the high cost caused by manual labeling is also reduced, and high-quality true value data is provided for an automatic driving Occupancy occupation task.

Description

Technical Field

[0001] This disclosure relates to the field of autonomous driving target recognition, and more specifically, to a dense occupancy truth generation method, apparatus, electronic device, and computer-readable storage medium based on point cloud detection and image segmentation. Background Technology

[0002] As autonomous driving demands increasingly higher adaptability to complex environments, achieving high-precision and robust environmental perception has become a key research focus. Occupancy Networks (ONs), as a next-generation perception paradigm, utilize voxelized spatial modeling to achieve dense geometric representations of obstacles of arbitrary shapes, effectively addressing the perception failure issues of traditional 3D object detection in scenarios with irregularly shaped objects, small sample sizes, and occlusion. However, ON training heavily relies on dense occupancy ground truth labels, but point cloud data is inherently sparse and affected by dynamic object shadows, resulting in a significantly higher cost for dense occupancy ground truth labeling compared to 3D detection labeling. Current mainstream dense occupancy ground truth generation schemes are based on point cloud semantic segmentation. This method first uses a point cloud semantic segmentation network to perform semantic classification on each frame of point cloud, then performs Poisson reconstruction to increase point cloud density, then performs semantic mapping on the reconstructed dense point cloud, and finally voxelizes it to generate dense occupancy labels. This method is affected by the sparsity, disorder and lack of structure of point clouds, making it difficult for semantic segmentation models to classify correctly. The segmentation results are prone to jagged boundaries, and it does not consider complete temporal information, resulting in incomplete modeling of the geometric dimensions of dynamic objects.

[0003] Based on the impact of the aforementioned point cloud characteristics, the core of the problem-solving lies in enhancing the density of the point cloud, the accuracy of semantic segmentation, and the completeness of dynamic targets. Specifically, to address the density of the point cloud, multi-frame fusion is performed based on temporal point clouds, and dynamic and static point clouds are classified for processing; to address the accuracy of semantic segmentation, spatial constraints are applied by fusing information from multiple sensors; and for the complete modeling of dynamic targets, geometric completion optimization of the targets under complete temporal conditions is performed based on a point cloud detection and tracking method.

[0004] In existing technology, there is a method for dense generation of dynamic and static ground truth for 3D occupancy tasks. The specific implementation scheme is as follows: First, information such as the position, shape, and size of objects in the environment is obtained by analyzing continuous LiDAR point cloud data and annotation labels. Dynamic and static point clouds are separated based on bounding box attributes to obtain the scene point cloud and the point cloud within the bounding box of the current frame. Then, an iterative nearest-point registration method is used, introducing inter-frame velocity changes to obtain the spatial alignment of coherent scene information across multiple frames. Next, a ground segmentation algorithm is used on the processed static scene points to obtain non-ground points. Finally, the point cloud information from multiple frames is superimposed in a dynamic-static separation manner, and spatial voxelization is performed to form a dense 3D occupancy map scene representation. This method can voxelize dense scenes, transforming high-density point cloud data into a compact and structured 3D occupancy supervision ground truth. However, this method requires complete point cloud data label annotations as input, and cannot achieve fully automatic generation from raw point cloud data to dense occupancy ground truth. Furthermore, this scheme distinguishes between dynamic and static objects based on 3D bounding box attributes, but static objects lack point cloud-level semantic annotations, and occupancy labels fail to effectively distinguish drivable areas. This invention differs from the aforementioned solutions by using raw point cloud data and image point cloud data as input. It then employs offline 4D point cloud detection algorithms and 3D image detection algorithms to perform 3D tracking of dynamic objects in the point cloud and image data, generating 3D detection boxes for dynamic targets. This provides pseudo-labels with dynamic-static separation for the downstream dense occupancy generation task. Simultaneously, a method, apparatus, electronic device, and storage medium for generating spatial semantic occupancy labels are also provided. The main steps of this method include: acquiring environmental point cloud data and environmental images corresponding to the vehicle's operating environment; performing category prediction on objects in the vehicle's operating environment based on the environmental point cloud data and environmental images to obtain 3D image detection boxes and 3D point cloud detection boxes; fusing and aligning the detection boxes from the two modalities to generate corresponding 3D spatial pseudo-labels; then, using the pseudo-labels to densify the initial sparse labels corresponding to the environmental point cloud data to obtain spatial semantic occupancy labels corresponding to the vehicle's operating environment. Finally, the pseudo-labels densify the initial sparse labels corresponding to the environmental point cloud data to reduce the sparsity of the initial sparse labels, resulting in denser and more accurate spatial semantic occupancy labels. This scheme integrates image 3D detection to improve the robustness of point cloud 3D detection, thereby further enhancing the completeness of dynamic object separation. However, the scheme only performs 3D detection on the image without performing complete pixel-level segmentation of the environment. The generated occupancy labels lack semantic classification of the static background, making it impossible to distinguish drivable road areas for occupancy perception tasks. This invention improves upon the above scheme by using an image semantic segmentation algorithm to perform pixel-level classification of the static background while performing point cloud and image 3D detection tasks. Furthermore, it projects the image segmentation results into three-dimensional space based on camera-radar extrinsic parameters to obtain point cloud-level semantic segmentation results, thereby generating more accurate occupancy labels in the downstream dense occupancy generation module.

[0005] The problems with the existing technologies mentioned above are as follows: 1. Due to the lack of high-precision point cloud detection labels, the generated ground truth occupancy positions are inaccurate or misaligned; or some inventions directly remove point cloud detection and only perform a semi-automatic function for dense occupancy generation. 2. Due to the lack of complete point cloud semantic classification, the ground occupancy labels generated by static reconstruction cannot effectively distinguish drivable areas. 3. For sparse or low-density point clouds, the reconstructed voxel occupancy mesh is prone to holes and long-tailed trailing of dynamic objects.

[0006] Therefore, one or more methods are needed to solve the above problems.

[0007] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0008] The purpose of this disclosure is to provide a dense occupancy truth generation method, apparatus, electronic device, and computer-readable storage medium based on point cloud detection and image segmentation, thereby overcoming at least to some extent one or more problems caused by the limitations and defects of related technologies.

[0009] According to one aspect of this disclosure, a dense occupancy ground truth generation method based on point cloud detection and image segmentation is provided, comprising:

[0010] Using temporal point clouds and temporal images as input, and based on the point cloud detection module, high-precision point cloud 3D labels for frame-by-frame instance dynamic objects are generated through an offline four-dimensional detection algorithm;

[0011] Using temporal point clouds and temporal images as input, the image segmentation module generates pixel-level semantic classification results through an image semantic segmentation algorithm, and projects the classification results onto the point cloud space.

[0012] Based on high-precision point cloud 3D tags, dynamic and static objects in a time-series scene are separated and multi-frame stitched together. The static scene is reconstructed based on the Poisson reconstruction algorithm, thereby achieving point cloud densification and filling in gaps.

[0013] The classification results processed by the image segmentation module are combined with the densed and hole-filled point cloud, and then the static and dynamic point clouds are fused and stitched together. Voxelization is performed, and semantic association between voxels is realized through the nearest neighbor algorithm.

[0014] In one exemplary embodiment of this disclosure, the method further includes:

[0015] A Dynamic Sparse Voxel Transformer (DSVT) is used as a 3D point cloud detection model to perform offline detection on the input temporal point cloud.

[0016] The target detection boxes are divided into high groups and low groups. The high groups and low groups are associated and updated separately. The target trajectories that are successfully matched are fused by using a weighted box fusion strategy through positional similarity association.

[0017] The target trajectory is optimized by both set size and position trajectory.

[0018] In one exemplary embodiment of this disclosure, the method further includes:

[0019] Employ multi-view Figure 2 The MV2D algorithm, a 3D object detector guided by the D target, performs 3D visual detection on temporal images in the same frame, projects the obtained 3D detection boxes onto the point cloud coordinate system, and performs cross-validation between the visual 3D detection boxes and the point cloud 3D detection boxes.

[0020] In one exemplary embodiment of this disclosure, the method further includes:

[0021] The target detection bounding boxes are divided into high-value groups and low-value groups;

[0022] The high-performing groups are associated with updated trajectories, and the unupdated trajectories are associated with the low-performing groups.

[0023] The low-level group is associated with updated trajectories, and the unupdated trajectories are associated with the high-level group;

[0024] By associating locations based on similarity, a weighted bounding box fusion strategy is used to fuse successfully matched target trajectories.

[0025] In one exemplary embodiment of this disclosure, the method further includes:

[0026] In the segmentation branch of YOLO11, a multi-scale feature enhancement module is introduced to fuse feature map information from different levels. A dynamic class weighting mechanism is introduced to adaptively adjust the class weights in the loss function according to the learning difficulty of different targets during the training phase. An edge-aware auxiliary supervision mechanism is introduced to guide the main segmentation branch to learn the target contour information by introducing an additional edge prediction branch.

[0027] By using camera calibration parameters to back-project the image pixel coordinates into three-dimensional space and spatially align them with the corresponding point cloud data, a pixel-to-point cloud mapping table is established to transfer the category labels from image segmentation to point cloud points, achieving preliminary point cloud semantic segmentation. Based on a graph neural network (GNN) point cloud post-processing module, neighborhood relationship modeling is performed on the preliminary segmentation results. Through edge information propagation and label optimization in the graph structure, misclassified points are corrected and the boundary integrity of point cloud segmentation is enhanced.

[0028] In one exemplary embodiment of this disclosure, the method further includes:

[0029] For each frame in the time sequence, dynamic objects are cropped from the point cloud based on the 3D detection boxes obtained by the detection module, and the static scene point cloud and the dynamic object point cloud are separated. The multi-frame point cloud sequence is unified to the world coordinate system through pose transformation. The static scene segmentation and dynamic object segmentation obtained by the image segmentation module for the corresponding frame are integrated into a set of coordinates respectively. Then, they are combined into a complete temporal point cloud in the scene through their coordinate matrix and pose matrix. Based on the position and pose of the object in the current frame, the coordinate system of the static scene and the dynamic object is transformed into the coordinate system of the current frame and the point cloud is merged.

[0030] The stitched point cloud is preprocessed, and the point cloud normal vector is estimated based on the spatial distribution of the local neighborhood. Principal component analysis (PCA) combined with neighborhood weighted optimization is used to improve the accuracy and robustness of the normal vector estimation. The point cloud with normal vectors is input into the Poisson reconstruction algorithm. By constructing implicit surface functions at different resolutions, a coarse-to-fine hierarchical reconstruction is achieved, generating a continuous triangular mesh model. At the same time, an adaptive mesh simplification algorithm based on curvature constraints is used to reduce redundant vertices while retaining key geometric features, generating a continuous and smooth triangular mesh model.

[0031] In one exemplary embodiment of this disclosure, the method further includes:

[0032] After removing point cloud noise and outliers, the point cloud is mapped to a unified 3D mesh space. The space is then uniformly divided according to the set voxel resolution to achieve voxelization of the point cloud.

[0033] In one aspect of this disclosure, a dense occupancy truth generation apparatus based on point cloud detection and image segmentation is provided, comprising:

[0034] The point cloud detection module is used to generate high-precision point cloud 3D labels for frame-by-frame instance dynamic objects by taking temporal point clouds and temporal images as input and using an offline four-dimensional detection algorithm.

[0035] The image segmentation module is used to take temporal point clouds and temporal images as input, generate pixel-level semantic classification results through image semantic segmentation algorithms, and project the classification results onto the point cloud space.

[0036] The dense reconstruction module is used to separate dynamic and static objects in a time-series scene based on high-precision point cloud 3D tags, and stitch them together in multiple frames. It then reconstructs the static scene based on the Poisson reconstruction algorithm, thereby achieving point cloud densification and filling in gaps.

[0037] The generation module is used to fuse and stitch together the classification results processed by the image segmentation module with the densed and hole-filled point cloud, perform voxelization processing, and realize semantic association between voxels through the nearest neighbor algorithm.

[0038] In one aspect of this disclosure, an electronic device is provided, comprising:

[0039] Processor; and

[0040] A memory storing computer-readable instructions that, when executed by the processor, implement the method according to any one of the preceding claims.

[0041] In one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method according to any one of the preceding claims.

[0042] An exemplary embodiment of this disclosure discloses a dense occupancy ground truth generation method based on point cloud detection and image segmentation. The method includes: generating high-precision 3D point cloud labels for frame-by-frame dynamic objects using an offline four-dimensional detection algorithm; generating pixel-level semantic classification results using an image semantic segmentation algorithm, and projecting the classification results onto the point cloud space; based on the high-precision 3D point cloud labels, separating dynamic and static objects in a temporal scene and stitching them together across multiple frames, reconstructing the static scene using a Poisson reconstruction algorithm to achieve point cloud density; fusing and stitching the dynamic and static point clouds, performing voxelization processing, and achieving semantic association between voxels using a nearest neighbor algorithm. This disclosure constructs a fully automated process from raw data to dense occupancy ground truth generation, improving the accuracy of 3D occupancy label generation and reducing the high cost of manual annotation, providing high-quality ground truth data for autonomous driving occupancy tasks.

[0043] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0044] The above and other features and advantages of this disclosure will become more apparent from the detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0045] Figure 1 A flowchart of a dense occupancy truth generation method based on point cloud detection and image segmentation according to an exemplary embodiment of the present disclosure is shown;

[0046] Figure 2 A schematic block diagram of a dense occupancy truth generation apparatus based on point cloud detection and image segmentation according to an exemplary embodiment of the present disclosure is shown.

[0047] Figure 3 A general block diagram of a dense occupancy truth generation apparatus based on point cloud detection and image segmentation according to an exemplary embodiment of the present disclosure is shown.

[0048] Figure 4A flowchart of a dense reconstruction module of a dense occupancy truth generation apparatus based on point cloud detection and image segmentation according to an exemplary embodiment of the present disclosure is shown.

[0049] Figure 5 A dense occupancy label generation map is shown according to an exemplary embodiment of the present disclosure of a dense occupancy truth generation apparatus based on point cloud detection and image segmentation;

[0050] Figure 6 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown schematically;

[0051] Figure 7 The illustration shows a schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0052] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0053] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details described, or other methods, components, materials, apparatuses, steps, etc., can be employed. In other instances, well-known structures, methods, apparatuses, implementations, materials, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0054] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, or in one or more software-hardened modules, or in different network and / or processor devices and / or microcontroller devices.

[0055] In this example embodiment, a dense occupancy ground truth generation method based on point cloud detection and image segmentation is first provided; refer to Figure 1 As shown, this dense occupancy ground truth generation method based on point cloud detection and image segmentation may include the following steps:

[0056] Step S110: Using temporal point cloud and temporal image as input, and based on point cloud detection module 410, generate high-precision point cloud 3D labels for frame-by-frame instance dynamic objects through offline four-dimensional detection algorithm;

[0057] Step S120: Using temporal point cloud and temporal image as input, based on image segmentation module 420, pixel-level semantic classification results are generated through image semantic segmentation algorithm, and the classification results are projected onto point cloud space;

[0058] Step S130: Based on high-precision point cloud 3D tags, separate dynamic and static objects in the time sequence scene, and stitch them together in multiple frames. Reconstruct the static scene based on the Poisson reconstruction algorithm to achieve point cloud densification and fill in the gaps.

[0059] Step S140: The classification results processed by the image segmentation module 420 and the densed and hole-filled point cloud are fused and stitched together to perform voxelization processing, and the semantic association between voxels is realized through the nearest neighbor algorithm.

[0060] An exemplary embodiment of this disclosure discloses a dense occupancy ground truth generation method based on point cloud detection and image segmentation. The method includes: generating high-precision 3D point cloud labels for frame-by-frame dynamic objects using an offline four-dimensional detection algorithm; generating pixel-level semantic classification results using an image semantic segmentation algorithm, and projecting the classification results onto the point cloud space; based on the high-precision 3D point cloud labels, separating dynamic and static objects in a temporal scene and stitching them together across multiple frames, reconstructing the static scene using a Poisson reconstruction algorithm to achieve point cloud density; fusing and stitching the dynamic and static point clouds, performing voxelization processing, and achieving semantic association between voxels using a nearest neighbor algorithm. This disclosure constructs a fully automated process from raw data to dense occupancy ground truth generation, improving the accuracy of 3D occupancy label generation and reducing the high cost of manual annotation, providing high-quality ground truth data for autonomous driving occupancy tasks.

[0061] The following will further explain a dense occupancy truth generation method based on point cloud detection and image segmentation in this example embodiment.

[0062] Example 1:

[0063] In step S110, time-series point clouds and time-series images can be used as inputs, and high-precision point cloud 3D labels for frame-by-frame instance dynamic objects can be generated based on the point cloud detection module 410 and the offline four-dimensional detection algorithm.

[0064] In this example embodiment, the method further includes:

[0065] A Dynamic Sparse Voxel Transformer (DSVT) is used as a 3D point cloud detection model to perform offline detection on the input temporal point cloud.

[0066] The target detection boxes are divided into high groups and low groups. The high groups and low groups are associated and updated separately. The target trajectories that are successfully matched are fused by using a weighted box fusion strategy through positional similarity association.

[0067] The target trajectory is optimized by both set size and position trajectory.

[0068] In this example embodiment, the method further includes:

[0069] Employ multi-view Figure 2 The MV2D algorithm, a 3D object detector guided by the D target, performs 3D visual detection on temporal images in the same frame, projects the obtained 3D detection boxes onto the point cloud coordinate system, and performs cross-validation between the visual 3D detection boxes and the point cloud 3D detection boxes.

[0070] In this example embodiment, the method further includes:

[0071] The target detection bounding boxes are divided into high-value groups and low-value groups;

[0072] The high-performing groups are associated with updated trajectories, and the unupdated trajectories are associated with the low-performing groups.

[0073] The low-level group is associated with updated trajectories, and the unupdated trajectories are associated with the high-level group;

[0074] By associating locations based on similarity, a weighted bounding box fusion strategy is used to fuse successfully matched target trajectories.

[0075] In step S120, a temporal point cloud and a temporal image can be used as input. Based on the image segmentation module 420, a pixel-level semantic classification result is generated through an image semantic segmentation algorithm, and the classification result is projected onto the point cloud space.

[0076] In this example embodiment, the method further includes:

[0077] In the segmentation branch of YOLO11, a multi-scale feature enhancement module is introduced to fuse feature map information from different levels. A dynamic class weighting mechanism is introduced to adaptively adjust the class weights in the loss function according to the learning difficulty of different targets during the training phase. An edge-aware auxiliary supervision mechanism is introduced to guide the main segmentation branch to learn the target contour information by introducing an additional edge prediction branch.

[0078] By using camera calibration parameters to back-project the image pixel coordinates into three-dimensional space and spatially align them with the corresponding point cloud data, a pixel-to-point cloud mapping table is established to transfer the category labels from image segmentation to point cloud points, achieving preliminary point cloud semantic segmentation. Based on a graph neural network (GNN) point cloud post-processing module, neighborhood relationship modeling is performed on the preliminary segmentation results. Through edge information propagation and label optimization in the graph structure, misclassified points are corrected and the boundary integrity of point cloud segmentation is enhanced.

[0079] In step S130, dynamic and static objects in a time-series scene can be separated based on high-precision point cloud 3D tags, and multi-frame stitching can be performed separately. The static scene can be reconstructed based on the Poisson reconstruction algorithm to achieve point cloud densification and fill in the gaps.

[0080] In this example embodiment, the method further includes:

[0081] For each frame in the time sequence, dynamic objects are cropped from the point cloud based on the 3D detection boxes obtained by the detection module, and the static scene point cloud and the dynamic object point cloud are separated. The multi-frame point cloud sequence is unified to the world coordinate system through pose transformation. The static scene segmentation and dynamic object segmentation obtained by the image segmentation module 420 for the corresponding frame are integrated into a set of coordinates. Then, they are combined into a complete temporal point cloud in the scene through their coordinate matrix and pose matrix. Based on the position and pose of the object in the current frame, the coordinate system of the static scene and the dynamic object is transformed into the coordinate system of the current frame and the point cloud is merged.

[0082] The stitched point cloud is preprocessed, and the point cloud normal vector is estimated based on the spatial distribution of the local neighborhood. Principal component analysis (PCA) combined with neighborhood weighted optimization is used to improve the accuracy and robustness of the normal vector estimation. The point cloud with normal vectors is input into the Poisson reconstruction algorithm. By constructing implicit surface functions at different resolutions, a coarse-to-fine hierarchical reconstruction is achieved, generating a continuous triangular mesh model. At the same time, an adaptive mesh simplification algorithm based on curvature constraints is used to reduce redundant vertices while retaining key geometric features, generating a continuous and smooth triangular mesh model.

[0083] In step S140, the classification results processed by the image segmentation module 420 and the densed and hole-filled point cloud can be fused and stitched together to perform voxelization processing, and semantic association between voxels can be realized through the nearest neighbor algorithm.

[0084] In this example embodiment, the method further includes:

[0085] After removing point cloud noise and outliers, the point cloud is mapped to a unified 3D mesh space. The space is then uniformly divided according to the set voxel resolution to achieve voxelization of the point cloud.

[0086] In this example embodiment, the present invention is mainly divided into four modules: a point cloud detection module 410, an image segmentation module 420, a dense reconstruction module 430, and an occupancy generation module 440. Specifically, the point cloud detection module 410 takes temporal point clouds and images as input and uses an offline four-dimensional detection algorithm to obtain high-precision 3D point cloud labels for dynamic objects in each frame; the image segmentation module 420 outputs pixel-level semantic classification results through an image semantic segmentation algorithm and projects them onto the point cloud space; the dense reconstruction module 430 separates dynamic and static objects in the entire temporal scene, performs multi-frame stitching on each, and then uses the Poisson reconstruction algorithm to reconstruct the static scene, achieving density and filling gaps; the occupancy generation module 440 fuses and stitches dynamic and static point clouds, performs voxelization, and then uses a nearest neighbor algorithm to achieve semantic association between voxels. This invention constructs a fully automated process from raw data to dense occupancy ground truth generation, further improving the accuracy of 3D occupancy label generation and reducing the high cost of manual annotation, providing high-quality ground truth data for autonomous driving occupancy tasks.

[0087] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0088] Furthermore, in this example embodiment, a dense occupancy truth generation apparatus based on point cloud detection and image segmentation is also provided. (Refer to...) Figure 2 As shown, the dense occupancy truth generation device 400 based on point cloud detection and image segmentation may include: a point cloud detection module 410, an image segmentation module 420, a dense reconstruction module 430, and an occupancy generation module 440. The overall framework of the technical solution is as follows: Figure 3 As shown.

[0089] The point cloud detection module 410 is used to generate high-precision point cloud 3D labels for frame-by-frame instance dynamic objects by taking temporal point clouds and temporal images as input and using an offline four-dimensional detection algorithm.

[0090] In this example embodiment, the point cloud detection module 410 is mainly divided into three parts: detection, tracking, and optimization.

[0091] Detection Phase: A Dynamic Sparse Voxel Transformer (DSVT) is used as the 3D point cloud detection model to perform offline detection on the input temporal point cloud. To improve detection accuracy, multi-view detection is also employed. Figure 2The Multi-View 2D Objects Guided 3D Object Detector (MV2D) algorithm performs 3D visual detection on temporal images within the same frame, projects the obtained 3D detection boxes onto the point cloud coordinate system, and performs cross-validation between the visual 3D detection boxes and the point cloud 3D detection boxes to reduce the rate of missed detections and false detections in point cloud detection.

[0092] Tracking phase: First, a two-stage association strategy is adopted to divide the target detection boxes into high-level and low-level groups. The high-level group is associated with updated trajectories, and the unupdated trajectories are associated with the low-level group. Then, the above method is executed in reverse to generate another set of trajectories, which are associated through positional similarity. Finally, a weighted box fusion (WBF) strategy is used to fuse all successfully matched target trajectories, further improving the integrity of the beginning and end of the dynamic target sequence.

[0093] Optimization Phase: First, the geometry of the detection box is optimized based on the transformer network. T frames are selected from the complete instance tracking sequence, and features of these t frames are extracted using the PointNet algorithm to generate the initial geometric query Q. _ The algorithm first uses geo, then randomly selects a set of point clouds from the tracking sequence, and uses PointNet to generate dense features, corresponding to geometric keys Key_geo and geometric values ​​V_geo. Finally, the geometric query and key-value pairs are fed into a cross-attention mechanism to supplement the viewpoint features and optimize the geometric dimensions. Next, the position trajectory is optimized by selecting the distances from fixed points in the instance bounding box to the box center and eight corner points as global position features, i.e., position keys K_pos and position values ​​V_pos. Features are then extracted from all points within the target trajectory as position queries Q_pos. Finally, the position queries and key-value pairs are fed into a cross-attention mechanism to encode the local to global position, predict offsets and angular shifts, thereby adjusting the target's position trajectory.

[0094] The image segmentation module 420 is used to take temporal point clouds and temporal images as input, generate pixel-level semantic classification results through an image semantic segmentation algorithm, and project the classification results onto the point cloud space.

[0095] In this example embodiment, the image segmentation module 420 is mainly divided into two parts: image semantic segmentation and point cloud spatial projection.

[0096] Image Semantic Segmentation: To address the issues of blurred semantic segmentation boundaries and insufficient small target recognition in complex scenes, this invention proposes structural and strategic improvements based on the YOLO11 framework. In terms of model structure, this invention introduces a multi-scale feature enhancement module into the segmentation branch of YOLO11. By fusing feature map information from different levels, it enhances the ability to express target boundaries and fine-grained structures. Regarding model training, a dynamic class weighting mechanism is introduced. Based on the learning difficulty of different targets during the training phase, the class weights in the loss function are adaptively adjusted to mitigate recognition bias caused by uneven data distribution. Furthermore, this invention designs an edge-aware auxiliary supervision mechanism. By introducing an additional edge prediction branch, it guides the main segmentation branch to learn target contour information more accurately, thereby improving the boundary fit and detail integrity of image segmentation.

[0097] Point cloud spatial projection: Based on obtaining high-quality image segmentation results, a cross-modal mapping relationship between the image and the point cloud is constructed. The image pixel coordinates are back-projected into 3D space using camera calibration parameters and spatially aligned with the corresponding point cloud data. By establishing a pixel-to-point cloud mapping table, the category labels from image segmentation are transferred to point cloud points, achieving preliminary point cloud semantic segmentation. To improve the continuity and consistency of the segmentation results, this invention introduces a point cloud post-processing module based on a graph neural network (GNN) to model neighborhood relationships in the preliminary segmentation results. Through edge information propagation and label optimization in the graph structure, misclassified points are corrected and the boundary integrity of the point cloud segmentation is enhanced.

[0098] The image segmentation module 420 is based on YOLO11 with structural optimization and improved training strategy, which significantly improves image segmentation performance and further realizes semantic alignment between images and point clouds.

[0099] The dense reconstruction module 430 is used to separate dynamic and static objects in a time-series scene based on high-precision point cloud 3D tags, and stitch them together in multiple frames. It then reconstructs the static scene based on the Poisson reconstruction algorithm, thereby achieving point cloud densification and filling in gaps.

[0100] In this example embodiment, the dense reconstruction module 430 is mainly divided into two parts: temporal point cloud stitching and dense reconstruction.

[0101] Temporal point cloud stitching: For each frame in the temporal sequence, dynamic objects are first cropped from the point cloud based on the 3D detection bounding boxes obtained by the detection module, separating the static scene point cloud and the dynamic object point cloud. Then, the multi-frame point cloud sequence is unified to the world coordinate system through pose transformation. The static scene segmentation and dynamic object segmentation obtained by the image segmentation module 420 for the corresponding frames are integrated into a set of coordinates, and then combined into a complete temporal point cloud within the scene using their coordinate matrix and pose matrix. The transformed static scene segmentation and dynamic object segmentation are denoted as follows: and Where K and M are the number of static scenes and dynamic objects in the sequence, respectively. Since the same object in different frames can be identified based on its bounding box ID, therefore, P... s It can represent the entire static scene, that is and Finally, based on the position and pose of objects in the current frame, the coordinate systems of the static scene and dynamic objects are transformed into the coordinate system of the current frame, and T... s and T o Merged into the point cloud of this frame: P = T s (P s )·T o (P o Using the above method, the current point cloud frame makes full use of the point cloud information of the entire scene sequence, enhancing the dense representation of the scene.

[0102] Dense Reconstruction: Although the point cloud density P after stitching together complete temporal point clouds is much higher than that of the original single-frame point cloud, gaps and holes still exist in the point cloud. This is mainly due to the sparse beams of the LiDAR or the limited vertical field of view, resulting in uneven point cloud distribution. To solve this problem, the stitched point cloud is first preprocessed, including denoising, outlier removal, and normal vector consistency correction, to improve reconstruction quality. Then, the point cloud normal vectors are estimated based on the spatial distribution of the local neighborhood, and principal component analysis (PCA) combined with neighborhood weighted optimization is used to improve the accuracy and robustness of normal vector estimation. On this basis, the point cloud with normal vectors is input into the Poisson reconstruction algorithm, and a multi-scale fusion strategy is introduced. That is, by constructing implicit surface functions at different resolutions, a layered reconstruction from coarse to fine is achieved, generating a continuous triangular mesh model. At the same time, an adaptive mesh simplification algorithm based on curvature constraints is used to reduce redundant vertices while retaining key geometric features, improving the compactness of the reconstructed model and generating a continuous and smooth triangular mesh model. This method effectively fills in the void regions of the point cloud, enhancing surface continuity and geometric details. The dense reconstruction module 430 workflow is as follows: Figure 4 As shown.

[0103] The generation module 440 is used to perform dynamic and static point cloud fusion and stitching by combining the classification results processed by the image segmentation module 420 with the densed and hole-filled point cloud, perform voxelization processing, and realize semantic association between voxels through the nearest neighbor algorithm.

[0104] In this example embodiment, the occupancy generation module 440 aims to convert dense point clouds into dense voxel representations with clear semantic boundaries. First, point cloud noise and outliers are removed to obtain a dense point cloud with a complete structure and uniform distribution. Then, the point cloud is mapped onto a unified three-dimensional mesh space, and the space is uniformly divided according to a set voxel resolution to achieve voxelization of the point cloud. During this process, voxels containing only point cloud data are marked as "occupied," while the rest are marked as "unoccupied," forming an initial sparse occupancy-labeled voxel mesh. To assign semantic information to all voxels, this invention employs a nearest neighbor algorithm to propagate semantics throughout the voxel space. Specifically, for each unoccupied voxel, the nearest occupied voxel in its spatial location is found using a K-nearest neighbor search strategy, and the semantic label carried by that neighboring voxel is passed to the current voxel. This process effectively fills semantic gaps, extending the originally sparse semantic labels to the entire voxel space, achieving continuous propagation of semantic information. Considering computational efficiency, this method can be accelerated in parallel using GPUs to improve the processing speed of semantic voxelization. The resulting dense voxels not only retain the geometric structure of the original point cloud but also possess clear semantic boundaries. The dense occupancy label generation process is as follows: Figure 5 As shown.

[0105] In this example embodiment, the present invention constructs a fully automated process from raw data to occupancy label generation. That is, by inputting raw temporal point cloud data and image data, dense occupancy labels can be generated, reducing manual annotation costs and ensuring high label quality. The system architecture of the present invention is decoupled, where the point cloud detection and image segmentation models can be replaced according to actual applications. Custom models can be defined or mature models can be reused, eliminating the need for complete training of all modules and making it highly operable.

[0106] It should be noted that although several modules or units of a dense occupancy truth generation apparatus 400 based on point cloud detection and image segmentation have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0107] Furthermore, in an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0108] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented as entirely hardware embodiments, entirely software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, collectively referred to herein as “circuit,” “module,” or “system.”

[0109] The following reference Figure 6 To describe an electronic device 500 according to such an embodiment of the present invention. Figure 6 The electronic device 500 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0110] like Figure 6 As shown, the electronic device 500 is manifested in the form of a general-purpose computing device. The components of the electronic device 500 may include, but are not limited to: at least one processing unit 510, at least one storage unit 520, a bus 530 connecting different system components (including storage unit 520 and processing unit 510), and a display unit 540.

[0111] The storage unit stores program code that can be executed by the processing unit 510, causing the processing unit 510 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 510 can perform actions such as... Figure 1 Steps S110 to S140 are shown in the diagram.

[0112] Storage unit 520 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 5201 and / or cache memory 5202, and may further include a read-only memory (ROM) 5203.

[0113] Storage unit 520 may also include a program / utility 5204 having a set (at least one) program module 5205, such program module 5205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0114] Bus 530 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0115] Electronic device 500 can also communicate with one or more external devices 570 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 500, and / or with any device that enables electronic device 500 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 550. Furthermore, electronic device 500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 560. As shown, network adapter 560 communicates with other modules of electronic device 500 via bus 530. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0116] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0117] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section above.

[0118] refer to Figure 7 As shown, a program product 600 for implementing the above-described method according to an embodiment of the present invention is described. It may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0119] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0120] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0121] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0122] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0123] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0124] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0125] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A dense occupancy truth generation method based on point cloud detection and image segmentation, characterized in that, The method includes: Using temporal point clouds and temporal images as input, and based on the point cloud detection module, high-precision point cloud 3D labels for frame-by-frame instance dynamic objects are generated through an offline four-dimensional detection algorithm; Using temporal point clouds and temporal images as input, the image segmentation module generates pixel-level semantic classification results through an image semantic segmentation algorithm, and projects the classification results onto the point cloud space. Based on high-precision point cloud 3D tags, dynamic and static objects in a time-series scene are separated and multi-frame stitched together. The static scene is reconstructed based on the Poisson reconstruction algorithm, thereby achieving point cloud densification and filling in gaps. The classification results processed by the image segmentation module are combined with the densed and hole-filled point cloud, and then the static and dynamic point clouds are fused and stitched together. Voxelization is performed, and semantic association between voxels is realized through the nearest neighbor algorithm.

2. The method as described in claim 1, characterized in that, The method further includes: A Dynamic Sparse Voxel Transformer (DSVT) is used as a 3D point cloud detection model to perform offline detection on the input temporal point cloud. The target detection boxes are divided into high groups and low groups. The high groups and low groups are associated and updated separately. The target trajectories that are successfully matched are fused by using a weighted box fusion strategy through positional similarity association. The target trajectory is optimized by both set size and position trajectory.

3. The method as described in claim 2, characterized in that, The method further includes: The MV2D algorithm, a 3D object detector guided by multi-view 2D objects, is used to perform 3D visual detection on temporal images in the same frame. The obtained 3D detection boxes are projected onto the point cloud coordinate system, and cross-validation is performed on the visual 3D detection boxes and the point cloud 3D detection boxes.

4. The method as described in claim 2, characterized in that, The method further includes: The target detection bounding boxes are divided into high-value groups and low-value groups; The high-performing groups are associated with updated trajectories, and the unupdated trajectories are associated with the low-performing groups. The low-level group is associated with updated trajectories, and the unupdated trajectories are associated with the high-level group; By associating locations based on similarity, a weighted bounding box fusion strategy is used to fuse successfully matched target trajectories.

5. The method as described in claim 1, characterized in that, The method further includes: In the segmentation branch of YOLO11, a multi-scale feature enhancement module is introduced to fuse feature map information from different levels. A dynamic class weighting mechanism is introduced to adaptively adjust the class weights in the loss function according to the learning difficulty of different targets during the training phase. An edge-aware auxiliary supervision mechanism is introduced to guide the main segmentation branch to learn the target contour information by introducing an additional edge prediction branch. By using camera calibration parameters to back-project the image pixel coordinates into three-dimensional space and spatially align them with the corresponding point cloud data, a pixel-to-point cloud mapping table is established to transfer the category labels from image segmentation to point cloud points, achieving preliminary point cloud semantic segmentation. Based on a graph neural network (GNN) point cloud post-processing module, neighborhood relationship modeling is performed on the preliminary segmentation results. Through edge information propagation and label optimization in the graph structure, misclassified points are corrected and the boundary integrity of point cloud segmentation is enhanced.

6. The method as described in claim 1, characterized in that, The method further includes: For each frame in the time sequence, dynamic objects are cropped from the point cloud based on the 3D detection boxes obtained by the detection module, and the static scene point cloud and the dynamic object point cloud are separated. The multi-frame point cloud sequence is unified to the world coordinate system through pose transformation. The static scene segmentation and dynamic object segmentation obtained by the image segmentation module for the corresponding frame are integrated into a set of coordinates respectively. Then, they are combined into a complete temporal point cloud in the scene through their coordinate matrix and pose matrix. Based on the position and pose of the object in the current frame, the coordinate system of the static scene and the dynamic object is transformed into the coordinate system of the current frame and the point cloud is merged. The stitched point cloud is preprocessed, and the point cloud normal vector is estimated based on the spatial distribution of the local neighborhood. Principal component analysis (PCA) combined with neighborhood weighted optimization is used to improve the accuracy and robustness of the normal vector estimation. The point cloud with normal vectors is input into the Poisson reconstruction algorithm. By constructing implicit surface functions at different resolutions, a coarse-to-fine hierarchical reconstruction is achieved, generating a continuous triangular mesh model. At the same time, an adaptive mesh simplification algorithm based on curvature constraints is used to reduce redundant vertices while retaining key geometric features, generating a continuous and smooth triangular mesh model.

7. The method as described in claim 1, characterized in that, The method further includes: After removing point cloud noise and outliers, the point cloud is mapped to a unified 3D mesh space. The space is then uniformly divided according to the set voxel resolution to achieve voxelization of the point cloud.

8. A dense occupancy truth generation device based on point cloud detection and image segmentation, characterized in that, The device includes: The point cloud detection module is used to generate high-precision point cloud 3D labels for frame-by-frame instance dynamic objects by taking temporal point clouds and temporal images as input and using an offline four-dimensional detection algorithm. The image segmentation module is used to take temporal point clouds and temporal images as input, generate pixel-level semantic classification results through image semantic segmentation algorithms, and project the classification results onto the point cloud space. The dense reconstruction module is used to separate dynamic and static objects in a time-series scene based on high-precision point cloud 3D tags, and stitch them together in multiple frames. It then reconstructs the static scene based on the Poisson reconstruction algorithm, thereby achieving point cloud densification and filling in gaps. The generation module is used to fuse and stitch together the classification results processed by the image segmentation module with the densed and hole-filled point cloud, perform voxelization processing, and realize semantic association between voxels through the nearest neighbor algorithm.

9. An electronic device, characterized in that, include Processor; and A memory storing computer-readable instructions that, when executed by the processor, implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images