3d scene semantic segmentation method and device, system, storage medium

Abstract

The application discloses a 3D scene semantic segmentation method and device, system and storage medium, comprising: preprocessing two-dimensional images and three-dimensional laser radar point cloud data synchronously acquired; carrying out double-branch heterogeneous feature extraction on the preprocessed two-dimensional images and three-dimensional laser radar point cloud data to obtain 2D visual features and 3D geometric features; carrying out shared-private feature decomposition and shared attention fusion on the 2D visual features and 3D geometric features to obtain multi-modal features; inputting a decoder and combining an up-sampling operation to restore the point cloud spatial resolution and outputting the probability distribution of the category to which each point belongs. The technical scheme of the application solves the problems of redundancy, unexplainability and weak cross-domain generalization ability existing in multi-modal feature fusion in the prior art large-scale three-dimensional semantic segmentation.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a 3D scene semantic segmentation method, device, system, and storage medium. Background Technology

[0002] Semantic segmentation of large-scale 3D point clouds is a core problem in the field of intelligent perception, providing crucial support for autonomous driving, urban reconstruction, and digital twin modeling. To achieve reliable scene understanding, current perception systems typically combine LiDAR measurement data with RGB images to simultaneously acquire accurate spatial geometry and rich color texture information. Although LiDAR and cameras acquire data using different principles, they describe the same physical world and share consistent semantic content in terms of object categories and spatial relationships.

[0003] However, existing multimodal fusion strategies and their practical applications still have the following technical shortcomings and challenges: Feature representation is redundant and obscure: Existing fusion methods often simply concatenate or mix 2D image features with 3D point cloud features, lacking an explicit semantic decoupling process. Due to the lack of effective feature separation between different modalities, the fused features contain a large amount of redundant information and ambiguity, making it difficult to accurately characterize consistent semantics across modalities.

[0004] The computational overhead is huge and the scalability is poor: Many existing solutions rely on heavy feature cascading or large-scale attention mechanisms, which greatly increases the computational burden and limits their real-time processing capabilities and scalability in large-scale outdoor scenarios.

[0005] Insufficient training stability and interpretability: Using full feature fusion often leads to unstable optimization during training. Furthermore, the lack of transparency in the fusion process makes it difficult to explain exactly which modalities the model learned and what specific semantic contributions, which is detrimental to system debugging and safety assessment.

[0006] Weak environmental generalization ability: In practical applications, models often face severe performance degradation due to changes in lighting, weather shifts, or different sensor configurations. Many frameworks handle modal fusion and domain adaptation separately, making it difficult for models to maintain stable perception accuracy when faced with unfamiliar urban layouts or environmental changes.

[0007] Cross-modal alignment is difficult: The sparse sampling mode of lidar and the geometric distortion that varies with the viewing angle in image observation make cross-modal feature alignment extremely complex, hindering stable multimodal information fusion.

[0008] In summary, existing multimodal fusion strategies still have significant gaps in terms of accuracy, efficiency, interpretability, and cross-domain robustness, necessitating a novel technical solution that can efficiently decouple cross-modal semantics and achieve stable fusion. Summary of the Invention

[0009] To address the problems existing in the prior art, this invention provides a 3D scene semantic segmentation method, device, system, and storage medium, which solves the problems of redundancy, lack of interpretability, and weak cross-domain generalization ability in the multimodal feature fusion of existing large-scale 3D semantic segmentation.

[0010] To achieve the above objectives, the present invention provides the following solution: A 3D scene semantic segmentation method, comprising: Step S1: Preprocess the synchronously acquired 2D images and 3D LiDAR point cloud data; Step S2: Perform bi-branch heterogeneous feature extraction on the preprocessed 2D image and 3D LiDAR point cloud data to obtain 2D visual features and 3D geometric features; Step S3: Perform shared-private feature decomposition and shared attention fusion on 2D visual features and 3D geometric features to obtain multimodal features; Step S4, The input decoder, combined with upsampling operations, restores the spatial resolution of the point cloud and outputs the probability distribution of the category to which each point belongs.

[0011] Preferably, in step S1, the projection index between two-dimensional pixels and three-dimensional points is calculated to make the two-dimensional image and the three-dimensional LiDAR point cloud data semantically aligned in space.

[0012] Preferably, step S3 includes: Through a learnable linear projection layer, 2D visual features and 3D geometric features are decomposed into shared components. and private components ; Based on shared components and private components Perform shared attention fusion to obtain shared features ; Shared features With three-dimensional private components The features are spliced ​​together to form multimodal features.

[0013] The present invention also provides a 3D scene semantic segmentation device, comprising: The first processing module is used to preprocess the synchronously acquired two-dimensional images and three-dimensional lidar point cloud data; The second processing module is used to perform bi-branch heterogeneous feature extraction on the preprocessed two-dimensional image and three-dimensional lidar point cloud data to obtain 2D visual features and 3D geometric features. The third processing module is used to perform shared-private feature decomposition and shared attention fusion on 2D visual features and 3D geometric features to obtain multimodal features; The fourth processing module is used to... The input decoder, combined with upsampling operations, restores the spatial resolution of the point cloud and outputs the probability distribution of the category to which each point belongs.

[0014] Preferably, the first output module calculates the projection index between two-dimensional pixels and three-dimensional points, so that the two-dimensional image and the three-dimensional LiDAR point cloud data are semantically aligned in space.

[0015] Preferably, the third processing module includes: The first processing unit is used to decompose 2D visual features and 3D geometric features into shared components through a learnable linear projection layer. and private components ; The second processing unit is used to process the shared components. and private components Perform shared attention fusion to obtain shared features ; The third processing unit is used to process shared features. With three-dimensional private components The features are spliced ​​together to form multimodal features.

[0016] The present invention also provides a 3D scene semantic segmentation system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a 3D scene semantic segmentation method when executed by the processor.

[0017] The present invention also provides a storage medium storing a computer program, which executes a 3D scene semantic segmentation method when running.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: This application achieves significant technical results in improving perception accuracy, enhancing model robustness, and optimizing computational efficiency by introducing a shared-private feature decomposition mechanism and structured regularization constraints. Attached Figure Description

[0019] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the 3D scene semantic segmentation method of the present invention. Figure 2 A schematic diagram showing the implementation details of the feature decoupling module. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Example 1 like Figure 1 , 2 As shown, the present invention provides a 3D scene semantic segmentation device, comprising: Step S1: Preprocess the synchronously acquired 2D image and 3D LiDAR point cloud data; wherein, the preprocessing includes coordinate alignment; Step S2: Perform bi-branch heterogeneous feature extraction on the preprocessed 2D image and 3D LiDAR point cloud data to obtain 2D visual features and 3D geometric features; Step S3: Perform shared-private feature decomposition and shared attention fusion on 2D visual features and 3D geometric features to obtain multimodal features; Step S4, The input decoder, combined with upsampling operations, restores the spatial resolution of the point cloud and outputs the probability distribution of the category to which each point belongs.

[0024] As one embodiment of the present invention, in step S1, the projection index between two-dimensional pixels and three-dimensional points is calculated so that the two-dimensional image and the three-dimensional LiDAR point cloud data are semantically aligned in space.

[0025] In one embodiment of the present invention, step S2, obtaining the 2D visual features includes: We employ SAM (Segment Anything Model) as the visual encoder, leveraging its powerful cross-dataset generalization ability to extract multi-scale visual tokens. A lightweight transformation layer projects the high-dimensional visual features onto a 256-dimensional unified latent space, yielding two-dimensional features. .

[0026] The calculation process for visual feature extraction is as follows, with the input being a two-dimensional RGB image. ; 1. Image segmentation and embedding: This involves segmenting the input image... The image is divided into non-overlapping blocks, each block being sized as follows: The resulting image patch sequence is: Each image patch is mapped to a linear projection layer. The initial visual token is obtained by embedding the dimensional space and superimposing positional encoding: in This is the hidden dimension of the SAM encoder.

[0027] 2. Multi-scale Transformer Encoding: Initial token... The input to SAM's Vision Transformer encoder, after... Multi-head self-attention (MSA) and feedforward network (FFN) processing: Select the first Layer output as shallow texture features , With the Layers as high-level semantic features This constitutes a multi-scale visual token set. 3. Spatial Reshaping and Interpolation The visual token sequence is reshaped into a two-dimensional feature map, and the spatial resolution is restored through bilinear interpolation to obtain a multi-scale feature map: 4. Lightweight projection onto a unified latent space. (Using...) Convolution as a lightweight transformation layer Projecting high-dimensional features onto a 256-dimensional unified latent space: in, It is a learnable projection matrix.

[0028] 5. Multi-scale fusion and output. Deep and shallow features are aligned and weighted through upsampling to obtain the final two-dimensional visual features: Final output, two-dimensional features .

[0029] In one embodiment of the present invention, step S2, obtaining the 3D geometric features includes: using Sparse Convolution-Transformer (SPTNet) as a geometric encoder; utilizing Sparse Convolution (SpConv) without altering the irregular distribution of the point cloud and without modifying the local geometric structure; introducing a Transformer block after each convolutional block, and modeling long-distance spatial relationships through a multi-head self-attention mechanism, thereby generating a structurally complete and semantically aligned 3D representation. .

[0030] The calculation process for 3D geometric feature extraction is as follows, with the input being 3D LiDAR point cloud data: Each point , Reflection intensity: 1. Voxelization and Sparse Tensor Construction: The sparse point cloud is discretized using a uniform voxel grid, with voxel size of... This yields a non-empty set of voxels: in, For the number of non-empty voxels, Average pooling is applied to in-voxel features. A sparse tensor is constructed: , Record the 3D mesh coordinates of each non-empty voxel. .

[0031] 2. Local Feature Extraction from Sparse Convolution To preserve the sparsity of point clouds, submanifold sparse convolution is used for the first... Sparse features of layer input The kernel weights are In the effective neighborhood Internal calculation: in, For relative offset indexes, only and The effective neighborhood is computed in a non-spatial manner. Layered geometric features are extracted through multi-layer sparse convolution.

[0032] 3. Voxel coordinate mapping and position encoding. The center coordinates of non-empty voxels are mapped to position embeddings, which are then summed as sparse features: 4. Sparse Transformer Global Relationship Modeling. A sparse Transformer block is introduced after each convolutional layer to model all non-empty voxel features. Perform multi-head self-attention calculation 5. Multi-scale feature fusion and upsampling, for the first... Layer output Upsampled to the original voxel resolution using trilinear interpolation: Weighted fusion of features from each layer: in, Align with 2D branch dimensions.

[0033] 6. Voxel-to-point mapping Mapping voxel features back to the original point cloud: Final output: point-by-point 3D features The 2D visual features are used as input for step S3 and then shared-private decomposition.

[0034] As one embodiment of the present invention, step S3 includes: Step S31: Decompose the 2D visual features and 3D geometric features into shared components through a learnable linear projection layer. and private components ; Step S32: Based on the shared components and private components Perform shared attention fusion to obtain shared features ; Step S33: Shared features With three-dimensional private components The features are spliced ​​together to form multimodal features.

[0035] Furthermore, in step S31, the two-dimensional features are... and three-dimensional features Decomposed into shared components and private components : Applying Gram matrix alignment loss By alignment and The second-order statistical properties (Gram matrix) force the shared subspace to capture the associated semantic structure.

[0036] Set shared features The calculation process is as follows: 1. Calculate the Gram matrix: 2. Alignment loss: By forcing the shared subspaces of the two modalities to have the same feature-related structure, the semantic consistency of the captured data is ensured.

[0037] Apply decorrelation loss By punishing private components and Cross-correlation between subspaces ensures that subspaces are independent of each other and prevents model-specific noise from interfering with the extraction of shared semantics.

[0038] Assume private characteristics The calculation process is as follows: 1. Calculate the cross-correlation matrix: 2. Eliminate related losses: Furthermore, step S32 is: using three-dimensional shared features For query, two-dimensional shared features For keys and values, attention weights are calculated using the Softmax function. This achieves point-level cross-modal semantic completion; the calculated weights A are used to perform a weighted summation of the bimodal features to obtain the final fused shared representation. .

[0039] Assume input sharing feature , Point cloud number, For shared feature dimensions: 1. Linear projection generates Q, K, and V matrices. in , For a learnable projection matrix, typically 2. Calculate attention weights in Calculate point cross-modal similarity. Use a scaling factor to prevent gradient vanishing.

[0040] 3. Weighted fusion Through attention weight Two-dimensional features Weighted extraction enables adaptive extraction of two-dimensional semantic information from three-dimensional query points, outputting a fused and shared representation. .

[0041] Furthermore, step S33 is: to merge the shared representation With three-dimensional private components The data is then concatenated to generate the final compact multimodal representation. .

[0042] In one embodiment of the present invention, step S4 further includes: designing the total loss function. Perform end-to-end optimization: Segmentation loss ( ): Combining cross-entropy and Lovász-Softmax loss optimizes the balance between class prediction accuracy and class boundaries.

[0043] Knowledge alignment loss ( ): Bidirectional semantic adaptation is achieved by aligning the predicted distributions of image branches and point cloud branches using bidirectional KL divergence.

[0044] Regularization loss: includes the above and This ensures the stability of the decomposition process.

[0045] The proposed method achieves 81.0% mIoU on the nuScenes validation set and 71.8% mIoU on SemanticKITTI. Experiments demonstrate that, compared to traditional full-feature concatenation methods, this approach effectively reduces semantic fragmentation and isolated artifacts, significantly improving the continuity of object contours.

[0046] This application achieves significant technical results in improving perception accuracy, enhancing model robustness, and optimizing computational efficiency by introducing a shared-private feature decomposition mechanism and structured regularization constraints.

[0047] 1. Significantly improved the perceptual accuracy of multimodal semantic segmentation: The model achieves an mIoU of 81.0% on the nuScenes benchmark and 71.8% on the SemanticKITTI benchmark, both outperforming existing representative fusion benchmarks. By integrating SAM visual priors with SPTNet geometric features, the model provides clearer object contours and more continuous semantic regions, effectively addressing the segmentation discontinuity problem when point cloud distribution is sparse.

[0048] 2. Enhanced interpretability of feature representation and significantly reduced information redundancy: The shared-private decomposition mechanism explicitly decouples multimodal features into modality-invariant components and modality-specific components, making the semantic contributions of different modalities in the fusion process transparent and interpretable. Constraints reduce cross-modal redundancy in the feature space, making multimodal embedding representations more compact and avoiding interference from irrelevant modal variations on shared semantic extraction.

[0049] 3. Excellent cross-domain generalization ability and environmental robustness: In cross-domain evaluation, the model achieved a mIoU of 74.5%, demonstrating extremely strong stability. The decomposition strategy separated unstable domain-related features and preserved a consistent semantic kernel across modalities, enabling the model to maintain reliable performance even when faced with changes in sensor configuration or heterogeneous urban layouts.

[0050] 4. Optimized balance between computational efficiency and inference latency: Employing a lightweight shared attention (SAF) module, cross-modal switching is limited to critical shared subspaces, resulting in an inference latency of only 240ms on SemanticKITTI. While maintaining high performance, redundant computation is limited, demonstrating higher computational efficiency and engineering practicality compared to other fusion schemes based on vision models.

[0051] 5. Improved model training stability and convergence quality: Introduced Gram matrix alignment loss. Alignment loss with predicted distribution This ensures consistent guidance of multimodal semantics during training, making the optimization process more stable and less prone to falling into local optima.

[0052] Example 2 The present invention also provides a 3D scene semantic segmentation device, comprising: The first processing module is used to preprocess the synchronously acquired two-dimensional images and three-dimensional lidar point cloud data; The second processing module is used to perform bi-branch heterogeneous feature extraction on the preprocessed two-dimensional image and three-dimensional lidar point cloud data to obtain 2D visual features and 3D geometric features. The third processing module is used to perform shared-private feature decomposition and shared attention fusion on 2D visual features and 3D geometric features to obtain multimodal features; The fourth processing module is used to... The input decoder, combined with upsampling operations, restores the spatial resolution of the point cloud and outputs the probability distribution of the category to which each point belongs.

[0053] As one embodiment of the present invention, the first output module calculates the projection index between two-dimensional pixels and three-dimensional points, so that the two-dimensional image and the three-dimensional LiDAR point cloud data are semantically aligned in space.

[0054] As one embodiment of the present invention, the third processing module includes: The first processing unit is used to decompose 2D visual features and 3D geometric features into shared components through a learnable linear projection layer. and private components ; The second processing unit is used to process the shared components. and private components Perform shared attention fusion to obtain shared features ; The third processing unit is used to process shared features. With three-dimensional private components The features are concatenated into multimodal features.

[0055] Example 3 The present invention also provides a 3D scene semantic segmentation system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a 3D scene semantic segmentation method when executed by the processor.

[0056] Example 4 The present invention also provides a storage medium storing a computer program, which executes a 3D scene semantic segmentation method when running.

[0057] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A 3D scene semantic segmentation method, characterized in that, include: Step S1: Preprocess the synchronously acquired 2D images and 3D LiDAR point cloud data; Step S2: Perform bi-branch heterogeneous feature extraction on the preprocessed 2D image and 3D LiDAR point cloud data to obtain 2D visual features and 3D geometric features; Step S3: Perform shared-private feature decomposition and shared attention fusion on 2D visual features and 3D geometric features to obtain multimodal features; Step S4, The input decoder, combined with upsampling operations, restores the spatial resolution of the point cloud and outputs the probability distribution of the category to which each point belongs.

2. The 3D scene semantic segmentation method as described in claim 1, characterized in that, In step S1, the projection index between two-dimensional pixels and three-dimensional points is calculated to make the two-dimensional image and the three-dimensional LiDAR point cloud data semantically aligned in space.

3. The 3D scene semantic segmentation method as described in claim 1, characterized in that, Step S3 includes: Through a learnable linear projection layer, 2D visual features and 3D geometric features are decomposed into shared components. and private components ; Based on shared components and private components Perform shared attention fusion to obtain shared features ; Shared features With three-dimensional private components The features are spliced ​​together to form multimodal features.

4. A 3D scene semantic segmentation device, characterized in that, include: The first processing module is used to preprocess the synchronously acquired two-dimensional images and three-dimensional lidar point cloud data; The second processing module is used to perform bi-branch heterogeneous feature extraction on the preprocessed two-dimensional image and three-dimensional lidar point cloud data to obtain 2D visual features and 3D geometric features. The third processing module is used to perform shared-private feature decomposition and shared attention fusion on 2D visual features and 3D geometric features to obtain multimodal features; The fourth processing module is used to... The input decoder, combined with upsampling operations, restores the spatial resolution of the point cloud and outputs the probability distribution of the category to which each point belongs.

5. The 3D scene semantic segmentation device as described in claim 4, characterized in that, The first output module calculates the projection index between two-dimensional pixels and three-dimensional points, enabling the two-dimensional image and the three-dimensional LiDAR point cloud data to be semantically aligned in space.

6. The 3D scene semantic segmentation device as described in claim 5, characterized in that, The third processing module includes: The first processing unit is used to decompose 2D visual features and 3D geometric features into shared components through a learnable linear projection layer. and private components ; The second processing unit is used to process the shared components. and private components Perform shared attention fusion to obtain shared features ; The third processing unit is used to process shared features. With three-dimensional private components The features are spliced ​​together to form multimodal features.

7. A 3D scene semantic segmentation system, characterized in that, include: A memory and a processor, wherein the memory stores a computer program executed by the processor, the computer program performing the 3D scene semantic segmentation method as described in any one of claims 1-3 when executed by the processor.

8. A storage medium, characterized in that, The storage medium stores a computer program, which executes the 3D scene semantic segmentation method as described in any one of claims 1-3 when it runs.

Images