Multimodal 3D instance segmentation method based on 3D Gaussian splatting

Abstract

The present invention relates to the field of computer vision technology, and discloses a multimodal three-dimensional instance segmentation method based on three-dimensional Gaussian splashing, comprising the steps of: generating a sparse three-dimensional point cloud by reconstructing an RGB image through a motion structure; optimizing the sparse three-dimensional point cloud to generate a three-dimensional Gaussian splashing distribution; fusing the two-dimensional features of the RGB image and the three-dimensional features of the three-dimensional Gaussian splashing distribution through a multimodal three-dimensional instance segmentation model to obtain a fused feature; extracting the two-dimensional features in the RGB image through a two-dimensional U-Net network, and extracting the three-dimensional features in the three-dimensional Gaussian splashing distribution through a sparse 3D U-Net network; fusing the two-dimensional features and the three-dimensional features through a cross-attention module of the multimodal three-dimensional instance segmentation model, and finally outputting the category label, mask and confidence score of the segmented instance. The present invention introduces the three-dimensional Gaussian splashing technology, combines the multimodal neural network with the shared weight, and improves the accuracy and efficiency of three-dimensional instance segmentation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multimodal three-dimensional instance segmentation method based on three-dimensional Gaussian splashing. Background Art

[0002] With the rapid development of artificial intelligence and autonomous driving technologies, environmental perception technology plays a vital role in many fields. Especially in application fields such as autonomous driving, drone navigation, and robot operation, the fine perception of the environment not only determines the decision-making efficiency of the system, but also directly affects its safety and reliability. Therefore, 3D instance segmentation technology has gradually become the focus of computer vision research. It can accurately identify and segment different targets in three-dimensional space, providing strong support for the understanding of complex scenes.

[0003] The core challenge of 3D instance segmentation is how to maintain accurate segmentation of target instances while efficiently processing large amounts of 3D data. Although traditional 3D instance segmentation methods have improved segmentation accuracy to a certain extent, they generally rely on a large amount of 3D point cloud data. This high dependence on data greatly limits the practicality of these methods, especially in practical applications where data annotation is expensive and it is difficult to obtain large-scale annotated data. In addition, the sparsity and irregularity of 3D point cloud data also increase the complexity of processing, further limiting the application of these methods in scenarios with limited real-time and computing resources. Summary of the invention

[0004] The purpose of the present invention is to introduce three-dimensional Gaussian splashing technology, combine it with multimodal neural network shared weights, and provide a multimodal three-dimensional instance segmentation method based on three-dimensional Gaussian splashing to improve the accuracy and efficiency of three-dimensional instance segmentation.

[0005] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:

[0006] The multimodal 3D instance segmentation method based on 3D Gaussian splashing includes the following steps:

[0007] Step 1: Based on the RGB image with two-dimensional instance labels, a sparse three-dimensional point cloud is generated through motion structure reconstruction; and the sparse three-dimensional point cloud is optimized through Voronoi diagram and adaptive density control to generate a three-dimensional Gaussian splash distribution;

[0008] Step 2: fusing the two-dimensional features of the RGB image and the three-dimensional features of the three-dimensional Gaussian splash distribution through a multimodal three-dimensional instance segmentation model to obtain a fused feature; the multimodal three-dimensional instance segmentation model includes a two-dimensional U-Net network with shared weights and a sparse 3D U-Net network, extracting the two-dimensional features in the RGB image through the two-dimensional U-Net network, and extracting the three-dimensional features in the three-dimensional Gaussian splash distribution through the sparse 3D U-Net network;

[0009] Step 3: The 2D features and 3D features are fused through the cross-attention module of the multimodal 3D instance segmentation model, and finally the category label, mask and confidence score of the segmented instance are output.

[0010] Compared with the prior art, the present invention has the following beneficial effects: the present invention integrates three-dimensional Gaussian splatter distribution (3DGS) with a multimodal neural network (and a two-dimensional U-Net network and a sparse 3D U-Net network). The framework does not require complex point cloud annotations and can effectively convert two-dimensional input into three-dimensional instance segmentation results, thereby achieving efficient and accurate three-dimensional scene understanding.

[0011] The present invention first uses structure-from-motion (SfM) technology to generate sparse 3D point clouds from multi-view RGB images with 2D instance labels, and then densifies these sparse 3D point clouds through Voronoi diagrams and adaptive density control, and initializes them to 3D Gaussian splash distribution. Then the weights of the 2D U-Net network and the sparse 3D U-Net network are shared to enhance the learning process and ensure the consistency and efficiency of feature learning. It also combines advanced attention mechanisms such as cross-attention mechanism and multi-head attention mechanism. The processed features are then passed to the prediction head module, which outputs category labels, masks and confidence scores for the segmented instances. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 It is a schematic diagram of the process of the present invention;

[0014] Figure 2 It is a schematic diagram of some modules of the multimodal 3D instance segmentation model of the present invention. DETAILED DESCRIPTION

[0015] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0016] It should be noted that similar numbers and letters represent similar items in the following figures, so once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance, or implying any such actual relationship or order between these entities or operations. In addition, the terms "connected", "connected", etc. can be directly connected between elements, or indirectly connected via other elements.

[0017] Embodiment 1:

[0018] The present invention is achieved through the following technical solutions: Figure 1 As shown, the multimodal 3D instance segmentation method based on 3D Gaussian splashing includes the following steps:

[0019] Step 1: Based on the RGB image with two-dimensional instance labels, a sparse three-dimensional point cloud is generated through motion structure reconstruction; and the sparse three-dimensional point cloud is optimized through Voronoi diagram and adaptive density control to generate a three-dimensional Gaussian splash distribution.

[0020] By processing RGB images with the SAM (Segment Anything Model) technique, 2D instance labels of RGB images can be generated. A set of RGB images with 2D instance labels are analyzed by the Structure from Motion (SfM) technique to reconstruct the 3D structure of the scene and generate a sparse 3D point cloud. Each point has an associated color. In the process of transferring the 2D instance label to the 3D domain, the color of each point is extracted from the RGB image using a projection matrix through a differentiable tile rasterizer. This projection can project the 3D point into the 2D coordinates of the RGB image. According to the color matching, the 2D instance label of the RGB image is assigned to the corresponding 3D Gaussian. Each 3D Gaussian is colored using the 2D instance label, thereby effectively transferring the 2D instance label to the 3D point cloud. This 2D to 3D label transfer ensures that the semantic information of the 2D domain is accurately reflected in the 3D representation, solves the dependence on large-scale 3D point cloud data annotation, does not require labor-intensive point cloud annotation, and can also improve the accuracy and efficiency of 3D instance segmentation tasks, achieving robust and accurate 3D scene understanding.

[0021] To improve the density and uniformity of the point cloud, more points are iteratively sampled from the sparse 3D point cloud, and the points are optimized and densified through Voronoi densification and adaptive density control. This process divides the sparse 3D point cloud into different regions in space, and samples additional points in each region based on the volume and existing point density, thereby improving the density and uniformity of the point cloud. The densified points are initialized to a 3D Gaussian splatter distribution, and each point is represented by its average position and geometric distribution.

[0022] Step 2: Fusing the two-dimensional features of the RGB image and the three-dimensional features of the three-dimensional Gaussian splash distribution through a multimodal three-dimensional instance segmentation model to obtain fused features; the multimodal three-dimensional instance segmentation model includes a two-dimensional U-Net network with shared weights and a sparse 3D U-Net network, extracting the two-dimensional features in the RGB image through the two-dimensional U-Net network, and extracting the three-dimensional features in the three-dimensional Gaussian splash distribution through the sparse 3D U-Net network.

[0023] See also Figure 1 The multimodal 3D instance segmentation model proposed in this scheme is an architecture that extracts and fuses 2D features and 3D features, and uses shared weights to enhance the learning process of 2D extraction and 3D extraction. The multimodal 3D instance segmentation model includes a 2D U-Net network, a sparse 3D U-Net network, a cross attention module, a multi-head attention module, and a prediction head module. The 2D U-Net network extracts 2D features F from the original RGB image. 2d;Sparse 3D U-Net network extracts 3D features F from 3D Gaussian splash distribution 3d .

[0024] The innovation of this solution is to achieve weight sharing learning between the 2D U-Net network and the sparse 3D U-Net network through the weight sharing lock mechanism, thereby improving the learning efficiency and effect of the model. The purpose of implementing the weight sharing lock mechanism is to ensure that the two networks can use the features and representations learned from each other, so as to make the cross-modal learning process more coherent. The core idea is to let the two networks share the weights of certain layers, especially those used to capture basic spatial features, so as to reduce redundancy and improve the ability of feature extraction.

[0025] The weight matrices of the 2D U-Net network and the sparse 3D U-Net network are represented as W 2D and W 3D , specify some layers as shared layers, and define the shared weights of the shared layers as W shared , shared weight W shared Initialized and updated via backpropagation during training:

[0026]

[0027]

[0028] in, represents the output feature map of the lth shared layer in the two-dimensional U-Net network; f 2D Represents the activation function used in the two-dimensional U-Net network; represents the shared weight of the lth shared layer; Represents the input feature map of the l-1th layer in the two-dimensional U-Net network; Represents the bias term of the lth layer in the two-dimensional U-Net network; represents the output feature map of the lth shared layer in the sparse 3D U-Net network; f 3D Represents the activation function used in the sparse 3D U-Net network; Represents the input feature map of the l-1th layer in the 3D U-Net network; Represents the bias term of the lth layer in the sparse 3D U-Net network.

[0029] The back-propagation process updates the shared weights W according to the combined gradients of the two networks. shared :

[0030]

[0031] in, Represents the shared weights updated according to the comprehensive gradient of the 2D U-Net network and the sparse 3D U-Net network; , The loss functions of the 2D U-Net network and the sparse 3D U-Net network are respectively. By sharing weights, the 2D and 3D networks can jointly learn spatial features, thereby improving the ability to capture complex structures in 2D images and 3D point clouds. This approach minimizes the redundancy of feature extraction by ensuring that the two networks use a common set of parameters for basic spatial features, thereby building a more compact and efficient model. Shared weights can train two networks simultaneously, thereby reducing the overall training time and the required computing resources. The shared weight lock mechanism improves the consistency and accuracy of the output of the multimodal 3D instance segmentation model, thereby achieving excellent 3D instance segmentation performance.

[0032] Step 3: The 2D features and 3D features are fused through the cross-attention module of the multimodal 3D instance segmentation model, and finally the category label, mask and confidence score of the segmented instance are output.

[0033] The two-dimensional feature F is extracted 2D and the three-dimensional feature F 3D Finally, the two features need to be fused to effectively combine 2D and 3D data for instance segmentation. In order to improve segmentation performance, the multimodal 3D instance segmentation model introduces cross-attention mechanism and multi-head attention mechanism to establish a strong connection between 2D features and 3D features, so that the system can more accurately identify and segment target objects when processing complex scenes. These features are integrated through cross-attention mechanism and multi-head attention mechanism, so that 2D features and 3D features can complement and combine together to complete the initial fusion.

[0034] At the same time, the Open3D voxelization method is used to obtain super-point features from the three-dimensional Gaussian splash distribution, and the point cloud is divided into super-points based on geometric and spatial criteria. The super-point features can efficiently represent the point cloud structure, laying a solid foundation for subsequent instance segmentation tasks. The multimodal three-dimensional instance segmentation model also further captures the three-dimensional features through the self-attention mechanism F 3D The dependency between the super-point features enhances the overall feature representation.

[0035] See also Figure 2 In addition to the cross-attention module, the multi-head attention module, and the prediction head module, the multimodal three-dimensional instance segmentation model also includes a self-attention module, and the self-attention module includes a self-attention layer, a first residual convolution layer, a first normalization layer, a first feedforward network, a second residual convolution layer, a second normalization layer, a super-point cross-attention layer, a third residual convolution layer, and a third normalization layer.

[0036] Based on the self-attention mechanism, the 3D features are input into the self-attention layer as keys (K) and values ​​(Q), and the query vector (Q) is also input into the self-attention layer. The query vector (Q) is used to represent the features for performing retrieval operations in multimodal features, similar to the key for querying in a database. Specifically, the query vector (Q) helps the model find relevant instances in 3D point clouds or image features by extracting and encoding key features, and generates accurate segmentation or classification results in the multimodal feature fusion process. The self-attention layer captures the global relationship between input features and enhances the representation ability of features, thereby extracting features with global context information; the first residual convolution layer provides jump connections to avoid gradient disappearance and maintain the stability of the network; the first normalization layer standardizes the features to make the network converge faster and the training stable; the first feedforward network provides nonlinear feature conversion to further map the global features extracted by the self-attention layer to improve the network's expression ability; the second residual convolution layer introduces the output of the first feedforward network into the information of the previous layer through residual connections, thereby enhancing the deep feature expression ability of the model; the second normalization layer standardizes the features to make the network converge faster and the training stable; the super point cross attention layer converts the three-dimensional feature F 3D The information interaction and fusion between the super-point features can effectively enhance the correlation of multimodal features through the cross-attention mechanism, ensuring that the interaction between two-dimensional features and three-dimensional features is more sufficient; the third residual convolution layer fuses the output of the second normalization layer with the output of the super-point cross-attention layer to form the final multimodal feature representation, and the residual connection further optimizes the feature transmission, making the multimodal features more robust and consistent in the fused three-dimensional segmentation; the third normalization layer standardizes the final multimodal features to make the network converge faster and the training more stable.

[0037] Then the multimodal features output by the self-attention module are combined with the two-dimensional features F 2D The features are input into the cross-attention module and the multi-head attention module for feature fusion to capture the interactions between various features. The cross-attention mechanism and the multi-head attention mechanism can establish a strong connection between two-dimensional features and three-dimensional features, enabling the system to more accurately identify and segment target objects when processing complex scenes. In addition, the multi-head attention mechanism allows the model to focus on multiple feature dimensions at the same time, thereby capturing more subtle and complex feature relationships and improving the overall segmentation performance of the system.

[0038] In the 2D and 3D feature fusion stage, point cloud descriptors of fused features, including position, density, color, and covariance matrix parameters, are generated. These point cloud descriptors are converted into high-dimensional feature vectors using the transformer model to facilitate effective neural network processing. The transformer model captures local and global features by utilizing a self-attention mechanism to ensure a robust high-dimensional representation for downstream tasks. This approach provides rich features for accurate and detailed 3D instance segmentation.

[0039] The second feed-forward network before the prediction head module provides nonlinear feature conversion, further maps the fused features extracted by the multi-head attention module, and improves the network's expressiveness. The prediction head module of the multimodal 3D instance segmentation model can output the category label, mask, and confidence score of the segmentation instance from the fused features through a multi-task learning strategy, thereby improving the efficiency and performance of the multimodal 3D instance segmentation model. The prediction head module generates a confidence score for each point, which helps to evaluate the accuracy of the multimodal 3D instance segmentation model.

[0040] In order to optimize the parameters of the multimodal 3D instance segmentation model and ensure accurate 3D reconstruction and instance segmentation, a comprehensive loss function is constructed:

[0041]

[0042] in, represents the reconstruction loss function; represents the classification loss function; represents the mask loss function; represents the structural similarity index loss function; are weights respectively.

[0043] The reconstruction loss function ensures the geometric accuracy of the 3D reconstruction by calculating the difference between the predicted ellipsoid parameters and the true ellipsoid parameters:

[0044]

[0045] in, They represent the predicted position, predicted density and predicted covariance matrix of the i-th point respectively; i=1,2,...,N, where N represents the number of points in the three-dimensional Gaussian splash distribution; They represent the true position, true density and true covariance matrix of the i-th point respectively.

[0046] The classification loss function helps ensure the accuracy of the predicted class labels:

[0047]

[0048] Among them, y icIndicates that the i-th point belongs to the ground truth label of the c-th category; c = 1, 2, ..., C, C represents the number of category labels; p ic It represents the predicted probability that the i-th point belongs to the c-th class.

[0049] The mask loss function ensures the accuracy of the predicted mask:

[0050]

[0051] Among them, m i represents the ground truth mask of the i-th point; Represents the predicted mask of the i-th point.

[0052] The structural similarity index loss function is used to maintain the geometric consistency and structural similarity between the predicted 3D structure and the ground truth 3D structure:

[0053]

[0054] Where SSIM represents the structural similarity index; V represents the three-dimensional volume of the ground truth; represents the predicted 3D volume.

[0055] These loss functions jointly guide the training process, adjust the parameters of the multimodal 3D instance segmentation model through back-propagation, improve the accuracy and fidelity of 3D reconstruction, and ultimately improve the instance segmentation results.

[0056] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multimodal 3D instance segmentation method based on 3D Gaussian splashing, characterized by: The following steps are involved: Step 1: Based on the RGB image with two-dimensional instance labels, a sparse three-dimensional point cloud is generated through motion structure reconstruction; and the sparse three-dimensional point cloud is optimized through Voronoi diagram and adaptive density control to generate a three-dimensional Gaussian splash distribution; Step 2, fusing the two-dimensional features of the RGB image and the three-dimensional features of the three-dimensional Gaussian splash distribution through a multimodal three-dimensional instance segmentation model to obtain a fused feature; The multimodal 3D instance segmentation model includes a weight-sharing 2D U-Net network and a sparse 3D U-Net network, wherein the 2D features in the RGB image are extracted through the 2D U-Net network, and the 3D features in the 3D Gaussian splash distribution are extracted through the sparse 3D U-Net network; While extracting 3D features, the voxelization method of Open3D is used to obtain super point features from the 3D Gaussian splash distribution; The multimodal three-dimensional instance segmentation model also includes a self-attention module; the self-attention module includes a self-attention layer, a first residual convolution layer, a first normalization layer, a first feedforward network, a second residual convolution layer, a second normalization layer, a super point cross attention layer, a third residual convolution layer, and a third normalization layer connected in sequence; Based on the self-attention mechanism, the three-dimensional feature F 3D The query vector is input into the self-attention layer as the key and value. The self-attention layer captures the global relationship between the input features and extracts features with global context information. The first residual convolution layer provides skip connections to avoid gradient disappearance. The first normalization layer normalizes the features. The first feedforward network provides nonlinear feature conversion to further map the global features extracted by the self-attention layer; the second residual convolution layer introduces the output of the first feedforward network into the information of the previous layer through the residual connection; the second normalization layer standardizes the features; The super point cross attention layer transforms the three-dimensional feature F 3D The third residual convolution layer fuses the output of the second normalization layer with the output of the super-point cross-attention layer to form the final multimodal feature representation; the third normalization layer standardizes the final multimodal features; Step 3: The 2D features and 3D features are fused through the cross-attention module of the multimodal 3D instance segmentation model, and finally the category label, mask and confidence score of the segmented instance are output; The step 3 specifically includes the following steps: the multimodal 3D instance segmentation model also includes a multi-head attention module, a second feedforward network, and a prediction head module; Multimodal features and two-dimensional features F output by the self-attention module 2D Input the cross attention module and the multi-head attention module together for feature fusion to obtain fused features; The second feedforward network provides nonlinear feature transformation and further maps the fused features extracted by the multi-head attention module; The prediction head module outputs the class label, mask, and confidence score of the segmented instance from the fused features.

2. The multimodal 3D instance segmentation method based on 3D Gaussian splatting according to claim 1, characterized in that: In step 2, the weight matrices of the two-dimensional U-Net network and the sparse 3D U-Net network are represented as W 2D and W 3D , specify multiple shared layers, and define the shared weights of the shared layers as W shared , shared weight W shared Initialization is done via backpropagation during training: in, represents the output feature map of the lth shared layer in the two-dimensional U-Net network; f 2D Represents the activation function used in the two-dimensional U-Net network; represents the shared weight of the lth shared layer; Represents the input feature map of the l-1th layer in the two-dimensional U-Net network; Represents the bias term of the lth layer in the two-dimensional U-Net network; represents the output feature map of the lth shared layer in the sparse 3D U-Net network; f 3D Represents the activation function used in the sparse 3D U-Net network; Represents the input feature map of the l-1th layer in the 3D U-Net network; Represents the bias term of the lth layer in the sparse 3D U-Net network; The back propagation process updates the shared weights W according to the comprehensive gradient of the 2D U-Net network and the sparse 3D U-Net network. shared : in, Represents the shared weights updated according to the comprehensive gradient of the 2D U-Net network and the sparse 3D U-Net network; , They are the loss functions of the two-dimensional U-Net network and the sparse 3D U-Net network respectively.

3. The multimodal 3D instance segmentation method based on 3D Gaussian splatting according to claim 1, characterized in that: The comprehensive loss function of the multimodal 3D instance segmentation model is: in, represents the reconstruction loss function; represents the classification loss function; represents the mask loss function; represents the structural similarity index loss function; are weights respectively.

Images