3D gaussian sputtering method based on deep feature fusion

Abstract

The application provides a 3D Gaussian sputtering method based on deep feature fusion, comprising: extracting multi-view features and monocular depth features through a multi-view Transformer network and a pre-trained monocular depth estimation model respectively; dynamically fusing the two types of features through a content-guided attention module; using a 2D U-Net network to perform depth regression on the fused features to obtain a robust depth distribution; projecting the depth distribution back to a 3D space to obtain Gaussian centers, and then using a 2D U-Net network to predict other Gaussian parameters; and rendering a high-quality three-dimensional model according to all Gaussian parameters. The 3D Gaussian sputtering method combines the complementary advantages of multi-view feature matching and monocular depth prior, significantly improves the reconstruction quality of complex scenes, and solves the problem of low scene reconstruction quality caused by inaccurate matching or missing matching information of traditional feature matching methods in complex scenes.

Description

Technical Field

[0001] This invention relates to the field of 3D Gaussian reconstruction, and in particular to a 3D Gaussian sputtering method based on deep feature fusion. Background Technology

[0002] With the rapid development of technologies such as virtual reality, autonomous driving, and digital twins, the demand for reconstructing high-quality 3D scenes from sparse multi-view images is becoming increasingly urgent. While traditional Neural Radiation Field (NeRF) methods demonstrate excellent reconstruction quality, their high computational cost and lengthy optimization time make them unsuitable for practical applications. 3D Gaussian Sputtering (3DGS), as a novel explicit 3D representation method, significantly improves rendering efficiency by modeling scene geometry and appearance using a Gaussian distribution. However, existing 3D Gaussian Sputtering methods based on multi-view feature matching suffer from inaccurate feature matching or missing feature matching information in complex scenes (such as occluded areas, textureless areas, and reflective areas), leading to insufficient robustness in depth estimation and severely impacting reconstruction quality.

[0003] Therefore, a 3D Gaussian sputtering method based on deep feature fusion is proposed to solve the problem of low scene reconstruction quality caused by inaccurate matching or missing matching information in complex scenes based on feature matching methods. Improving the robustness and reconstruction quality of depth estimation in complex scenes has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, the present invention provides a 3D Gaussian sputtering method based on deep feature fusion to solve the problems existing in the prior art.

[0005] This invention provides a 3D Gaussian sputtering method based on deep feature fusion, comprising:

[0006] S1: Acquire multi-view scene images and perform image preprocessing to obtain preprocessed multi-view scene images, wherein there is overlap between images of adjacent views;

[0007] S2: Extract features of each view from the preprocessed image through a multi-view Transformer network structure. The multi-view Transformer network structure enhances cross-view perception through the Swing Transformer Block and introduces a multi-scale dilated attention module to capture semantic information at different scales.

[0008] S3: Construct the cost volume of each view based on its characteristics;

[0009] S4: Use the pre-trained monocular depth estimation model UniDepthV2 to extract monocular depth features for each view, and align the resolution of the monocular depth features for each view with the resolution of its cost volume using bilinear interpolation.

[0010] S5: The content-guided attention module fuses the monocular depth features and cost volume of each view to obtain the fused features;

[0011] S6: Use a 2D U-Net network to perform depth regression on the fused features of each view to obtain the pixel-wise depth distribution corresponding to each view;

[0012] S7: Backproject the depth distribution of each view into 3D space to obtain the Gaussian center of each pixel in each view;

[0013] S8: Use a 2D U-Net network to predict the Gaussian transparency, Gaussian covariance matrix, and Gaussian color of each pixel in each view based on the fused features of each view;

[0014] S9: Render a 3D model using rasterization based on the Gaussian center, Gaussian transparency, Gaussian covariance matrix, and Gaussian color.

[0015] Preferably, in S1, the image preprocessing includes denoising, normalization, and cropping / scaling.

[0016] Further optimization, in S3, the cost volume C of view i i The construction method is as follows:

[0017] S31: Sample 128 depth candidate values ​​d for view i. m The candidate depth values ​​range from 0.5 to 100.

[0018] S32: Projection matrix P through camera i and the depth candidate value d m The feature F of view j adjacent to view i j Projecting the distortion feature onto the image plane of view i yields the distortion characteristic.

[0019] S33: Calculate the feature F of view i using the dot product. i With distortion features The similarity is used to construct the cost volume C. i ,in,

[0020] Further optimization, in S5, the monocular depth features of view i are... and cost volume C i The features are then fused to obtain the fused characteristics. The method is as follows:

[0021] S51: Transfer the monocular depth features With cost and volume C i By splicing along the channel dimension, we obtain The formula is as follows:

[0022] S52: Generate fusion weights through the content-guided attention module in,

[0023] σ is the Sigmoid function, GC represents group convolution, and CS represents channel shuffling;

[0024] S53: Based on fusion weights Feature fusion is performed to obtain the fused features. in,

[0025] Further optimization is made in S6, where the loss function is a linear combination of MSE and LPIPS.

[0026] Further optimization, in S6, the loss function L = L MSE +0.05L LPIPS .

[0027] Further optimization, in S7, the j-th pixel (u) in view i j v j Gaussian center μ j The calculation method is as follows:

[0028] S71: Calculate the j-th pixel (u) in view i j v j The depth value d j , where d j =D i (u j v j ), D i This represents the depth distribution of view i;

[0029] S72: Using the camera intrinsic parameter matrix K and extrinsic parameter matrix [R|t], the pixel (u j v j ) and depth value d j Calculate the Gaussian center μ j ,in,

[0030] This invention provides a 3D Gaussian sputtering method based on depth feature fusion. By combining the complementary advantages of multi-view feature matching and monocular depth prior, it significantly improves the reconstruction quality of complex scenes and solves the problem of low scene reconstruction quality caused by inaccurate matching or missing matching information in traditional feature matching methods in complex scenes. This method first extracts multi-view features and monocular depth features using a multi-view Transformer network and a pre-trained monocular depth estimation model, respectively. Then, it dynamically fuses the two types of features using a Content-Guided Attention (CGA) module. A 2D U-Net network is used to perform depth regression on the fused features to obtain a robust depth distribution. The depth distribution is back-projected into 3D space to obtain Gaussian center parameters. A lightweight 2D U-Net network is then used to predict other Gaussian parameters. Finally, a high-quality 3D model can be rendered by rasterization based on all Gaussian parameters. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart of the 3D Gaussian sputtering method based on deep feature fusion provided by the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, to avoid obscuring the invention with unnecessary details, only processing steps closely related to the solution of this invention are shown in the drawings, while other details not closely related to this invention are omitted.

[0034] like Figure 1 As shown, this invention provides a 3D Gaussian sputtering method based on deep feature fusion, comprising the following steps:

[0035] S1: Acquire multi-view scene images and perform image preprocessing to obtain preprocessed multi-view scene images, wherein there is overlap between images of adjacent views;

[0036] Among them, multi-view scene images can be acquired by drones or camera arrays, and sufficient overlap between adjacent view images is ensured;

[0037] The image preprocessing includes denoising, normalization, and cropping / scaling;

[0038] The denoising process is as follows: First, select an appropriate denoising algorithm (e.g., Gaussian filtering, median filtering, bilateral filtering) based on the type of noise in the image; then, set the parameters of the denoising algorithm (e.g., filter size, number of iterations) based on the noise level and resolution of the image; finally, use an image processing library (e.g., OpenCV or Pillow) to denoise the image.

[0039] The normalization process is as follows: The pixel values ​​of the denoised image are normalized to the range of [0,1] using an image processing library;

[0040] The cropping and scaling steps are as follows: The image is uniformly scaled to 256×256 resolution using bilinear interpolation;

[0041] S2: Extract features of each view from the preprocessed image using a multi-view Transformer network structure. The multi-view Transformer network structure enhances cross-view perception through the Swing Transformer Block and introduces a multi-scale dilated attention module (MSDA) to capture semantic information at different scales. The MSDA module can obtain richer contextual information and better capture details and structures in the image, thereby improving the model's accuracy, robustness, and generalization ability.

[0042] S3: Construct the cost volume of each view based on its characteristics;

[0043] Wherein, the cost volume C of view i i The construction method is as follows:

[0044] S31: Sample 128 depth candidate values ​​d for view i. m The candidate depth values ​​range from 0.5 to 100.

[0045] S32: Projection matrix P through camera i and the depth candidate value d m The feature F of view j adjacent to view i j Projecting the distortion feature onto the image plane of view i yields the distortion characteristic.

[0046] S33: Calculate the feature F of view i using the dot product. i With distortion features The similarity is used to construct the cost volume C. i ,Right now:

[0047] S4: Use the pre-trained monocular depth estimation model UniDepthV2 to extract monocular depth features for each view, and align the resolution of the monocular depth features for each view with the resolution of its cost volume using bilinear interpolation.

[0048] S5: The content-guided attention module (CGA) fuses the monocular depth features and cost volume of each view to obtain fused features. The fusion of monocular depth features and multi-view cost volume can effectively alleviate the limitations of feature matching-based multi-view depth estimation methods when dealing with complex scenes such as occluded areas, textureless areas, and reflective areas, improve the robustness and accuracy of depth estimation, and enhance the quality of 3D reconstruction. The CGA module can dynamically allocate weights according to the importance and relevance of features, thereby better preserving useful information, improving gradient propagation efficiency, avoiding information loss, and also solving the problem that the differences between low-level and high-level features in encoding methods and receptive fields lead to ineffective fusion.

[0049] Among them, the monocular depth features of view i and cost volume C i The features are then fused to obtain the fused characteristics. The method is as follows:

[0050] S51: Transfer the monocular depth features With cost and volume C i By splicing along the channel dimension, we obtain Right now:

[0051] S52: Generate fusion weights via the Content-Guided Attention Module (CGA) in, σ is the Sigmoid function, GC represents group convolution, and CS represents channel shuffling;

[0052] S53: Based on fusion weights Feature fusion is performed to obtain the fused features. in,

[0053] S6: Use a 2D U-Net network to perform depth regression on the fused features of each view to obtain the pixel-wise depth distribution corresponding to each view. The loss function is a linear combination of MSE and LPIPS, i.e., loss function L = L MSE +0.05L LPIPS ;

[0054] S7: Backproject the depth distribution of each view into 3D space to obtain the Gaussian center of each pixel in each view;

[0055] Wherein, the j-th pixel (u) in view i j v j Gaussian center μ j The calculation method is as follows:

[0056] S71: Calculate the j-th pixel (u) in view i j v j The depth value d j , where d j =D i (u j v j ), D i This represents the depth distribution of view i;

[0057] S72: Using the camera intrinsic parameter matrix K and extrinsic parameter matrix [R|t], the pixel (u j v j ) and depth value d j Calculate the Gaussian center μ j ,in,

[0058] S8: Use a lightweight 2D U-Net network to predict the Gaussian transparency, Gaussian covariance matrix, and Gaussian color of each pixel in each view based on the fused features of each view;

[0059] S9: Render a 3D model using rasterization based on the Gaussian center, Gaussian transparency, Gaussian covariance matrix, and Gaussian color.

[0060] This invention provides a 3D Gaussian sputtering method based on depth feature fusion. By combining the complementary advantages of multi-view feature matching and monocular depth prior, it significantly improves the reconstruction quality of complex scenes and solves the problem of low scene reconstruction quality caused by inaccurate matching or missing matching information in traditional feature matching methods in complex scenes. This method first extracts multi-view features and monocular depth features using a multi-view Transformer network and a pre-trained monocular depth estimation model, respectively. Then, it dynamically fuses the two types of features using a Content-Guided Attention (CGA) module. A 2D U-Net network is used to perform depth regression on the fused features to obtain a robust depth distribution. The depth distribution is back-projected into 3D space to obtain Gaussian center parameters. A lightweight 2D U-Net network is then used to predict other Gaussian parameters. Finally, a high-quality 3D model can be rendered by rasterization based on all Gaussian parameters.

[0061] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention; however, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A 3D Gaussian sputtering method based on deep feature fusion, characterized in that, include: S1: Acquire multi-view scene images and perform image preprocessing to obtain preprocessed multi-view scene images, wherein there is overlap between images of adjacent views; S2: Extract features of each view from the preprocessed image through a multi-view Transformer network structure. The multi-view Transformer network structure enhances cross-view perception through the Swing Transformer Block and introduces a multi-scale dilated attention module to capture semantic information at different scales. S3: Construct the cost volume of each view based on its characteristics; S4: Use the pre-trained monocular depth estimation model UniDepthV2 to extract monocular depth features for each view, and align the resolution of the monocular depth features for each view with the resolution of its cost volume using bilinear interpolation. S5: The content-guided attention module fuses the monocular depth features and cost volume of each view to obtain the fused features. The view... Monocular depth features and cost volume The features are then fused to obtain the fused characteristics. The method is as follows: S51: Transfer the monocular depth features With cost and volume By splicing along the channel dimension, we obtain The formula is as follows: ; S52: Generate fusion weights through the content-guided attention module ,in, , It is the Sigmoid function. Represents group convolution. Indicates mixed washing of channels; S53: Based on fusion weights Feature fusion is performed to obtain the fused features. ,in, ; S6: Use a 2D U-Net network to perform depth regression on the fused features of each view to obtain the pixel-wise depth distribution corresponding to each view; S7: Backproject the depth distribution of each view into 3D space to obtain the Gaussian center of each pixel in each view; S8: Use a 2D U-Net network to predict the Gaussian transparency, Gaussian covariance matrix, and Gaussian color of each pixel in each view based on the fused features of each view; S9: Render a 3D model using rasterization based on the Gaussian center, Gaussian transparency, Gaussian covariance matrix, and Gaussian color.

2. The 3D Gaussian sputtering method based on deep feature fusion according to claim 1, characterized in that: In S1, the image preprocessing includes denoising, normalization, and cropping / scaling.

3. The 3D Gaussian sputtering method based on deep feature fusion according to claim 1, characterized in that: In S3, the view Cost Volume The construction method is as follows: S31: View Sample 128 depth candidate values The candidate depth values ​​range from 0.5 to 100. S32: Projection matrix via camera and the depth candidate value , will be with view Features of adjacent view j Projected onto view On the image plane, distortion features are obtained. ; S33: Use dot product calculation view Features With distortion features The similarity is used to construct the cost volume. ,in, .

4. The 3D Gaussian sputtering method based on deep feature fusion according to claim 1, characterized in that: In S6, the loss function is a linear combination of MSE and LPIPS.

5. The 3D Gaussian sputtering method based on deep feature fusion according to claim 4, characterized in that: In S6, the loss function .

6. The 3D Gaussian sputtering method based on deep feature fusion according to claim 1, characterized in that: In S7, the view The j-th pixel Gauss Center The calculation method is as follows: S71: Calculation View The j-th pixel depth value ,in, , Represents a view Depth distribution; S72: Utilizing the camera intrinsic parameter matrix and extrinsic parameter matrix Pixels and depth value Calculate the Gaussian center ,in, .

Images