Transform and U-Net-based sparse multi-view image 3D Gaussian reconstruction method

Abstract

The invention provides a sparse multi-view image 3D Gaussian reconstruction method based on Transform and U-Net. The sparse multi-view image 3D Gaussian reconstruction method comprises the following steps: preprocessing a sparse multi-view image; performing feature extraction and fusion on the preprocessed multi-view-angle image to obtain a feature map containing multi-view-angle information; constructing a cost volume according to the feature map of each visual angle image, and performing refining and normalization to obtain probability distribution of pixels of each preprocessed visual angle image belonging to different depth candidates; sampling the probability distribution to obtain a depth map of each view angle image; predicting the probability distribution of the Gaussian parameter of each pixel by using the U-Net network and obtaining the Gaussian distribution parameter of each pixel; and generating a three-dimensional model according to the Gaussian distribution parameters of each pixel by using a 3D Gaussian drawing technology. According to the sparse multi-view image 3D Gaussian reconstruction method, the reconstruction accuracy and the reconstruction efficiency of a three-dimensional reconstruction method can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of 3D Gaussian reconstruction, and in particular to a sparse multi-view image 3D Gaussian reconstruction method based on Transformer and U-Net. Background Art

[0002] Traditional 3D reconstruction methods include two key links: depth estimation and 3D representation. However, traditional depth estimation methods are easily affected by factors such as noise and occlusion, making it difficult to ensure the accuracy of depth estimation. Traditional 3D representation methods require the use of complex data structures and a large amount of storage space, resulting in low computing speed and rendering efficiency. Due to the existence of the above problems, the reconstruction efficiency and reconstruction accuracy of traditional 3D reconstruction methods are poor, making them difficult to apply to actual scenes.

[0003] Therefore, it has become an urgent problem to propose a sparse multi-view image 3D Gaussian reconstruction method based on Transformer and U-Net to improve the reconstruction accuracy and efficiency of the three-dimensional reconstruction method. Summary of the invention

[0004] In view of this, the present invention provides a sparse multi-view image 3D Gaussian reconstruction method based on Transformer and U-Net to solve the problems existing in the prior art.

[0005] The present invention provides a sparse multi-view image 3D Gaussian reconstruction method based on Transformer and U-Net, comprising:

[0006] S1: Collect sparse multi-view images and perform image preprocessing to obtain preprocessed multi-view images;

[0007] S2: Use ResNet CNN to extract the features of each preprocessed view image and downsample it to obtain a low-resolution feature map of each view image;

[0008] S3: Use the multi-view Transformer to extract and fuse the features in the feature map of each view image to obtain a feature map containing multi-view information;

[0009] S4: Construct a cost volume based on the feature map of each view image;

[0010] S5: using a U-Net network to refine the cost volume to obtain a cost volume with higher accuracy;

[0011] S6: Use the softmax operation to normalize the refined cost volume to obtain the probability distribution of each pixel of the preprocessed view image belonging to different depth candidates;

[0012] S7: sampling the probability distribution obtained in S6 to obtain the preprocessed depth value of each pixel of each perspective image and the depth map of each perspective image;

[0013] S8: Use the U-Net network to predict the probability distribution of the Gaussian parameters of each pixel, wherein the input of the U-Net network includes the depth map of each perspective image, the feature map containing multi-perspective information obtained in S3, and each perspective image after preprocessing, and the output of the U-Net network is the probability distribution of the Gaussian parameters of each pixel;

[0014] S9: sampling the probability distribution obtained in S8 to obtain Gaussian distribution parameters of each pixel;

[0015] S10: Generate a three-dimensional model based on the Gaussian distribution parameters of each pixel using 3D Gaussian rendering technology.

[0016] Preferably, in S1, the image preprocessing includes denoising, normalization, and cropping and scaling.

[0017] Further preferably, S3 includes:

[0018] S31: Input the feature map of each view image into the multi-view Transformer, where the multi-view Transformer includes a self-attention mechanism, a cross-attention mechanism, and a multi-scale expansion attention mechanism, and extracts the feature relationship within each view image through the self-attention mechanism, extracts the feature relationship between images of different views through the cross-attention mechanism, and extracts global and local feature information through the multi-scale expansion attention mechanism;

[0019] S32: All feature relationships and feature information extracted by the above attention mechanism are integrated to obtain a feature map containing multi-view information. Finally, the output features of the Transformer are upsampled to restore the image resolution.

[0020] Further preferably, S4 includes:

[0021] S41: Calculate the similarity information of each perspective using the feature map of each perspective image, wherein the method for obtaining the similarity information of any perspective A is as follows:

[0022] S411: taking the feature map of the viewing angle A as the feature map of the current viewing angle, and the feature maps of the other viewing angles as the feature maps of other viewing angles, and then projecting and distorting the feature maps of the other viewing angles by plane scanning so as to align them with the feature map of the current viewing angle;

[0023] S412: Calculate the similarity between the current viewing angle feature map and the distorted other viewing angle feature maps to obtain the similarity information of the current viewing angle;

[0024] S42: Store the similarity information of all perspectives in a cost volume of dimension H×W×D, where H and W represent the height and width of the image, respectively, and D represents the number of depth candidates. Each element in the cost volume represents the similarity of each pixel in the image under different depth candidates.

[0025] Further preferably, in S42, the method for determining the number of depth candidates is as follows:

[0026] Determine the sampling range of depth candidates according to the near and far depth values;

[0027] A series of depth candidates are uniformly sampled within the sampling range of the depth candidates, and the sampled depth candidates are converted into inverse depths to avoid numerical stability problems caused by too large depth values.

[0028] Further preferably, in S5, the U-Net network includes a depthwise separable convolution, a SENet module, a residual connection and a skip connection, wherein the depthwise separable convolution is used to reduce parameters while ensuring no loss of accuracy, the residual connection and the skip connection are used to effectively fuse features of different layers, and the SENet module is used to automatically learn the importance of different feature channels.

[0029] Further preferably, in S9, the Gaussian distribution parameters include position, covariance, opacity and color

[0030] The sparse multi-view image 3D Gaussian reconstruction method based on Transformer and U-Net provided by the present invention uses an improved multi-view Transformer, Unet and cost volume representation to effectively improve the robustness of the model to interference such as noise and occlusion, thereby performing stable reconstruction in complex scenes and improving the quality of the reconstructed model; uses 3D Gaussian distribution to represent feature points in three-dimensional space, and then renders the 3D model, thereby achieving efficient rendering, improving rendering speed and reducing computing cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 It is a flowchart of the sparse multi-view image 3D Gaussian reconstruction method based on Transformer and U-Net provided by the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in order to avoid blurring the present invention due to unnecessary details, only the processing steps closely related to the scheme of the present invention are shown in the accompanying drawings, and other details that are not closely related to the present invention are omitted.

[0034] like Figure 1 As shown, the present invention provides a sparse multi-view image 3D Gaussian reconstruction method based on Transformer and U-Net, comprising the following steps:

[0035] S1: Collect sparse multi-view images and perform image preprocessing to obtain preprocessed multi-view images;

[0036] The sparse multi-view images are collected by a camera, specifically: a suitable camera device (such as a mobile phone camera, a SLR camera or a drone camera) and a shooting strategy (such as selecting a suitable shooting distance, angle and camera motion trajectory) are selected to collect multi-view images, wherein the multi-view images include images of at least two perspectives;

[0037] The image preprocessing includes denoising, normalization, and cropping and scaling;

[0038] The denoising process is as follows: first, according to the noise type of the image, select a suitable denoising algorithm (such as Gaussian filtering, median filtering, bilateral filtering); then, according to the noise level and resolution of the image, set the parameters of the denoising algorithm (such as filter size, number of iterations); finally, use an image processing library (such as OpenCV and Pillow) to denoise the image;

[0039] The normalization process is as follows: the pixel values ​​of the denoised image are normalized to the range of [0,1] using the image processing library;

[0040] The steps of cropping and scaling are as follows: Use the image processing library to crop and scale the normalized image to a fixed size, such as 256×256 pixels;

[0041] S2: Use ResNet CNN to extract the features of each preprocessed view image and downsample it to obtain a low-resolution feature map of each view image;

[0042] S3: Use the multi-view Transformer to extract and fuse the features in the feature map of each view image to obtain a feature map containing multi-view information;

[0043] The specific steps are as follows:

[0044] S31: Input the feature map of each view image into the multi-view Transformer, where the multi-view Transformer includes a self-attention mechanism, a cross-attention mechanism, and a multi-scale expansion attention mechanism, and extracts the feature relationship within each view image through the self-attention mechanism, extracts the feature relationship between images of different views through the cross-attention mechanism, and extracts global and local feature information through the multi-scale expansion attention mechanism;

[0045] S32: Fuse all feature relationships and feature information extracted by the above attention mechanism to obtain a feature map containing multi-view information, and finally upsample the output features of the Transformer to restore the image resolution;

[0046] S4: Construct a cost volume based on the feature map of each view image;

[0047] The specific steps are as follows:

[0048] S41: Calculate the similarity information of each perspective using the feature map of each perspective image, wherein the method for obtaining the similarity information of any perspective A is as follows:

[0049] S411: taking the feature map of the viewing angle A as the feature map of the current viewing angle, and the feature maps of the other viewing angles as the feature maps of other viewing angles, and then projecting and distorting the feature maps of the other viewing angles by plane scanning so as to align them with the feature map of the current viewing angle;

[0050] S412: Calculate the similarity between the current view feature map and the distorted feature maps of other view points to obtain similarity information of the current view point, wherein the similarity can be measured using a dot product or cosine similarity method, and the similarity indicates the degree of matching between features of different view points. The higher the similarity, the more likely it is that the two features correspond to the same point in the scene.

[0051] S42: Store the similarity information of all perspectives in a cost volume of dimension H×W×D, where H and W represent the height and width of the image, respectively, and D represents the number of depth candidates. Each element in the cost volume represents the similarity of each pixel in the image under different depth candidates, which is used to provide an important geometric basis for depth estimation. The method for determining the number of depth candidates is as follows:

[0052] Determine the sampling range of depth candidates according to the near and far depth values;

[0053] Uniformly sample a series of depth candidates within the sampling range of the depth candidates, and convert the sampled depth candidates into inverse depths to avoid numerical stability problems caused by excessively large depth values, wherein the sampling of the depth candidates can be performed using linear or logarithmic space;

[0054] S5: using a U-Net network to refine the cost volume to obtain a cost volume with higher accuracy;

[0055] Among them, the U-Net network includes deep separable convolution, SENet module, residual connection and skip connection. Deep separable convolution can reduce parameters while ensuring accuracy without loss. Residual connection and skip connection can effectively fuse features of different layers to improve reconstruction quality. SENet module can automatically learn the importance of different feature channels, so as to highlight useful features and suppress unimportant features. Through the above means, the cost volume can be refined to obtain a cost volume with higher accuracy.

[0056] S6: Use the softmax operation to normalize the refined cost volume to obtain the probability distribution of each pixel of the preprocessed view image belonging to different depth candidates;

[0057] S7: sampling the probability distribution obtained in S6 to obtain the preprocessed depth value of each pixel of each perspective image and the depth map of each perspective image;

[0058] S8: Use the U-Net network to predict the probability distribution of the Gaussian parameters of each pixel, wherein the input of the U-Net network includes the depth map of each perspective image, the feature map containing multi-perspective information obtained in S3, and each perspective image after preprocessing, and the output of the U-Net network is the probability distribution of the Gaussian parameters of each pixel;

[0059] S9: sampling the probability distribution obtained in S8 to obtain Gaussian distribution parameters of each pixel, wherein the Gaussian distribution parameters include position, covariance, opacity and color;

[0060] S10: Generate a three-dimensional model based on the Gaussian distribution parameters of each pixel using 3D Gaussian rendering technology.

[0061] The sparse multi-view image 3D Gaussian reconstruction method based on Transformer and U-Net provided by the present invention improves the reconstruction quality by fusing image features of multiple viewpoints and utilizing complementary information between different viewpoints. The similarity information between multi-view features is stored by cost volume representation, providing reliable geometric clues for depth estimation and improving the accuracy of depth estimation. A three-dimensional model is generated according to the Gaussian distribution parameters of each pixel through 3D Gaussian rendering technology.

[0062] It should be noted that the purpose of publishing the embodiments is to help further understand the present invention, but those skilled in the art can understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be subject to the scope defined in the claims.

Claims

1. A sparse multi-view image 3D Gaussian reconstruction method based on Transformer and U-Net, characterized in that: include: S1: Collect sparse multi-view images and perform image preprocessing to obtain preprocessed multi-view images; S2: Use ResNet CNN to extract the features of each preprocessed view image and downsample it to obtain a low-resolution feature map of each view image; S3: Use the multi-view Transformer to extract and fuse the features in the feature map of each view image to obtain a feature map containing multi-view information; S4: Construct a cost volume based on the feature map of each view image; S5: using a U-Net network to refine the cost volume to obtain a cost volume with higher accuracy; S6: Use the softmax operation to normalize the refined cost volume to obtain the probability distribution of each pixel of the preprocessed view image belonging to different depth candidates; S7: sampling the probability distribution obtained in S6 to obtain the preprocessed depth value of each pixel of each perspective image and the depth map of each perspective image; S8: Use the U-Net network to predict the probability distribution of the Gaussian parameters of each pixel, wherein the input of the U-Net network includes the depth map of each perspective image, the feature map containing multi-perspective information obtained in S3, and each perspective image after preprocessing, and the output of the U-Net network is the probability distribution of the Gaussian parameters of each pixel; S9: sampling the probability distribution obtained in S8 to obtain Gaussian distribution parameters of each pixel; S10: Generate a three-dimensional model based on the Gaussian distribution parameters of each pixel using 3D Gaussian rendering technology.

2. The sparse multi-view image 3D Gaussian reconstruction method based on Transformer and U-Net according to claim 1, characterized in that: In S1, the image preprocessing includes denoising, normalization, and cropping and scaling.

3. The sparse multi-view image 3D Gaussian reconstruction method based on Transformer and U-Net according to claim 1, characterized in that: S3 includes: S31: Input the feature map of each view image into the multi-view Transformer, where the multi-view Transformer includes a self-attention mechanism, a cross-attention mechanism, and a multi-scale expansion attention mechanism, and extracts the feature relationship within each view image through the self-attention mechanism, extracts the feature relationship between images of different views through the cross-attention mechanism, and extracts global and local feature information through the multi-scale expansion attention mechanism; S32: All feature relationships and feature information extracted by the above attention mechanism are integrated to obtain a feature map containing multi-view information. Finally, the output features of the Transformer are upsampled to restore the image resolution.

4. The sparse multi-view image 3D Gaussian reconstruction method based on Transformer and U-Net according to claim 1, characterized in that: S4 include: S41: Calculate the similarity information of each perspective using the feature map of each perspective image, wherein the method for obtaining the similarity information of any perspective A is as follows: S411: taking the feature map of the viewing angle A as the feature map of the current viewing angle, and the feature maps of the other viewing angles as the feature maps of other viewing angles, and then projecting and distorting the feature maps of the other viewing angles by plane scanning so as to align them with the feature map of the current viewing angle; S412: Calculate the similarity between the current viewing angle feature map and the distorted other viewing angle feature maps to obtain the similarity information of the current viewing angle; S42: Store the similarity information of all perspectives in a cost volume of dimension H×W×D, where H and W represent the height and width of the image, respectively, and D represents the number of depth candidates. Each element in the cost volume represents the similarity of each pixel in the image under different depth candidates.

5. The sparse multi-view image 3D Gaussian reconstruction method based on Transformer and U-Net according to claim 4, characterized in that: In S42, the method for determining the number of depth candidates is as follows: Determine the sampling range of depth candidates according to the near and far depth values; A series of depth candidates are uniformly sampled within the sampling range of the depth candidates, and the sampled depth candidates are converted into inverse depths to avoid numerical stability problems caused by too large depth values.

6. The sparse multi-view image 3D Gaussian reconstruction method based on Transformer and U-Net according to claim 1, characterized in that: In S5, the U-Net network includes deep separable convolution, SENet module, residual connection and skip connection. The deep separable convolution is used to reduce parameters while ensuring accuracy without loss. The residual connection and skip connection are used to effectively fuse features of different layers. The SENet module is used to automatically learn the importance of different feature channels.

7. The sparse multi-view image 3D Gaussian reconstruction method based on Transformer and U-Net according to claim 1, characterized in that: In S9, the Gaussian distribution parameters include position, covariance, opacity and color.

Images