Three-dimensional reconstruction method based on improved TGS network

Abstract

The invention provides a three-dimensional reconstruction method based on an improved TGS network, and relates to the technical field of computer three-dimensional reconstruction, G-TGS introduces a GAN (Generative Adversarial Network) into TGS, geometric features are better generated by using a discriminator in the GAN, and the problem of reconstruction accuracy is solved; a DilateFormer module is introduced to replace an image encoder in the TGS to obtain image features, a better balance is sought between the calculation complexity and the size of a participated receptive field, and the reconstruction precision is improved; after gauss parameter information is obtained, pruning operation is carried out by calculating global significance scores, redundant gauss are removed, and memory usage is reduced.

Description

Technical Field

[0001] The invention relates to the technical field of computer three-dimensional reconstruction, and in particular to a three-dimensional reconstruction method based on an improved TGS network. Background Art

[0002] With the rapid development of computer vision, graphics, and machine learning, this technology not only plays a key role in games and movies, bringing immersive experiences to audiences, but also demonstrates its powerful practicality in applications such as simultaneous positioning and mapping, industrial manufacturing, and environmental monitoring. In addition, 3D reconstruction technology can create a new perspective from existing data, helping people observe and analyze objects from different angles, and providing more intuitive and comprehensive information.

[0003] In the field of 3D reconstruction, various technologies have emerged one after another, and a variety of 3D model generation strategies have been constructed. Initially, 3D reconstruction mainly relied on traditional methods such as feature geometry. Over time, the introduction of deep learning technology has greatly promoted the development of 3D reconstruction, mainly including methods based on convolutional neural networks (CNN), Transformer-based methods, large reconstruction models (LRM)-based methods, Diffusion-based methods, and 3DGS-based methods, making it more efficient and accurate to recover 3D models from images. Based on the traditional method of feature geometry, Hamzah et al. proposed a new high-precision stereo matching algorithm for 3D surface reconstruction in "Improvement of stereo matching algorithm for 3D surface reconstruction". The matching cost calculation uses a fixed window gradient-based matching technology, and then introduces adaptive support weights based on iterative guided filters to improve the edges of object matching during cost aggregation. In the final stage, a joint weighted guided filter is used to reduce the residual noise on the disparity map. Liu et al. proposed to use the 3D geometry of the scene to synthesize new views in "Geometry-aware deep network for single-image novel view synthesis". Specifically, a fixed number of planes are used to approximate real-world scenes, and a set of homographies and their corresponding region masks are learned to transform the input image into a new view. Afifi et al. proposed a simple but powerful CNN model with an encoder-generator shape in "Pixel2Point: 3D object reconstruction from a single image using CNN and initial sphere". The model can generate point clouds of objects from a single image, but is usually limited by the fixedness of the network architecture and is not flexible enough to capture geometric structures. Inspired by the remarkable success of the Transformer model in multiple fields, Peiris et al. proposed a framework called 3D Volume Transformer in "Arobust volumetric transformer for accurate 3D tumor segmentation". This framework integrates feature extraction and view fusion in a single Transformer network, and uses the self-attention mechanism between multiple unordered inputs to explore the relationship between views, which improves the accuracy of 3D reconstruction, but the model has high computational complexity and training cost. With the rise of neural rendering technology, researchers have begun to directly use neural networks to learn continuous representations of scenes.Neural Radiance Fields (NeRF) uses deep neural networks to learn continuous volume functions of scenes in "Nerf: Representing scenes as neural radiance fields for view synthesis" to synthesize realistic new view images from sparse viewpoints, but neural rendering models are often computationally expensive and require a lot of memory and computing resources. Zhou et al. in "Sparsefusion: Distilling view-conditioned diffusion for 3D reconstruction" recover reasonable 3D representations by extracting 3D consistent scene representations from view-conditioned latent diffusion models, and their rendering is both accurate and realistic. et al. used monocular 2D supervised training to generate 3D representations from any view rendering in Renderdiffusion: Image diffusion for 3D reconstruction, inpainting and generation. Although Diffusion-based 3D reconstruction technology has made breakthrough progress in generating high-quality, high-resolution 3D content, it is very sensitive to the distribution of training data. Hong et al. proposed LRM in LRM: Large Reconstruction Model for Single Image to 3D, which uses a highly scalable Transformer-based architecture to predict the 3D model of an object from a single input image in just 5 seconds. Tochilkin et al. used the Transformer architecture for fast feedforward 3D generation based on the LRM network architecture in TripoSR: Fast 3D Object Reconstruction from a Single Image, which can generate 3D meshes from a single image in less than 0.5 seconds, but may produce blurred textures in unseen areas. Kerbl et al. in "3D Gaussian Splatting for Real-Time Radiance Field Rendering" use trainable 3D Gaussian models to represent scenes. These models can capture the complexity of the scene in a flexible and efficient way, which not only improves the real-time and quality of rendering, but also reduces the time required to train complex scenes and high-resolution images. Guédon et al. added a regularization term in "Sugar: Surface-aligned Gaussian splatting for efficient 3D mesh reconstruction and high-quality mesh rendering" to align the Gaussian well with the scene surface and improve the rendering quality; Kung et al. introduced a deep supervision strategy for shape alignment in "SAD-GS: Shape-aligned Depth-supervised Gaussian Splatting" to produce smooth and accurate reconstruction by adding additional constraints to the Gaussian shape; Yu et al. introduced a 3D smoothing filter in "Mip-splatting: Alias-free 3D Gaussian splatting" to constrain the size of the 3D Gaussian primitives according to the maximum sampling frequency caused by the input view, eliminating high-frequency artifacts when changing the camera distance.However, the reconstruction process usually requires huge computing resources, especially when dealing with complex scenes or high-resolution images. Adaptive control and multiple rounds of iterative optimization can also significantly increase computing time and memory usage.

[0004] Zou et al. proposed TGS in "Triplane meets gaussian splatting: Fast and generalizable single-view 3D reconstruction with transformers". They used two Transformer-based networks, namely point decoder and triplane decoder, to reconstruct 3D objects using a mixed triplane Gaussian intermediate representation. This hybrid representation achieves a balance, achieving faster rendering speed than implicit representation while providing higher rendering quality than explicit representation, but there are problems such as high memory usage and blurred reconstruction of the back of the image. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a 3D reconstruction method based on an improved TGS network. On the basis of TGS, a generative adversarial network, a DilateFormer module, and a pruning module are added to solve the problems of high memory usage and blurred reconstruction of the back of the image in the traditional TGS network in 3D reconstruction.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A three-dimensional reconstruction method based on an improved TGS network, wherein the improved TGS network introduces a GAN network into the TGS and introduces a DilateFormer module to replace the image encoder in the TGS;

[0008] The method comprises the following steps:

[0009] Step 1: Input a single-view image and the camera parameters of the single-view image, extract image features and integrate the camera parameters through the DliateFormer module;

[0010] Step 2: Input the extracted image features into the point cloud decoder to generate a rough point cloud;

[0011] Step 3: Enhance the point cloud features through projection-aware upsampling, and further enhance the accuracy of the point cloud through the GAN network;

[0012] Step 4: The enhanced point cloud features are encoded with geometric perception to generate three-plane initial position embedding, and then processed by the three-plane decoder to output the three-plane features;

[0013] Step 5: Combining point cloud, three-plane features and image features, the 3D Gaussian attributes are decoded by a 3D Gaussian decoder, and pruning is performed to remove redundant Gaussian balls;

[0014] Step 6: Output the reconstructed 3D model based on the generated 3D Gaussian attributes.

[0015] Preferably, in step 1, the process of extracting image features and integrating camera parameters through the DirectFormer module is:

[0016] The MSDA module is used in the first two stages to capture detailed information, and the MHSA module is used in the last two stages for high-level feature interaction.

[0017] Preferably, in step three, the GAN network uses the point cloud decoder as a generator, distinguishes real data from generated data through the discriminator of the GAN network, and the generator uses the feedback provided by the discriminator to optimize the accuracy of the point cloud.

[0018] Preferably, in step five, during the pruning operation, the redundancy of the Gaussian is evaluated based on global saliency calculation.

[0019] Preferably, the MSDA module consists of a multi-scale sliding window dilated attention operation SWDA, a splicing and a linear layer, expressed as:

[0020]

[0021] Among them, x is the input feature, k 1 ,k 2 ,…,k n is the expansion rate at different scales, represents a concatenation operation, which combines feature information of different scales, integrates multi-scale information through a linear layer, and outputs the final feature representation;

[0022] The output corresponding component x of the SWDA operation ij The definition is as follows:

[0023]

[0024] q ij is the element at position (i,j) in the query matrix Q, K r and V r are the key and value matrices selected according to the expansion rate r, d k is the dimension of the key vector, W and H are the width and height of the feature map respectively.

[0025] The evaluation process based on global significance is:

[0026] 1) According to the Gaussian training view, the i-th pixel r i The contribution of j );2) Obtain the global saliency score of each Gaussian by weighting by opacity and volume. The global saliency score is expressed as:

[0027]

[0028] where j is the Gaussian index, i represents a pixel, M, H, and W represent the number of training views, the height and width of the image, respectively, 1(G(Xj),ri) is an indicator function that takes the value 1 if the jth Gaussian intersects the i-th pixel and 0 otherwise, and γ(Σ j ) represents the volume of the jth Gaussian.

[0029] Beneficial effects:

[0030] 1. The invented G-TGS integrates the GAN network architecture module, DilateFormer module and global importance-based pruning module, enabling the model to pay more attention to the reconstruction of the back of the image and achieve low-memory, high-quality 3D reconstruction.

[0031] 2. In the GAN framework, the discriminator of the present invention is responsible for evaluating the generated data and the real data, and continuously optimizing the point cloud decoder through feedback to generate a more accurate point cloud model, thereby enhancing the integrity and realism of the three-dimensional model and making the reconstructed three-dimensional model more accurate.

[0032] 3. The present invention introduces an innovative DilateFormer module to replace the traditional TGS image encoder. The DilateFormer module can capture a wider range of contextual information in the image by expanding the receptive field. This design enhances the feature extraction capability and improves the accuracy of 3D reconstruction, making the reconstruction result more refined and accurate.

[0033] 4. After decoding and obtaining 3D Gaussian parameters, the present invention adopts pruning technology based on global significance score to reduce memory usage. By identifying and eliminating those unimportant or redundant parameters, the computational cost and storage requirements of the model are reduced, while maintaining the high quality of the reconstruction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The G-TGS structure diagram of the embodiment of the present invention is

[0035] Figure 2 The architecture diagram of the GAN of the embodiment of the present invention is

[0036] Figure 3 The DilateFormer module diagram of the embodiment of the present invention is

[0037] Figure 4 The MSDA structure diagram of the embodiment of the present invention is

[0038] Figure 5 Schematic diagram of the pruning operation module of an embodiment of the present invention

[0039] Figure 6 This is a TGS module diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] The present invention provides a three-dimensional reconstruction method based on an improved TGS network, wherein the improved TGS network introduces a GAN network into the TGS, and introduces a DilateFormer module to replace the image encoder in the TGS;

[0042] The method comprises the following steps:

[0043] Step 1: Input a single-view image and the camera parameters of the single-view image, extract image features and integrate the camera parameters through the DliateFormer module;

[0044] Step 2: Embed the image features with the initial position and use the point cloud decoder to generate a rough point cloud;

[0045] Step 3: Enhance the point cloud features through projection-aware upsampling, and further enhance the accuracy of the point cloud through the GAN network;

[0046] Step 4: The enhanced point cloud features are encoded with geometric perception to generate three-plane initial position embedding, and then processed by the three-plane decoder to output the three-plane features;

[0047] Step 5: Combining point cloud, three-plane features and image features, the 3D Gaussian attributes are decoded by a 3D Gaussian decoder, and pruning is performed to remove redundant Gaussian balls;

[0048] Step 6: Render the image of the new perspective according to the 3D Gaussian properties, and finally output the reconstructed 3D model and multiple new perspective 3D scene representations.

[0049] G-TGS introduces GAN (Generative Adversarial Network) into TGS, and uses the discriminator in GAN to better generate geometric features and solve the problem of reconstruction accuracy; introduces the DilateFormer module to replace the image encoder in TGS to obtain image features, seeks a better trade-off between computational complexity and the size of the participating receptive field, and improves reconstruction accuracy; after obtaining Gaussian parameter information, prunes by calculating the global significance score to remove redundant Gaussians and reduce memory usage

[0050] like Figure 1 As shown in the figure, G-TGS first inputs a single-view image and the corresponding camera parameters, extracts image features and integrates camera parameters through the DliateFormer module, then uses the point cloud decoder to generate a rough point cloud in combination with the initial position embedding, then enhances the point cloud features through projection-aware upsampling, and then further enhances the accuracy of the point cloud through GAN. The enhanced point cloud features are encoded with geometric awareness to generate a three-plane initial position embedding, which is processed by the three-plane decoder to output three-plane features; finally, the point cloud, three-plane features and image features are combined, and the 3D Gaussian decoder is used to decode the 3D Gaussian attributes, and pruning operations are performed to remove redundant Gaussian balls to reduce memory usage, and then the Gaussian Splatting rendering technology is used to render new perspective images from these attributes, and finally the reconstructed high-quality 3D model and multiple new perspective 3D scene representations are output.

[0051] like Figure 2 As shown in Figure 1, the accuracy of reconstruction is optimized by adding the GAN discriminator after the upsampling operation after the point cloud decoder. The discriminator is responsible for distinguishing real data from generated data, while the generator uses the feedback provided by the discriminator to optimize its parameters. Discriminator loss function:

[0052]

[0053] Among them, D is the discriminator, G is the generator, x is the real data, z is the noise vector, p data(x) and p z(z) are the distribution of data and noise respectively. As a generator, the point cloud decoder continuously receives feedback from the discriminator during adversarial training to adjust its parameters. This process aims to improve the authenticity and accuracy of the generated features, so that the generated data is closer to the real data in terms of quality and credibility.

[0054] Through this iterative optimization mechanism, the point cloud decoder can gradually improve its performance, generate more detailed and reliable three-dimensional point cloud data, and solve the accuracy problems of traditional methods in image reconstruction, thereby achieving a more realistic three-dimensional reconstruction effect.

[0055] like Figure 3As shown in the figure, by using the DilateFormer module to replace the image encoder in TGS, the computational cost is reduced while the ability to extract features is improved. DilateFormer adopts a pyramid architecture, using multi-scale dilated attention blocks (MSDA) in the first two stages to capture detailed information and multi-head self-attention (MHSA) in the last two stages for high-level feature interaction. MSDA consists of a multi-scale sliding window dilated attention operation (SWDA), concatenation, and linear layers.

[0056]

[0057] Among them, x is the input feature, k 1 ,k 2 ,…,k n is the expansion rate at different scales, Represents the concatenation operation. The concatenation operation combines feature information of different scales, integrates multi-scale information through a linear layer, and outputs the final feature representation.

[0058] like Figure 4 As shown, SWDA performs sliding window dilation self-attention with different dilation rates on different heads to capture multi-scale information in the image. For position (i, j), the output component x of the SWDA operation ij The definition is as follows:

[0059]

[0060] q ij is the element at position (i,j) in the query matrix Q, K r and V r are the key and value matrices selected according to the expansion rate r, d k is the dimension of the key vector, W and H are the width and height of the feature map, respectively. For each query q ij , we start from K r Select and q ij The relevant keys are calculated, their dot products are then passed through the softmax function to get the weights, and finally these weights are used to calculate V r The values ​​in are weighted summed to get the output x ij .

[0061] MHSA can automatically capture the dependencies between sequences when processing input sequences, thereby better understanding contextual information and improving model performance. The overlapping word segmenter performs patch embedding, adjusts the resolution of the output feature map by controlling the stride size of the convolution kernel, and uses the overlapping downsampler to merge patches from the previous stage. This design enables the model to capture local details and global contextual information at the same time, improving the model's overall understanding of the image.

[0062] like Figure 5 As shown in Figure 1, after the Gaussian decoder obtains the parameters of the Gaussian, a pruning operation is performed to make it occupy less memory. Pruning is used to eliminate the most representative redundant Gaussians without affecting the overall performance of the network. Pruning Gaussians based on simplified criteria (such as the opacity of points) will significantly reduce the performance of the model. Therefore, it is necessary to comprehensively consider the global saliency calculation to evaluate the redundancy of Gaussians.

[0063] According to the Gaussian i The contribution of j )), and then weighted by opacity and volume to obtain the global saliency score of each Gaussian. The global saliency score can be summarized as:

[0064]

[0065] where j is the Gaussian index, i represents a pixel, M, H, and W represent the number of training views, the height and width of the image, respectively. 1(G(Xj),ri) is an indicator function that takes the value 1 if the jth Gaussian intersects the i-th pixel and 0 otherwise. j Represents the opacity of the jth Gaussian, which is used to determine the Gaussian's contribution to color and opacity during rendering. j ) represents the volume of the jth Gaussian. The larger the volume, the wider the range occupied by the Gaussian in space and the more contribution it makes to the pixel.

[0066] The higher the score, the more important the Gaussian is in scene reconstruction, and vice versa. In this way, Gaussians with less impact on visual quality can be identified and pruned to reduce the complexity and computational cost of the model while maintaining rendering quality as much as possible.

[0067] The specific reconstruction process of this embodiment is as follows:

[0068] 1. DilateFormer module such as Figure 3 Shown

[0069] 1.1 Input Processing

[0070] Input image: DilateFormer accepts the input image I and uses overlapping tokenizers (multiple overlapping 3x3 convolution modules) to block it and generate the initial feature map.

[0071] Position encoding: Use Conditional Position Embedding (CPE) to add position information to the feature map to obtain X, enhancing the model's ability to perceive spatial structure.

[0072] 1.2 Feature Extraction

[0073] 1.2.1MSDA block captures local information

[0074] Implementation steps:

[0075] 1. Use linear projection to obtain query Q, key K, and value V from feature graph X.

[0076] 2. Assign Q, K, and V channels to different heads.

[0077] 3. Use different dilation rates r in different heads i Perform multi-scale SWDA.

[0078] 4. Output all headers {h i} and then sent to the linear layer for feature aggregation.

[0079] Input and Output:

[0080] Input: Feature map X.

[0081] Output: Aggregated feature map X out .

[0082] Specific formula:

[0083] Q=W Q X+P Q

[0084] K=W K X+P K

[0085] V=W V X=P V

[0086] Where: X is the feature map after patch embedding. W Q , W K , W V is a learnable linear transformation matrix. Q , P K , P V are the corresponding positional encodings, which can be learnable.

[0087] h i =SWDA(Q i ,K i ,V i ,r i )

[0088] X out =Linear(Concat[h 1 ,…,h n ])

[0089] Among them, h i is the output of the ith head, r i is the expansion rate of the ith head.

[0090] 1.2.2 SWDA

[0091] Implementation steps:

[0092] 1. Input query matrix Q, key matrix K, value matrix V.

[0093] 2. Sparsely select keys and values ​​in K and V according to the dilation rate r and window size w×w.

[0094] 3. Perform self-attention on the representative patches in the sliding window centered at the query Q.

[0095] Input and Output:

[0096] Input: query matrix Q, key matrix K, value matrix V.

[0097] Output: Updated feature map X.

[0098] Specific formula:

[0099] X=SWDA(Q,K,V,r)

[0100]

[0101] Among them, x ij is the (i,j)th element of the output X, Kr and Vr are the key and value selected according to the dilation rate r, d k is the dimension of the key vector.

[0102] 1.2.3MHSA blocks perform global information exchange

[0103] At a high level, the MHSA block performs a standard multi-head self-attention operation to capture global dependencies. 1.3 Overlapping Tokenizer

[0104] Overlapping tokenizers usually use a set of overlapping convolutional kernels to process the input image, dividing the image into overlapping patches.

[0105] Input and Output:

[0106] Input: Input image I.

[0107] Output: A series of overlapping patches X tokens ,These patches are the result of the convolution operation on the input image and can be regarded as feature maps.

[0108] Specific formula:

[0109] X tokens =Conv(I;W conv )

[0110] Where: I is the input image. W conv The weight of the convolution kernel. X tokens is the sequence of patches after the convolution operation.

[0111] 1.4 Overlapping Downsampler

[0112] Overlapping downsamplers usually use convolutional kernels with overlap to reduce the resolution of feature maps.

[0113] Input and Output:

[0114] Input: Feature map X from the previous stage prev .

[0115] Output: downsampled feature map X down , which has a lower resolution than the input feature map.

[0116] This process can be expressed by the following formula:

[0117] X down =Conv(X prev ; W down )

[0118] Where: X prev is the feature map of the previous stage. down is the weight of the convolution kernel in the downsampler. X down It is the feature map after downsampling.

[0119] 1.5 Output

[0120] The final output is the feature map used for 3D reconstruction.

[0121] 2. GAN network architecture module Figure 2 Shown

[0122] 2.1 Discriminator

[0123] The discriminator determines the difference between the generated 3D point cloud and the real point cloud and feeds the difference back to the generator part.

[0124] enter:

[0125] True data x: real samples from the training set.

[0126] Generated data: 3D point cloud generated by the point cloud decoder.

[0127] Output:

[0128] The discriminator D outputs a probability value, which indicates the probability that the input data is real data.

[0129] Implementation steps:

[0130] 1. The discriminator D receives real data x and generated data.

[0131] 2. The discriminator D outputs a probability value, indicating the probability that the input data is real data.

[0132] 3. The discriminator D is trained to better distinguish between real data and generated data by comparing the two.

[0133] Specific formula:

[0134]

[0135] in, For the real data distribution p data(x) log D(x) is the logarithm of the output of the discriminator D for the true sample x. For the noise distribution p z(z) The expected value of the sample z is log(1-D(G(z))) The discriminator D generates the sample G (z) The complement of the logarithm of the output.

[0136] 2.2 Generator (point cloud decoder) structure:

[0137] enter:

[0138] Image feature vector: The features generated by the DilateFormer module based on the input image.

[0139] Initial position embedding: used to generate the initial position information of the point cloud.

[0140] Output:

[0141] 3D point cloud data: point cloud representation of the generated 3D model.

[0142] Implementation steps:

[0143] 1. The generator (point cloud decoder) accepts the features and initial position embedding generated by the DilateFormer module.

[0144] 2. The generator (point cloud decoder) generates preliminary point cloud data.

[0145] Through adversarial training with the discriminator D, the generator (point cloud decoder) continuously adjusts parameters to generate more realistic point cloud data.

[0146] Specific formula:

[0147]

[0148] 3. Pruning modules based on global importance such as Figure 5 Shown

[0149] enter:

[0150] Gaussian parameter information obtained by Gaussian decoder.

[0151] Output:

[0152] The pruned Gaussian parameter information removes redundant Gaussians.

[0153] Specific steps:

[0154] 1. Calculate the contribution of each Gaussian to each pixel in the training view to get the initial saliency score.

[0155] 2. Obtain the global saliency score of each Gaussian by weighting it by opacity and volume.

[0156] 3. Based on the global saliency score, identify Gaussians that have less impact on visual quality.

[0157] 4. Cut off these unimportant Gaussians to reduce the complexity and computational cost of the model.

[0158] Related formula:

[0159] The global saliency score can be expressed as:

[0160]

[0161] where j is the Gaussian index, i represents a pixel, M, H, and W represent the number of training views, the height and width of the image, respectively. 1(G(Xj),ri) is an indicator function that takes the value 1 if the jth Gaussian intersects the i-th pixel and 0 otherwise. j represents the opacity of the jth Gaussian, γ(Σ j ) represents the volume of the jth Gaussian.

[0162] In summary, the G-TGS of the present invention integrates the network architecture module of GAN, the DilateFormer module and the pruning module based on global importance, showing many advantages. On the basis of TGS, the addition of the generative adversarial network, the DilateFormer module and the pruning module enables the model to pay more attention to the reconstruction of the back of the image, compared with the traditional TGS network (such as Figure 6 As shown in the figure, low-memory and high-quality 3D reconstruction is achieved.

Claims

1. A three-dimensional reconstruction method based on an improved TGS network, characterized in that: The improved TGS network introduces a GAN network into TGS, and introduces a DilateFormer module to replace the image encoder in TGS; the method comprises the following steps: Step 1: Input a single-view image and the camera parameters of the single-view image, extract image features and integrate the camera parameters through the DliateFormer module; Step 2: Input the extracted image features into the point cloud decoder to generate a rough point cloud; Step 3: Enhance the point cloud features through projection-aware upsampling, and further enhance the accuracy of the point cloud through the GAN network; Step 4: The enhanced point cloud features are encoded with geometric perception to generate three-plane initial position embedding, and then processed by the three-plane decoder to output the three-plane features; Step 5: Combining point cloud, three-plane features and image features, the 3D Gaussian attributes are decoded by a 3D Gaussian decoder, and pruning is performed to remove redundant Gaussian balls; Step 6: Output the reconstructed 3D model based on the generated 3D Gaussian attributes.

2. The three-dimensional reconstruction method based on the improved TGS network according to claim 1, characterized in that: In step 1, the process of extracting image features and integrating camera parameters through the DirectFormer module is as follows: The MSDA module is used in the first two stages to capture detailed information, and the MHSA module is used in the last two stages for high-level feature interaction.

3. The three-dimensional reconstruction method based on the improved TGS network according to claim 1, characterized in that: In step three, the GAN network uses the point cloud decoder as a generator, and distinguishes real data from generated data through the discriminator of the GAN network. The generator uses the feedback provided by the discriminator to optimize the accuracy of the point cloud.

4. The three-dimensional reconstruction method based on the improved TGS network according to claim 1, characterized in that: In step 5, during the pruning operation, the redundancy of Gaussian is evaluated based on the global saliency calculation.

5. The three-dimensional reconstruction method based on the improved TGS network according to claim 2, characterized in that: The MSDA module consists of a multi-scale sliding window dilated attention operation SWDA, concatenation and linear layers, expressed as: Among them, x is the input feature, k1,k2,…,k n is the expansion rate at different scales, represents a concatenation operation, which combines feature information of different scales, integrates multi-scale information through a linear layer, and outputs the final feature representation; The output corresponding component x of the SWDA operation ij The definition is as follows: q ij is the element at position (i,j) in the query matrix Q, K r and V r are the key and value matrices selected according to the expansion rate r, d k is the dimension of the key vector, W and H are the width and height of the feature map respectively.

6. The three-dimensional reconstruction method based on the improved TGS network according to claim 4, characterized in that: The evaluation process based on global significance is: 1) According to the Gaussian training view, the i-th pixel r i The contribution of j ); 2) The global saliency score of each Gaussian is obtained by weighting by opacity and volume. The global saliency score is expressed as: where j is the Gaussian index, i represents a pixel, M, H, and W represent the number of training views, the height and width of the image, respectively, 1(G(Xj),ri) is an indicator function that takes the value 1 if the jth Gaussian intersects the i-th pixel and 0 otherwise, and γ(Σ j ) represents the volume of the jth Gaussian.

Images