A method for three-dimensional reconstruction of gaussian sputtering based on joint attention mechanism

Abstract

The present application relates to the field of three-dimensional reconstruction, and more particularly to a Gaussian sputtering three-dimensional reconstruction method based on a joint attention mechanism, which combines channel attention mechanism and spatial attention mechanism, so that the model can discover and utilize the possible position correlation between different Gaussians, optimizes different parameters of the three-dimensional Gaussian through position weight, and performs reconstruction and rendering using the optimized parameters, thereby improving the overall quality and local performance of the three-dimensional reconstruction image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional reconstruction, and particularly relates to a Gaussian Splatting three-dimensional reconstruction method based on a joint attention mechanism. BACKGROUND

[0002] Three-dimensional reconstruction (3D Reconstruction) refers to establishing a mathematical model suitable for computer representation and processing of a three-dimensional object. This process is the basis for processing, operating and analyzing the properties of a three-dimensional object in a computer environment, and is also a key technology for establishing a virtual reality in a computer to express the objective world. Three-dimensional reconstruction is widely used in computer science and technology, computer application, multimedia, computer graphics, computer vision and other fields.

[0003] Gaussian Splatting technology simulates and reconstructs a three-dimensional scene through Gaussian ellipsoid small particles, which are defined by four important properties: position, covariance matrix, opacity and spherical harmonic (SH) coefficient. Among them, the position determines the position of the ellipsoid center in space; the covariance matrix determines the scaling and rotation degree of the ellipsoid; the opacity determines the degree of blocking light by the ellipsoid; and the spherical harmonic (SH) coefficient determines the color distribution of the ellipsoid surface. Compared with other methods (such as NeRF) that require a large amount of memory and computing resources, Gaussian Splatting technology still has certain advantages in memory and computing requirements. However, the position correlation between a group of Gaussians is often ignored in related research in reality, which to some extent affects the overall quality and local performance of image reconstruction.

[0004] Due to the lack of attention to position correlation, the current Gaussian Splatting three-dimensional reconstruction method often has certain deficiencies in image reconstruction quality and image local performance, so it is of great significance to find a method to adaptively focus on the possible position correlation between different Gaussians. SUMMARY

[0005] The present application aims to provide a Gaussian Splatting three-dimensional reconstruction method based on a joint attention mechanism to solve the technical problem that the current Gaussian Splatting three-dimensional reconstruction method often has certain deficiencies in image reconstruction quality and image local performance due to the lack of attention to position correlation.

[0006] The Gaussian Splatting three-dimensional reconstruction method based on a joint attention mechanism in the present application includes the following steps:

[0007] Step 1: construct a joint attention neural network model for obtaining position weight coefficients according to the following strategy:

[0008] The input of the joint attention neural network model includes two parts, one of which is a pre-set three-plane data structure, which includes a trainable original three-plane data, and the three-plane data structure is mainly used to store a feature vector representing a Gaussian position coordinate, which is a trainable parameter, and the structure is three matrices of the same size with a size of (H, W, C), H represents height, W represents width, and C represents depth or channel number.

[0009] The original three-plane data is respectively input into a channel attention network module for internal self-attention calculation and a spatial attention network module for spatial attention calculation.

[0010] The three-plane data after passing through the channel attention network module and the spatial attention network module is respectively orthogonalized to obtain a channel three-plane and a spatial three-plane.

[0011] The other part of the model input is the position coordinates contained in each Gaussian point cloud data in a set of 3D Gaussian point clouds constructed by the same three-dimensional reconstruction scene; by projecting the position coordinates onto the channel three-plane and the spatial three-plane respectively, a first attention feature vector corresponding to the channel three-plane and a second attention feature vector corresponding to the spatial three-plane are obtained.

[0012] The first attention feature vector and the second attention feature vector are spliced to obtain a spliced feature vector, which is input into a multi-layer perception (MLP) for perception to obtain a position weight coefficient corresponding to the position coordinates.

[0013] Step 2: input the position coordinates contained in the 3D Gaussian point cloud to be reconstructed and the optimized original three-plane data into the optimized joint attention neural network model to obtain the corresponding position weight coefficient.

[0014] Step 3: use the position weight coefficient to optimize the to-be-optimized parameters contained in the reconstructed 3D Gaussian point cloud for Gaussian sputtering three-dimensional reconstruction to obtain the optimized parameters.

[0015] Step 4: use the 3D Gaussian point cloud with the optimized to-be-optimized parameters for Gaussian sputtering three-dimensional reconstruction.

[0016] Further, the three-dimensional reconstruction scene includes multiple high-definition images taken from different angles and corresponding camera parameters of each high-definition image, which can construct a set of sparse SFM point clouds and be initialized as a set of 3D Gaussian point clouds.

[0017] Each Gaussian point cloud includes position coordinates, to-be-optimized parameters, and a covariance matrix, and the to-be-optimized parameters include a rotation coefficient, a scaling coefficient, and an opacity.

[0018] Furthermore, in the optimization process of the original three-plane data and the joint attention neural network model, the Gaussian sputtering 3D reconstruction obtained from the training set data is rendered as the final rendered image, and the parameters are optimized using the total loss function based on the difference between the final rendered image and the real image of the corresponding training set data.

[0019] The total loss function includes the image quality loss function and the mean absolute error between the final rendered image and the real image.

[0020] Furthermore, the total loss function is expressed as:

[0021] Loss=(1-λ)L1+λ·L D-SSIM

[0022] Where Loss represents the total loss function; λ represents the weight coefficient, λ≤1, L D-SSIM Let L1 represent the image quality loss function, where L1 is the mean absolute error.

[0023] Furthermore, in step 1, SENet is used as the channel attention network module to assign weights to the feature vectors of each location coordinate.

[0024] Swin-Transformer is used as the spatial attention network module to solve the positional correlation between feature vectors at different location coordinates.

[0025] Furthermore, the so-called orthogonalization specifically includes: the three matrices in the three-plane data each form a gridded plane, and are arranged orthogonally to each other in space to form a gridded square space. The orthogonalized three-plane data is represented as follows: the three matrices in the three-plane data structure correspond to the gridded xy plane, yz plane and xz plane in space, respectively, and the correspondence is determined at the time of initialization and remains consistent.

[0026] Furthermore, the first and second attention feature vectors are the feature values ​​corresponding to the position coordinates in the gridded space, i.e., Tri_orth(x,y,z)=(ω xy ,ω yz ,ω zx ), ω xy ,ω yz ,ω zx These represent the eigenvectors corresponding to the projections of the position coordinates (x, y, z) onto the xy plane, yz plane, and xz plane, respectively. The specific values ​​are obtained through double interpolation.

[0027] Furthermore, the process of calculating the position weight coefficients corresponding to the position coordinates is as follows:

[0028] Tri1 = SE(Tri);

[0029] Tri1_orth = orth(Tri1);

[0030] F1 = Tri1_orth(x, y, z);

[0031] Tri2 = ST(Tri);

[0032] Tri2_orth = orth(Tri2);

[0033] F2 = Tri2_orth(x, y, z);

[0034]

[0035] Wherein, Tri represents original tri-plane data, SE represents a channel attention neural network in a channel attention network module, orth represents orthogonalization, Tri1 represents channel tri-plane data obtained by passing the original tri-plane data through the channel attention neural network, Tri1_orth represents the orthogonalized channel tri-plane, F1 represents a first attention feature vector, ST represents a spatial attention neural network in a spatial attention network module, Tri2 represents spatial tri-plane data obtained by passing the original tri-plane data through the spatial attention neural network, Tri2_orth represents the orthogonalized spatial tri-plane, F2 represents a second attention feature vector, represents vector concatenation operation, MLP represents a perception process of a multi-layer perception, f θ (x, y, z) represents a position weight coefficient.

[0036] Further, in step 4, the initial rotation coefficient, the initial scaling coefficient and the initial opacity of the 3D Gaussian point cloud are respectively optimized by the position weight coefficient to obtain the corresponding optimized rotation coefficient, the optimized scaling coefficient and the optimized opacity, and the specific formula is as follows:

[0037] O' = O * f θ (x, y, z);

[0038] S' = S * f θ (x, y, z);

[0039] R' = R * f θ (x, y, z);

[0040] Wherein, f θ (x, y, z) is a position weight coefficient, O represents an initial opacity, S represents an initial scaling coefficient, R represents an initial rotation coefficient, O' represents an optimized opacity, S' optimized represents a scaling coefficient, and R' represents an optimized rotation coefficient.

[0041] Furthermore, in step 4, Gaussian adaptive density control is performed based on the position coordinates of the 3D Gaussian point cloud, the optimized parameters, and the covariance matrix. The 3D Gaussian point cloud obtained after density control is then rendered to obtain the final rendered image.

[0042] The embodiments of this application have the following beneficial effects:

[0043] The model and method proposed in this invention combine channel attention and spatial attention mechanisms, enabling the model to discover and utilize the potential positional correlations between different Gaussians. By optimizing different parameters of the 3D Gaussians through positional weights, this method achieved an average SSIM index of 0.852 and an average PSNR index of 24.02 in the Tanks & Temples dataset, thus improving the overall quality and local performance of the 3D reconstructed image. Attached Figure Description

[0044] To more clearly illustrate the technical solution of the present invention, the accompanying drawings required in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and therefore should not be regarded as a limitation on the scope of protection of the present invention. In the various drawings, similar components are numbered similarly.

[0045] Figure 1 This illustration shows a schematic diagram of the determination of position weight coefficients based on a joint attention neural network model and position coordinates in a Gaussian sputtering 3D reconstruction method based on a joint attention mechanism proposed in an embodiment of this application.

[0046] Figure 2 The diagram shows a flowchart of a Gaussian sputtering 3D reconstruction method based on a joint attention mechanism proposed in an embodiment of this application. Detailed Implementation

[0047] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0048] To more clearly demonstrate the implementation steps and advantages of this invention, the specific implementation methods are described below with reference to the illustrations.

[0049] This application proposes a Gaussian sputtering 3D reconstruction method based on a joint attention mechanism. This method can discover and utilize the possible positional correlations between a set of Gaussian point clouds, thereby improving the overall quality and local performance of the 3D reconstructed image.

[0050] The method in this example is aimed at a set of 3D Gaussian point clouds constructed by the same three-dimensional reconstructed scene. First, each three-dimensional reconstructed scene contains multiple high-definition images taken from different angles and the corresponding camera parameters of each high-definition image, and a set of sparse SFM (Structure From Motion) point clouds is constructed on this technology. Then, the constructed set of sparse SFM point clouds is initialized as a set of 3D Gaussian point clouds. Each Gaussian point cloud includes position coordinates, parameters to be optimized, and a covariance matrix. The parameters to be optimized include rotation coefficients, scaling coefficients, opacity, and several main parameters.

[0051] Specifically, the method includes the following steps:

[0052] First, a joint attention neural network model as shown in Figure 1 is established. The input of the model includes two parts. One of them is a pre-set three-plane data structure, which includes a trainable original three-plane data. The three-plane data structure is mainly used to store feature vectors representing Gaussian position coordinates. The feature vectors are trainable parameters. The structure is three matrices of the same size with dimensions (H, W, C), where H represents height, W represents width, and C represents depth or the number of channels. In the initialization stage, the parameters are generally obtained randomly. For example, the trainable three-plane data is represented by a tensor Tri with a size of (3, 512, 24, 24). The 3 represents 3 planes, 24 represents width, 24 represents height, and 512 represents depth or the number of channels.

[0053] As shown in Figure 1As shown, in the model, the original three-plane data are respectively input into a channel attention network module for internal self-attention calculation and a spatial attention network module for spatial attention calculation. In this case, a joint attention neural network model is built based on SENet (Squeeze-and-Excitation Networks) and Swin-Transformer. The SENet is a convolutional neural network structure for deep learning, which optimizes feature extraction by introducing a channel attention mechanism. In other words, the SENet in this case is a channel attention network module, which is used to assign weights to the feature vectors of each position coordinate, i.e., for internal self-attention calculation, which is equivalent to a feature self-attention mechanism module. Such methods are well known to those skilled in the art and will not be described here. The Swin-Transformer is a method for processing computer vision tasks using a transformer architecture. In this case, the Swin-Transformer network is used as a spatial attention network module to solve the positional correlation between feature vectors of different position coordinates, so that the feature vectors of each position coordinate have global correlation, i.e., for spatial attention calculation. Such methods are well known to those skilled in the art and will not be described here.

[0054] The three-plane data processed by the channel attention network module and the spatial attention network module are respectively orthogonalized to obtain channel three-plane and spatial three-plane. The so-called orthogonalization is basically as follows Figure 1 As shown in the middle, the three matrices in the three-plane data each form a grid plane and are arranged orthogonally in space to form a grid square space. The orthogonalized three-plane data is represented as Tri_orth = (Tri xy , Tri yz , Tri xz ). It can be understood that the three matrices in the three-plane data structure correspond to the grid xy plane, yz plane and xz plane in space respectively, and the corresponding relationship is determined at initialization and remains consistent. Orthogonalization converts a linearly independent vector system into an orthogonal system (i.e., vectors are orthogonal to each other, i.e., their inner product is zero), which aims to simplify the processing and analysis of complex problems. By generating a new set of orthogonal vectors, these vectors maintain the original linear correlation, but their characteristics make the problem easier to solve.

[0055] At this time, the model introduces another part of input, i.e. the aforementioned set of 3D Gaussian point clouds, and the position coordinates contained in each Gaussian point cloud data. By projecting the position coordinates to the channel three planes and the spatial three planes respectively, the first attention feature vector corresponding to the channel three planes and the second attention feature vector corresponding to the spatial three planes are obtained. Specifically, the first and second attention feature vectors are the feature values corresponding to the position coordinates in the grid space, i.e. Tri_orth(x,y,z)=(ω xy ,ω yz ,ω zx ), ω xy ,ω yz ,ω zx respectively represent the feature vectors corresponding to the position coordinates (x,y,z) at the projection positions on the xy plane, the yz plane and the xz plane, and the specific values are obtained by the double interpolation method.

[0056] The first attention feature vector and the second attention feature vector are spliced, and in this example, the splicing is performed in the depth dimension of the attention feature vector; the spliced splicing feature vector is sent to a multi-layer perception (MLP) for perception to obtain the position weight coefficient corresponding to the position coordinates.

[0057] Exemplarily, the process of calculating the position weight coefficient corresponding to the position coordinates is shown as follows:

[0058] Tri1=SE(Tri);

[0059] Tri1_orth=orth(Tri1);

[0060] F1=Tri1_orth(x,y,z);

[0061] Tri2=ST(Tri);

[0062] Tri2_orth=orth(Tri2);

[0063] F2=Tri2_orth(x,y,z);

[0064]

[0065] Where Tri represents the original three-plane data, SE represents the channel attention neural network in the channel attention network module, orth represents orthogonalization, Tri1 represents the channel three-plane data obtained by passing the original three-plane data through the channel attention neural network, Tri1_orth represents the orthogonalized channel three-plane data, F1 represents the first attention feature vector, ST represents the spatial attention neural network in the spatial attention network module, Tri2 represents the spatial three-plane data obtained by passing the original three-plane data through the spatial attention neural network, Tri2_orth represents the orthogonalized spatial three-plane data, and F2 represents the second attention feature vector. This represents vector concatenation operation; MLP represents the perception process of a multilayer perceptron; f θ (x,y,z) represents the position weight coefficient.

[0066] Immediately afterwards, such as Figure 2 As shown, the method designed in this example is to optimize the parameters to be optimized using position weight coefficients to obtain optimized parameters; Gaussian adaptive density control is then performed based on the position coordinates, optimized parameters, and covariance matrix, and the resulting 3D Gaussian image is rendered to obtain the final rendered image. The parameters to be optimized include the initial rotation coefficient, initial scaling coefficient, and initial opacity.

[0067] Specifically, the initial rotation coefficient, initial scaling coefficient, and initial opacity are optimized by using the position weight coefficients of the Gaussian point cloud, resulting in the corresponding optimized rotation coefficient, optimized scaling coefficient, and optimized opacity.

[0068] The specific formula for the above optimization process is as follows:

[0069] O′O*f θ (x,y,z);

[0070] S′=S*f θ (x,y,z);

[0071] R′=R×f θ (x,y,z);

[0072] Among them, f θ (x,y,z) represents the position weight coefficients, O represents the initial opacity, S represents the initial scaling factor, R represents the initial rotation factor, O′ represents the optimized opacity, S′ represents the optimized scaling factor, and R′ represents the optimized rotation factor.

[0073] The Gaussian adaptive density control is performed according to the position coordinates, the covariance matrix, the optimized rotation coefficient, the optimized scaling coefficient and the optimized opacity of the Gaussian point cloud, so that the total number of the 3D Gaussian point cloud is ensured in a reasonable interval. The Gaussian adaptive density control is a technology commonly introduced when processing 3D Gaussian splatting or related 3D scene representation methods. The adaptive density control is a mechanism for dynamically adjusting the Gaussian density according to the geometric complexity of the scene, which helps to improve the compactness and efficiency of the 3D scene representation and plays an important role in various 3D rendering technologies. Such methods are well known to those skilled in the art and will not be described here.

[0074] The 3D Gaussian point cloud obtained after the density control is sent to the renderer for rendering to obtain a final rendered image. Exemplarily, the 3D Gaussian point cloud can be rendered using fast differentiable rasterization to obtain the final rendered image.

[0075] For the training of the aforementioned joint attention neural network model, the preferred but not limited parameter optimization is based on a loss function representing the difference between the final rendered image and the real image, to obtain an optimized joint attention neural network model.

[0076] In the exemplary training, in each training, the final rendered image is obtained according to the method shown, and the average absolute error (L1 Loss) and the structural similarity index (Structural Similarity Index, SSIM) loss function between the final rendered image and the real image corresponding to the final rendered image are weighted and added as the total loss function Loss for optimizing the joint attention neural network model. The joint attention neural network model is trained according to the pre-set training times until the joint attention neural network model converges, and an optimized joint attention neural network model is obtained.

[0077] The total loss function Loss formula of the model is:

[0078] Loss = (1 - λ) L1 + λ · L D-SSIM

[0079] Wherein, Loss represents the total loss function; λ represents the weight coefficient, λ≤1, L D-SSIM represents the image quality loss function, and L1 is the average absolute error.

[0080] After obtaining the optimized joint attention neural network model, the test set in the three-dimensional reconstruction data is input into the optimized joint attention neural network model to obtain the corresponding three-dimensional reconstruction image.

[0081] The three-dimensional reconstruction data set used for training in this embodiment is taken from the Tanks&Temples data set, which is divided into a training set and a test set. The Tanks&Temples data set is a large multi-view data set for three-dimensional reconstruction research, released by Arno Knapitsch, Jaesik Park, Qian-Yi Zhou and Vladlen Koltun of Intel Labs in 2017. The training set thereof contains 7 high-resolution videos of 7 scenes. The test set is divided into an intermediate group and an advanced group. The intermediate group contains scenes such as sculptures, large vehicles and house buildings with appearance camera trajectories, which are relatively simple and suitable for preliminary three-dimensional reconstruction testing; the advanced group contains indoor scenes taken from the inside and large outdoor scenes, which have complex geometric layouts and camera trajectories and pose higher requirements on three-dimensional reconstruction algorithms. The data set has a wide range of applications and influence in the field of three-dimensional reconstruction and is used as a benchmark data set by multiple research projects and papers.

[0082] The performance of the method in this example on the Tanks&Temples data set is compared with the current most advanced result as shown in the following table:

[0083]

[0084]

[0085] The PSNR (Peak signal-to-noise ratio) is an engineering term representing the ratio of the maximum possible power of a signal to the power of the disruptive noise affecting it. Since many signals have very wide dynamic ranges, the peak signal-to-noise ratio is often expressed in the logarithmic decibel scale. The peak signal-to-noise ratio is often used as a measure of signal reconstruction quality in fields such as image compression, which is often simply defined by the mean square error (MSE).

[0086] The model and method in this example combine channel attention mechanisms and spatial attention mechanisms, enabling the model to discover and utilize possible position correlations between different Gaussians, and optimizing different parameters of three-dimensional Gaussians through position weights. In the Tanks&Temples data set, the model achieves an average SSIM index of 0.852 and an average PSNR index of 24.02, improving the overall image quality and local detail performance of the three-dimensional reconstruction image.

[0087] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which shall be covered within the protection scope of the present application.

Claims

1. A method for Gaussian sputtering three-dimensional reconstruction based on joint attention mechanism, characterized in that, Includes the following steps: Step 1: Construct a joint attention neural network model for obtaining position weight coefficients using the following strategy: The input of the joint attention neural network model includes two parts, one of which is a pre-set three-plane data structure, which includes a trainable original three-plane data. The three-plane data structure is mainly used to store feature vectors representing Gaussian position coordinates, and these feature vectors are trainable parameters. The original three-plane data are respectively input into the channel attention network module for internal self-attention calculation and the spatial attention network module for spatial attention calculation; The three-plane data, after passing through the channel attention network module and the spatial attention network module, are orthogonalized to obtain the channel three-plane and the spatial three-plane; Another part of the model's input is the position coordinates contained in each Gaussian point cloud data in a set of 3D Gaussian point clouds constructed through the same 3D reconstructed scene; by projecting the position coordinates onto the channel three planes and the spatial three planes respectively, the first attention feature vector corresponding to the channel three planes and the second attention feature vector corresponding to the spatial three planes are obtained. The first attention feature vector and the second attention feature vector are concatenated, and the concatenated feature vector is fed into a multilayer perceptron (MLP) for perception to obtain the position weight coefficients corresponding to the position coordinates; Step 2: Input the position coordinates contained in the 3D Gaussian point cloud to be reconstructed and the optimized original three-plane data into the optimized joint attention neural network model to obtain the corresponding position weight coefficients. Step 3 uses the position weighting coefficients to optimize the parameters to be optimized in the reconstructed 3D Gaussian point cloud for Gaussian sputtering 3D reconstruction, and obtains the optimized parameters. Step 4: Perform Gaussian sputtering 3D reconstruction using the 3D Gaussian point cloud with optimized parameters.

2. The method of claim 1, wherein, The 3D reconstructed scene contains multiple high-definition images taken from different angles and the camera parameters corresponding to each high-definition image, which can construct a set of sparse SFM point clouds and be initialized as a set of 3D Gaussian point clouds. Each Gaussian point cloud includes position coordinates, parameters to be optimized, and a covariance matrix. The parameters to be optimized include rotation coefficients, scaling coefficients, and opacity.

3. The method of claim 1, wherein, In the optimization process of the original three-plane data and the joint attention neural network model, the Gaussian sputtering 3D reconstruction obtained from the training set data is rendered as the final rendered image. The parameters are optimized using the total loss function that represents the difference between the final rendered image and the real image of the corresponding training set data. The total loss function includes the image quality loss function and the mean absolute error between the final rendered image and the real image.

4. The method of claim 3, wherein, The total loss function is expressed as: ， wherein, represents a total loss function; represents a weight coefficient, , represents an image quality loss function, and L1 is a mean absolute error.

5. The method of claim 1, wherein, In step 1, SENet is used as the channel attention network module to assign weights to the feature vectors of each location coordinate; Swin-Transformer is used as the spatial attention network module to solve the positional correlation between feature vectors of different location coordinates.

6. The method of claim 5, wherein, The so-called orthogonalization specifically includes that three matrices in the three-plane data each form a gridded plane and are arranged orthogonally to each other in space to form a gridded cubic space, and the three-plane data after orthogonalization is represented as, that is, the three matrices in the three-plane data structure correspond to the gridded xy plane, yz plane and xz plane in space respectively, and the corresponding relationship is determined at initialization and is consistent all the time.

7. The method of claim 6, wherein, The first attention feature vector and the second attention feature vector are feature values corresponding to the position coordinates in the grid space, that is , respectively represent the feature vectors corresponding to the projections of the position coordinates on the xy plane, the yz plane and the xz plane. The specific values are obtained by a double interpolation method.

8. The method of claim 7, wherein, The process of calculating the position weight coefficient corresponding to the position coordinate is as follows: ， ， ， ， ， ， ， wherein, represents original tri-plane data, represents a channel attention neural network in a channel attention network module, represents orthogonalization, represents channel tri-plane data obtained by passing the original tri-plane data through the channel attention neural network, represents the orthogonalized channel tri-plane, represents a first attention feature vector, represents a spatial attention neural network in a spatial attention network module, represents spatial tri-plane data obtained by passing the original tri-plane data through the spatial attention neural network, represents the orthogonalized spatial tri-plane, represents a second attention feature vector, represents a vector concatenation operation, and MLP represents a perception process of a multi-layer perception machine, represents a position weight coefficient.

9. The method of claim 2, wherein, In step 4, the initial rotation coefficient, the initial scaling coefficient and the initial opacity of the 3D Gaussian point cloud are respectively optimized by the position weight coefficient to obtain the corresponding optimized rotation coefficient, optimized scaling coefficient and optimized opacity, and the specific formula is as follows: ； ； ； wherein, is a position weight coefficient, denotes an initial opacity, denotes an initial scaling coefficient, denotes an initial rotation coefficient, denotes an optimized opacity, optimizes a scaling coefficient, denotes an optimized rotation coefficient.

10. The method of claim 9, wherein, In step 4, the Gaussian adaptive density control is performed based on the position coordinates of the 3D Gaussian point cloud, the optimized parameters and the covariance matrix, and the 3D Gaussian point cloud obtained after the density control is rendered to obtain a final rendering image.

Images