Dynamic three-dimensional scene representation method based on Gaussian splashing

Abstract

The invention discloses a dynamic three-dimensional scene representation method based on Gaussian splashing, and belongs to the field of machine vision. A first time feature constructed through a first neural network based on the time index is fused with the Gaussian feature of the original Gaussian point cloud, and then decoding is performed to obtain various parameters of the original Gaussian point cloud; and fusing a second time feature obtained through second neural network coding based on the time index with the feature obtained by the micro renderer, so that time domain information is introduced in the image reconstruction process to assist in constructing the incidence relation of adjacent video frames, and the time domain expression capability of the model is further enhanced. According to the method, the convolutional neural network is combined with Gaussian splash, the modeling capability of a model for a dynamic scene can be effectively enhanced by fusing time domain features, and efficient expression of a dynamic three-dimensional scene is realized; time domain redundancy is eliminated by modeling a time domain incidence relation, and efficient expression of a dynamic scene is achieved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of machine vision, and more particularly relates to a dynamic three-dimensional scene representation method based on Gaussian splatting. BACKGROUND

[0002] A three-dimensional scene can be represented in various forms, such as a point cloud, a mesh, a multi-view, a neural radiance field, etc. A three-dimensional Gaussian point cloud is a new three-dimensional scene representation method that can achieve real-time rendering of photo-level images and is suitable for representing and compressing three-dimensional video.

[0003] A three-dimensional Gaussian point cloud is composed of a large number of basic Gaussian point clouds, each of which contains position coordinates, opacity, size, rotation, color, etc.

[0004] Gaussian splatting can effectively represent a three-dimensional scene and achieve high-quality real-time rendering, but Gaussian splatting models a static three-dimensional scene and lacks a time-domain modeling mechanism for a dynamic three-dimensional scene, resulting in poor modeling quality for a dynamic three-dimensional scene. SUMMARY

[0005] In view of the above defects or improvement needs of the prior art, the present application provides a dynamic three-dimensional scene representation method based on Gaussian splatting, which can effectively represent a dynamic three-dimensional scene.

[0006] To achieve the above-mentioned purpose, according to a first aspect of the present application, a dynamic three-dimensional scene representation method based on Gaussian splatting is provided, comprising:

[0007] Based on the two-dimensional images of the three-dimensional scene at time t in the immersive video under different camera poses, a three-dimensional standard point cloud of the three-dimensional scene is generated, and a three-dimensional Gaussian point cloud of the three-dimensional scene is initialized according to the three-dimensional standard point cloud; t = 1, 2, …, T; T is the duration of the immersive video;

[0008] inputting the time t into a first neural network to obtain a first time feature f t1 , inputting the Gaussian feature f g of the three-dimensional Gaussian point cloud into a second neural network to obtain a second time feature f t1 , and fusing f fuse and f fuse to obtain a first fused feature f ;

[0009] According to the parameters of the three-dimensional Gaussian point cloud, the three-dimensional Gaussian point cloud is projected onto the two-dimensional plane corresponding to each camera pose, respectively, to obtain a two-dimensional Gaussian point cloud corresponding to each camera pose; rasterizing the two-dimensional Gaussian point cloud corresponding to each camera pose to obtain a feature map f i ; inputting the time t into a second neural network to obtain a second time feature f t2and f t2 and f i fusion to obtain a second fusion feature f o ;

[0010] f o is input into a reconstruction network for reconstruction to obtain a rendered image, and a 4D GS model including the first and second neural networks and the reconstruction network and a three-dimensional Gaussian point cloud are optimized by minimizing a difference loss between the rendered image and the two-dimensional image under each camera pose;

[0011] The optimized three-dimensional Gaussian point cloud is used as a representation of the three-dimensional scene at time t in the immersive video.

[0012] According to a second aspect of the present application, an electronic device is provided, comprising: a computer readable storage medium and a processor;

[0013] The computer readable storage medium is configured to store executable instructions;

[0014] The processor is configured to read the executable instructions stored in the computer readable storage medium and execute the method according to the first aspect.

[0015] According to a third aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for causing a processor to execute the method according to the first aspect.

[0016] According to a fourth aspect of the present application, a computer program product is provided, comprising computer programs or instructions, which, when executed by a processor, implement the method according to the first aspect.

[0017] Overall, the above technical solutions conceived by the present application can achieve the following beneficial effects compared with the prior art:

[0018] The dynamic 3D scene representation method based on Gaussian splashing provided by this invention can model immersive videos as neural radiation fields based on Gaussian splashing. It uses neural networks to implicitly model temporal correlations, taking the parameters of static Gaussian point clouds as Gaussian features. Considering the characteristics of dynamic scenes, in addition to the Gaussian features contained in the static Gaussian point clouds, a first temporal feature is constructed based on a time index through a first MLP network. The Gaussian features and the first temporal features are fused to obtain Gaussian features containing temporal information, and the fused feature is input into the neural network to calculate various parameters of the original Gaussian point cloud. In addition, to further improve the model's temporal expressiveness, a temporal encoding branch is introduced. The second temporal feature obtained by encoding the time index information through a second MLP network is fused with the features obtained by the differentiable renderer, thereby introducing temporal information into the image reconstruction process to assist in constructing the correlation between adjacent video frames, further enhancing the model's temporal expressiveness. The method provided by this invention combines convolutional neural networks with Gaussian splashing. By fusing temporal features, it effectively enhances the model's ability to model dynamic scenes, achieving efficient representation of dynamic 3D scenes. This method can effectively model temporal correlations, eliminate temporal redundancy, and achieve efficient representation of dynamic scenes. Compared with existing static 3D scene modeling methods, the method provided by this invention can solve problems such as dynamic blur and temporal discontinuity, achieving high-quality rendering of dynamic 3D scenes.

[0019] As a further preferred embodiment, the method provided by the present invention also employs a third MLP network to extract temporal features of the time index information to participate in image reconstruction. This can strengthen the guidance of temporal information during the image reconstruction process, avoid the loss of temporal information, effectively enhance the model's temporal modeling capability, and improve temporal continuity. Attached Figure Description

[0020] Figure 1 A flowchart of a dynamic 3D scene representation method based on Gaussian splashing provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0022] This invention provides a method for representing dynamic 3D scenes based on Gaussian splashing, such as... Figure 1 As shown, it includes:

[0023] Based on the 2D images of the 3D scene at time t in the immersive video under different camera poses, a 3D standard point cloud of the 3D scene is generated, and a 3D Gaussian point cloud of the 3D scene is initialized according to the 3D standard point cloud; t = 1, 2, ..., T; T is the duration of the immersive video;

[0024] The time t is input into the first neural network to obtain the first time feature f. t1 The Gaussian features f of the three-dimensional Gaussian point cloud g with f t1 The first fusion feature f is obtained through fusion. fuse and for f fuse Decoding is performed to obtain the parameters of the three-dimensional Gaussian point cloud;

[0025] Based on the parameters of the 3D Gaussian point cloud, the 3D Gaussian point cloud is projected onto the 2D plane corresponding to each camera pose, respectively, to obtain the 2D Gaussian point cloud corresponding to different camera poses; the 2D Gaussian point cloud corresponding to each camera pose is rasterized to obtain the feature map f. i The time t is input into the second neural network to obtain the second time feature f. t2 and f t2 with f i The fusion yields the second fusion feature f o ;

[0026] f o The image is input into the reconstruction network to reconstruct and obtain a rendered image. By minimizing the difference loss between the rendered image and the 2D image at each camera pose, the 4DGS model and the 3D Gaussian point cloud, including the first and second neural networks and the reconstruction network, are optimized.

[0027] The optimized 3D Gaussian point cloud is used as the representation of the 3D scene at time t in the immersive video.

[0028] It is understandable that one time point t corresponds to one video frame.

[0029] Understandably, algorithms such as SFM, Colmap, HSfM, and PSfM can be used to generate the standard 3D point cloud of this 3D scene.

[0030] Preferably, the 4DGS model further includes a third neural network for extracting a third time feature f at time t. t3 The third time feature f t3 With the second fusion feature f o The images are fed into the NeRGS reconstruction network for reconstruction to obtain the rendered image.

[0031] Preferably, before inputting time t into the first neural network and before inputting time t into the second neural network, the method further includes:

[0032] Position encoding is performed at time t.

[0033] Preferably, the 4DGS model further includes a fourth neural network for analyzing f. fuse The parameters of the three-dimensional Gaussian point cloud are obtained by decoding.

[0034] The first to fourth neural networks can be any existing neural network, such as an MLP network, and the embodiments of the present invention do not limit this to a single type.

[0035] The following section, using the example that the first to fourth neural networks are all MLP networks, further explains the specific process of the dynamic 3D scene representation method based on Gaussian splashing provided by this invention:

[0036] 1. Calculate the first time feature f based on the time index t. t1 and the first time feature f t1 Gaussian features f of 3D Gaussian point cloud g The fusion results in the fusion feature f. fuse :

[0037] f t1 =MLP t1 (t)

[0038]

[0039] Among them, MMLP t1 (·) is used to obtain the first time feature f t1 The first MLP network, f t1 The first temporal features obtained by encoding the MLP network.

[0040] 2. Utilizing the decoding and fusion features of the fourth MLP network f fuse The parameters of the Gaussian point cloud are obtained. Each Gaussian point cloud includes position coordinates, size, rotation, color features and opacity.

[0041] M, S, R, F, A = MLP(f fuse )

[0042] Where M represents the coordinate attribute of the Gaussian point cloud, S represents the size attribute of the Gaussian point cloud, R represents the rotation attribute of the Gaussian point cloud, F represents the color attribute of the Gaussian point cloud, A represents the opacity attribute of the Gaussian point cloud, and MLP(·) represents the third MLP network used to obtain the various attributes of the Gaussian point cloud.

[0043] 3. Given a camera pose V and a transformation matrix W from world coordinates to camera coordinates, project a 3D Gaussian point cloud onto a 2D plane to obtain a 2D Gaussian point cloud. Calculate the coordinates of the projected 2D Gaussian point cloud. and covariance matrix

[0044] ∑=RSS T R T

[0045]

[0046] Where R is the rotation attribute of the Gaussian point cloud, S is the size attribute of the Gaussian point cloud, W is the transformation matrix from world coordinates to camera coordinates, J is the Jacobian matrix, M is the coordinate attribute of the Gaussian point cloud, V is the given camera pose, and Projection(·) is the process of projecting the 3D Gaussian point cloud onto a 2D plane.

[0047] 4. The feature map f is obtained by rasterizing the two-dimensional Gaussian point cloud using a differentiable fragment renderer. i :

[0048]

[0049] in, The coordinates of the projected Gaussian point cloud are... Let F be the projected covariance matrix, F be the color attribute of the Gaussian point cloud, A be the opacity attribute of the Gaussian point cloud, and V be the given camera pose.

[0050] The feature value f of a single pixel during rasterization can be calculated using Alpha Blending:

[0051]

[0052] α i =A i G i (x)

[0053]

[0054] Among them, M i Let A be the coordinates of the i-th Gaussian point cloud. i Let F be the opacity of the i-th Gaussian point cloud. i Let N be the color feature of the i-th Gaussian point cloud, and N be the number of Gaussian point clouds.

[0055] 5. Calculate the second time feature f based on the time index. t2 The feature map f obtained by the rasterization process i With time branch features f t2 The fusion results in the fusion feature f.o The input to the NeRGS reconstruction network yields the rendered image.

[0056] f t2 =MLP t2 (t)

[0057]

[0058] Among them, MLP t2 (·) represents obtaining the second time feature f t2 The second MLP network, NeRGS(·), represents the implicit reconstruction network. For the final rendered image.

[0059] Considering only the second fusion feature f is input o Since temporal information decay may occur after multiple reconstruction network blocks, it is necessary to introduce temporal features between reconstruction network blocks to enhance temporal information. Based on this, preferably, the 4DGS model also includes a third MLP network, MLP. t3 The third time feature f is used to extract the time t. t3 The third time feature f t3 With the second fusion feature f o The images are fed into the NeRGS reconstruction network for reconstruction to obtain the rendered image.

[0060] The above is the complete rendering process. The following is the 4DGS model optimization process. The loss between the rendered image and the real image (i.e., the 2D image) is calculated, and the gradient is updated using backpropagation. The gradient descent algorithm is used to optimize the network parameters of the 4DGS model, including the first to fourth MLP networks and the reconstruction network NeRGS, as well as the feature attributes of the 3D Gaussian point cloud.

[0061] Understandably, the difference loss between the reconstructed image and the two-dimensional image can be calculated using L1 loss function, L2 loss function, SSIM loss function, perceptual loss function, etc.

[0062] Position encoding of time t maps time t to a high-dimensional space, thereby introducing high-frequency information into the input data. This avoids the model overfitting to low-frequency information and allows the model to better express high-frequency information. Therefore, preferably, before inputting time t into the first neural network and the second neural network, the method further includes:

[0063] Position encoding is performed at time t.

[0064] The process of position encoding at time t includes: given a sequence of video frames, normalizing the input video frame time index t to [0,1], and performing position encoding using the following formula:

[0065] PE(t)=(sin(b 0 πt), cos(b) 0 πt), ..., sin(b) l-1 πt), cos(b) l-1 πt))

[0066] Where b and l are hyperparameters, and PE(t) is the encoded frame index.

[0067] The reconstruction network can be a cascaded network structure that includes upsampling layers (such as interpolation, deconvolution, subpixel convolution, up pooling, etc.).

[0068] Preferably, in one optional implementation, the reconstructed network includes: a cascaded first convolutional layer, a first activation layer, a GS-NeRV module, a second convolutional layer, and a second activation layer;

[0069] The GS-NeRV module includes one GS-NeRV block or multiple cascaded GS-NeRV blocks; the GS-NeRV block includes: cascaded grouped convolutional layers, attention mechanism modules, upsampling layers and ordinary convolutional layers.

[0070] Preferably, in one optional implementation, the attention mechanism module is a convolutional block attention mechanism module (CBAM); the upsampling layer is a pixel shuffle model.

[0071] In one optional implementation, the first convolutional layer is a convolutional layer with a kernel size of 1×1 for both the first and second convolutional layers; the first activation layer is a Gelu activation layer; and the second convolutional layer is a Sigmoid activation layer.

[0072] Preferably, the reconstruction network further includes a regularization layer located between the first convolutional layer and the first activation layer. The regularization layer can standardize the input data, enabling the reconstruction network to be trained faster and more efficiently.

[0073] This invention provides an electronic device, including: a computer-readable storage medium and a processor;

[0074] The computer-readable storage medium is used to store executable instructions;

[0075] The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any of the above embodiments.

[0076] This invention provides a computer-readable storage medium storing computer instructions that cause a processor to perform the method described in any of the above embodiments.

[0077] This invention provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the method described in any of the above embodiments.

[0078] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A dynamic 3D scene representation method based on Gaussian splashing, characterized in that, include: Based on the 2D images of the 3D scene at time t in the immersive video under different camera poses, a 3D standard point cloud of the 3D scene is generated, and a 3D Gaussian point cloud of the 3D scene is initialized according to the 3D standard point cloud; t = 1, 2, ..., T; T is the duration of the immersive video; The time t is input into the first neural network to obtain the first time feature f. t1 The Gaussian features f of the three-dimensional Gaussian point cloud g with f t1 The first fusion feature f is obtained through fusion. fuse and for f fuse Decoding is performed to obtain the parameters of the three-dimensional Gaussian point cloud; Based on the parameters of the 3D Gaussian point cloud, the 3D Gaussian point cloud is projected onto the 2D plane corresponding to each camera pose, respectively, to obtain the 2D Gaussian point cloud corresponding to different camera poses; the 2D Gaussian point cloud corresponding to each camera pose is rasterized to obtain the feature map f. i The time t is input into the second neural network to obtain the second time feature f. t2 and f t2 with f i The fusion yields the second fusion feature f o ; f o The image is input into the reconstruction network to reconstruct and obtain a rendered image. By minimizing the difference loss between the rendered image and the 2D image at each camera pose, the 4DGS model and the 3D Gaussian point cloud, including the first and second neural networks and the reconstruction network, are optimized. The optimized 3D Gaussian point cloud is used as the representation of the 3D scene at time t in the immersive video.

2. The method as described in claim 1, characterized in that, The 4DGS model also includes a third neural network for extracting the third time feature f at time t. t3 The third time feature f t3 With the second fusion feature f o The images are fed into the NeRGS reconstruction network for reconstruction to obtain the rendered image.

3. The method as described in claim 1 or 2, characterized in that, Before inputting time t into the first neural network and before inputting time t into the second neural network, the method further includes: Position encoding is performed at time t.

4. The method as described in claim 1 or 2, characterized in that, The 4DGS model also includes a fourth neural network for analyzing f. fuse The parameters of the three-dimensional Gaussian point cloud are obtained by decoding.

5. The method as described in claim 1 or 2, characterized in that, The first to fourth neural networks are all MLP networks.

6. The method according to claim 1 or 2, characterized in that, The reconstructed network includes: a cascaded first convolutional layer, a first activation layer, a GS-NeRV module, a second convolutional layer, and a second activation layer; The GS-NeRV module includes one GS-NeRV block or multiple cascaded GS-NeRV blocks; the GS-NeRV block includes: cascaded grouped convolutional layers, attention mechanism modules, upsampling layers and ordinary convolutional layers.

7. The method according to claim 6, characterized in that, The attention mechanism module is a convolutional block attention mechanism module; the upsampling layer is a Pixel Shuffle model.

8. An electronic device, characterized in that, include: Computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any one of claims 1-5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a processor to perform the method as described in any one of claims 1-5.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1-5.

Images