A three-dimensional scene representation method, device, storage medium and program product

Abstract

The application discloses a three-dimensional scene representation method, device, storage medium and program product, and belongs to the technical field of machine vision; considering that a large amount of redundancy exists in the distribution of a Gaussian point cloud in an existing method for representing a three-dimensional scene based on a 3D Gaussian sputtering generated Gaussian point cloud, a reconstruction network is used to restore a photo-level image rendering, so that the rasterization process only needs to output a feature map with a small resolution, the densification process of the Gaussian point cloud is limited in the model optimization process, the number of the Gaussian point cloud is greatly reduced, the storage space required for three-dimensional scene representation can be further reduced under the premise of ensuring accuracy, and in addition, after the three-dimensional Gaussian point cloud and the reconstruction network are optimized, the optimized three-dimensional Gaussian point cloud and the reconstruction network can be used to obtain a visual representation of a three-dimensional scene under a camera pose specified by a user, and more intuitive and effective three-dimensional scene representation is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of machine vision technology, and more specifically, relates to a three-dimensional scene representation method, device, storage medium and program product. Background Art

[0002] Three-dimensional scene representation methods can capture effective information about the entire three-dimensional scene and store the three-dimensional spatial information in a compact data format, making it easy to transmit and present the entire three-dimensional scene. These methods have wide applications in virtual reality (VR), augmented reality (AR), autonomous driving, and other fields. Therefore, research on a three-dimensional scene representation method is of great significance.

[0003] Three-dimensional scenes can be represented in a variety of forms, such as point clouds, meshes, multi-views, and neural radiance fields. Three-dimensional Gaussian point clouds are a novel representation method for 3D scenes. They are composed of a large number of basic Gaussian point clouds, each containing attributes such as position coordinates, opacity, size, rotation, and color. They can effectively express 3D scenes and achieve high-quality real-time rendering. However, the number of Gaussian point clouds is enormous, typically requiring hundreds of thousands or even millions of Gaussian point clouds to represent a 3D scene. This requires a large amount of storage space, posing significant challenges to storage and transmission. Summary of the Invention

[0004] In response to the above-mentioned defects or improvement needs of the prior art, the present invention provides a three-dimensional scene representation method, device, storage medium and program product, the purpose of which is to provide a more efficient three-dimensional scene expression to reduce the number of Gaussian point clouds, and further reduce the storage space required for three-dimensional scene representation while ensuring accuracy.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a three-dimensional scene representation method, comprising:

[0006] Generate a 3D standard point cloud of the 3D scene based on the 2D images of the 3D scene at different camera positions; where the size of the 2D image is W×H, and W and H are both positive integers;

[0007] Initialize the 3D Gaussian point cloud of the 3D scene based on the 3D standard point cloud;

[0008] The 3D Gaussian point cloud is projected onto the 2D plane corresponding to each camera pose to obtain the 2D Gaussian point cloud corresponding to different camera poses; the 2D Gaussian point cloud corresponding to each camera pose is rasterized to obtain the corresponding feature map of size αW×βH, which is then input into the reconstruction network for reconstruction to obtain a reconstructed image of size W×H; the parameters of the reconstruction network and the 3D Gaussian point cloud are optimized by minimizing the difference loss between the reconstructed image and the 2D image at each camera pose; 0<α<1, 0<β<1;

[0009] The optimized three-dimensional Gaussian point cloud is used as the representation of the three-dimensional scene.

[0010] Further preferably, the above three-dimensional scene representation method further includes:

[0011] Obtain a set of camera poses specified by the user, project the optimized 3D Gaussian point clouds onto the 2D planes corresponding to different camera poses in the set, and obtain the 2D Gaussian point clouds corresponding to the different camera poses; rasterize the 2D Gaussian point clouds corresponding to each camera pose, obtain the corresponding feature maps of size αW×βH, and input them into the optimized reconstruction network for reconstruction to obtain the corresponding reconstructed images;

[0012] The reconstructed image group corresponding to the group of camera poses is used as a visual representation of the three-dimensional scene.

[0013] Further preferably, the reconstruction network comprises: a cascaded first convolutional layer, a first activation layer, a GS-NeRV module, a second convolutional layer, and a second activation layer;

[0014] Among them, the GS-NeRV module includes one GS-NeRV block, or multiple cascaded GS-NeRV blocks; the GS-NeRV block includes: a cascaded group convolution layer, an attention mechanism module, an upsampling layer, and a normal convolution layer.

[0015] Further preferably, the attention mechanism module is a convolutional block attention mechanism module; and the upsampling layer is a PixelShuffle model.

[0016] Further preferably, the above reconstruction network further includes: a regularization layer located between the first convolutional layer and the first activation layer.

[0017] Further preferably, the parameters of the reconstruction network and the three-dimensional Gaussian point cloud are optimized, including:

[0018] The difference loss is back-propagated to the reconstruction network to optimize the parameters of the reconstruction network, and then back-propagated to the 3D Gaussian point cloud through the rasterization and projection process to optimize the properties and distribution of the 3D Gaussian point cloud;

[0019] Through the optimization of preset rounds, the difference loss is minimized.

[0020] Further preferably, based on the two-dimensional images of the three-dimensional scene at different camera poses, the SFM algorithm is used to generate a three-dimensional standard point cloud of the three-dimensional scene.

[0021] In a second aspect, the present invention provides an electronic device comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the three-dimensional scene representation method provided in the first aspect of the present invention when executing the computer program.

[0022] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the three-dimensional scene representation method provided in the first aspect of the present invention is executed.

[0023] In a fourth aspect, the present invention further provides a computer program product, comprising a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the steps of the three-dimensional scene representation method provided in the first aspect are implemented.

[0024] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0025] 1. The present invention provides a three-dimensional scene representation method. Considering that there is a lot of redundancy in the distribution of Gaussian point clouds in existing methods of representing three-dimensional scenes based on Gaussian point clouds generated by 3D Gaussian sputtering, a reconstruction network is used to restore photo-level image rendering, so that the rasterization process only needs to output feature maps with a smaller resolution, thereby limiting the densification process of the Gaussian point cloud during the model optimization process, greatly reducing the number of Gaussian point clouds, and further reducing the storage space required for three-dimensional scene representation while ensuring accuracy.

[0026] 2. Furthermore, the three-dimensional scene representation method provided by the present invention can project the optimized three-dimensional Gaussian point cloud onto two-dimensional planes corresponding to different camera poses in a set of camera poses specified by the user after the optimization of the three-dimensional Gaussian point cloud and the reconstruction network. After rasterization, the obtained feature map is input into the optimized reconstruction network for reconstruction to obtain the corresponding reconstructed image, and then a visual representation of the three-dimensional scene under the camera pose specified by the user is obtained, thereby achieving a more intuitive and effective three-dimensional scene representation.

[0027] 3. Furthermore, in the three-dimensional scene representation method provided by the present invention, the reconstruction network is constructed based on the GS-NeRV block, wherein the GS-NeRV block includes: a cascaded group convolution layer, an attention mechanism module, an upsampling layer and a normal convolution layer; wherein the group convolution layer, the attention mechanism module and the normal convolution layer can well restore the image color information from the image features, and the upsampling layer can restore the low-resolution image to the high-resolution image; based on this, the reconstruction network can restore the image features to the image color information while improving the image resolution, thereby being able to well restore the texture information in the three-dimensional scene with higher rendering quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flow chart of a three-dimensional scene representation method provided by the present invention;

[0029] Figure 2 A schematic diagram of the structure of a reconstruction network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0031] In order to achieve the above objectives, in a first aspect, the present invention provides a three-dimensional scene representation method, such as Figure 1 Shown, including:

[0032] S1. Generate a 3D standard point cloud of the 3D scene based on 2D images of the 3D scene at different camera poses. The 2D image size is W × H. Both W and H are positive integers. In one optional embodiment, W is 1920 and H is 1080. Specifically, the attributes of the 3D standard point cloud include position coordinates and color.

[0033] It should be noted that the 3D standard point cloud of the 3D scene can be generated by using the SFM algorithm, Colmap algorithm, HSfM algorithm, PSfM algorithm, etc. Preferably, in an optional implementation, the 3D standard point cloud of the 3D scene is generated by using the SFM algorithm.

[0034] S2, initializing the 3D Gaussian point cloud of the 3D scene based on the 3D standard point cloud;

[0035] A 3D Gaussian point cloud is composed of a large number of basic Gaussian point clouds. Each Gaussian point cloud contains attributes such as position coordinates M, opacity A, size S, rotation R, and color F. A Gaussian point cloud is represented in 3D space by a 3D covariance matrix ∑ and a center point μ:

[0036]

[0037] Where x is the coordinate of a point in three-dimensional space.

[0038] S3, project the three-dimensional Gaussian point cloud onto the two-dimensional plane corresponding to each camera pose to obtain a two-dimensional Gaussian point cloud corresponding to each camera pose; rasterize the two-dimensional Gaussian point cloud corresponding to each camera pose to obtain a feature map with a size of aW x bH, and input the feature map into the reconstruction network for reconstruction to obtain a reconstructed image with a size of W x H; optimize the parameters of the reconstruction network and the three-dimensional Gaussian point cloud by minimizing the difference loss between the reconstructed image and the two-dimensional image under each camera pose; 0 < a < 1, 0 < b < 1; in an optional implementation, a is 0.5 and b is 0.5.

[0039] Specifically, for a camera pose V, the conversion matrix from the world coordinate to the camera coordinate is W, the three-dimensional Gaussian point cloud is projected onto the two-dimensional plane corresponding to the camera pose V, and the coordinates in the projected camera coordinates are calculated and the covariance matrix are as follows:

[0040]

[0041] where J is the Jacobian matrix of the projective transformation affine approximation, which is used to convert a three-dimensional vector into a two-dimensional vector. In order to keep the covariance matrix semi-definite during optimization, the covariance matrix is decomposed into a scaling matrix S (i.e. the size information of the Gaussian point cloud) and a rotation matrix R (i.e. the rotation information of the Gaussian point cloud), specifically: ∑ = RSS T R T .

[0042] Rasterize the two-dimensional Gaussian point cloud to obtain a feature map with a size of aW x bH

[0043] During the rasterization process, different pixel points will be divided, and each pixel point will correspond to a two-dimensional Gaussian point cloud subset in the pixel point region. The two-dimensional Gaussian point cloud subsets corresponding to different pixel points may have intersections. The feature value f w,h of a single pixel point in the feature map (1 ≤ w ≤ aW, 1 ≤ h ≤ bH) can be calculated by the AlphaBlending algorithm, specifically:

[0044]

[0045] where a i = A i G(x);

[0046] N w,h is the two-dimensional Gaussian point cloud subset corresponding to the pixel; F i is the feature value of the i-th two-dimensional Gaussian point cloud in N w,h .i is N w,h opacity of the i-th two-dimensional Gaussian point cloud in the N i is N w,h center point coordinates of the i-th two-dimensional Gaussian point cloud in the N

[0047] It should be noted that the reconstruction network can be a cascaded network structure containing an upsampling layer (such as interpolation, deconvolution, sub-pixel convolution, and up-pooling, etc.). Preferably, in an optional embodiment, the reconstruction network comprises a cascaded first convolutional layer, a first activation layer, a GS-NeRV module, a second convolutional layer, and a second activation layer.

[0048] The GS-NeRV module comprises one GS-NeRV block or a plurality of cascaded GS-NeRV blocks; the GS-NeRV block comprises a cascaded grouped convolutional layer, an attention mechanism module, an upsampling layer, and a normal convolutional layer.

[0049] Preferably, in an optional embodiment, the attention mechanism module is a convolutional block attention mechanism module CBAM; and the upsampling layer is a Pixel Shuffle model.

[0050] In an optional embodiment, the first convolutional layer and the second convolutional layer are convolutional layers with a kernel size of 1x1; the first activation layer is a Gelu activation layer; and the second convolutional layer is a Sigmoid activation layer.

[0051] Preferably, the reconstruction network further comprises a regularization layer between the first convolutional layer and the first activation layer. The regularization layer can standardize the input data, so that the reconstruction network can be trained more quickly and efficiently.

[0052] Figure 2 FIG. 1 is a structural schematic diagram of the reconstruction network in an optional embodiment.

[0053] In an optional embodiment, the parameters of the reconstruction network and the three-dimensional Gaussian point cloud are optimized, comprising:

[0054] The difference loss is back-propagated to the reconstruction network to optimize the parameters of the reconstruction network, and then back-propagated to the three-dimensional Gaussian point cloud through rasterization and projection processes to optimize the attributes and distribution of the three-dimensional Gaussian point cloud;

[0055] Through the optimization of a preset number of rounds, the minimization of the difference loss is achieved.

[0056] Specifically, a difference loss between the reconstructed image and the two-dimensional image is calculated, and an update gradient is back propagated to optimize the parameters of the reconstruction network and the attributes of the three-dimensional Gaussian point cloud using a gradient descent method, and the distribution of the three-dimensional Gaussian point cloud is adaptively controlled in the optimization process. Through the optimization of the preset number of rounds, the adaptive density control is performed on the three-dimensional Gaussian point cloud, so that the distribution of the Gaussian point cloud is more consistent with the three-dimensional scene structure.

[0057] It should be noted that the difference loss between the reconstructed image and the two-dimensional image can be calculated using an L1 loss function, an L2 loss function, an SSIM loss function, a perceptual loss function, etc.

[0058] S3, taking the optimized three-dimensional Gaussian point cloud as the representation of the three-dimensional scene.

[0059] In summary, the present application considers that there is a large amount of redundancy in the distribution of Gaussian point cloud in 3D Gaussian sputtering, and uses a reconstruction network to restore a photo-level image rendering, so that the rasterization process of 3D Gaussian sputtering only needs to output a feature map with a smaller resolution, thereby limiting the densification process of the Gaussian point cloud in the model optimization process, greatly reducing the number of Gaussian point clouds, and further reducing the storage space required for the three-dimensional scene representation under the premise of ensuring accuracy.

[0060] In an optional embodiment, the above three-dimensional scene representation method further comprises:

[0061] obtaining a group of camera poses specified by a user, projecting the optimized three-dimensional Gaussian point cloud into a two-dimensional plane corresponding to different camera poses in the group of camera poses to obtain two-dimensional Gaussian point clouds corresponding to the different camera poses, respectively, rasterizing the two-dimensional Gaussian point cloud corresponding to each camera pose to obtain a corresponding feature map with a size of aW x bH, and inputting the feature map into the optimized reconstruction network for reconstruction to obtain a corresponding reconstructed image;

[0062] taking a group of reconstructed images corresponding to the group of camera poses as a visual representation of the three-dimensional scene.

[0063] Through the above process, a more intuitive and effective three-dimensional scene representation under different camera poses can be obtained.

[0064] In a second aspect, the present application provides an electronic device comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to execute the three-dimensional scene representation method provided in the first aspect of the present application.

[0065] The related technical solution is the same as the three-dimensional scene representation method provided in the first aspect of the present application, which will not be repeated here.

[0066] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the three-dimensional scene representation method provided in the first aspect of the present invention is executed.

[0067] The related technical solutions are the same as the three-dimensional scene representation method provided in the first aspect of the present invention, and will not be described in detail here.

[0068] In a fourth aspect, the present invention further provides a computer program product, comprising a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the steps of the three-dimensional scene representation method provided in the first aspect are implemented.

[0069] The related technical solutions are the same as the three-dimensional scene representation method provided in the first aspect of the present invention, and will not be described in detail here.

[0070] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A three-dimensional scene representation method, characterized in that: include: Generate a 3D standard point cloud of the 3D scene based on 2D images of the 3D scene at different camera positions; the size of the 2D image is W×H; W and H are both positive integers; Initialize a three-dimensional Gaussian point cloud of the three-dimensional scene according to the three-dimensional standard point cloud; Project the three-dimensional Gaussian point cloud onto the two-dimensional plane corresponding to each camera pose to obtain the two-dimensional Gaussian point cloud corresponding to different camera poses; Rasterize the two-dimensional Gaussian point cloud corresponding to each camera pose to obtain the corresponding size α W× β After the feature map of H is obtained, it is input into the reconstruction network for reconstruction to obtain a reconstructed image of size W×H; by minimizing the difference loss between the reconstructed image and the two-dimensional image at each camera pose, the parameters of the reconstruction network and the three-dimensional Gaussian point cloud are optimized; , ; The optimized three-dimensional Gaussian point cloud is used as the representation of the three-dimensional scene; The optimization of the parameters of the reconstruction network and the three-dimensional Gaussian point cloud includes: Back-propagating the difference loss to the reconstruction network to optimize the parameters of the reconstruction network, and then back-propagating it to the three-dimensional Gaussian point cloud through rasterization and projection processes to optimize the properties and distribution of the three-dimensional Gaussian point cloud; The difference loss is minimized through preset rounds of optimization.

2. The three-dimensional scene representation method according to claim 1, characterized in that: Also includes: Get a set of camera poses specified by the user, project the optimized 3D Gaussian point cloud onto the 2D plane corresponding to different camera poses in the set of camera poses, and obtain the 2D Gaussian point cloud corresponding to different camera poses; rasterize the 2D Gaussian point cloud corresponding to each camera pose, and obtain the corresponding size α W× β After the feature map of H is obtained, it is input into the optimized reconstruction network for reconstruction to obtain the corresponding reconstructed image; The reconstructed image group corresponding to the group of camera poses is used as a visual representation of the three-dimensional scene.

3. The three-dimensional scene representation method according to claim 1 or 2, characterized in that: The reconstruction network includes: a cascaded first convolutional layer, a first activation layer, a GS-NeRV module, a second convolutional layer and a second activation layer; The GS-NeRV module includes a GS-NeRV block or multiple cascaded GS-NeRV blocks; the GS-NeRV block includes: a cascaded group convolution layer, an attention mechanism module, an upsampling layer and a normal convolution layer.

4. The three-dimensional scene representation method according to claim 3, characterized in that: The attention mechanism module is a convolutional block attention mechanism module; the upsampling layer is a Pixel Shuffle model.

5. The three-dimensional scene representation method according to claim 3, characterized in that: The reconstruction network further includes: a regularization layer located between the first convolutional layer and the first activation layer.

6. The three-dimensional scene representation method according to claim 1 or 2, characterized in that: Based on the 2D images of the 3D scene at different camera poses, the SFM algorithm is used to generate the 3D standard point cloud of the 3D scene.

7. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the three-dimensional scene representation method according to any one of claims 1 to 6 when executing the computer program.

8. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the three-dimensional scene representation method according to any one of claims 1 to 6 is executed.

9. A computer program product, characterized in that The method comprises a computer program or an instruction, wherein when the computer program or the instruction is executed by a processor, the steps of the three-dimensional scene representation method according to any one of claims 1 to 6 are implemented.

Images