Dynamic three-dimensional scene reconstruction and real-time rendering method and device based on free space-time Gaussian sputtering

Abstract

The invention discloses a dynamic three-dimensional scene reconstruction and real-time rendering method based on free space-time Gaussian sputtering, and the method comprises the steps: firstly obtaining a multi-view-angle dynamic scene video, extracting cross-view-angle feature points for three-dimensional reconstruction, initializing free space-time Gaussian primitives, and carrying out the real-time rendering of the free space-time Gaussian primitives; an optimizable explicit motion function and a time opacity function are set for each Gaussian primitive and are used for representing the geometry and appearance of the dynamic three-dimensional scene; then sputtering and rendering the Gaussian primitive at the current moment based on the observation visual angle, and outputting a high-fidelity dynamic scene image of the corresponding visual angle; finally, Gaussian primitive parameters are optimized in a combined mode through a rendering loss function and a four-dimensional regularization strategy, and meanwhile low-influence Gaussian primitives are relocated periodically. According to the method, the movable Gaussian primitive is introduced at any position in space and time, the explicit motion function and the time opacity function are combined, a complex dynamic scene is effectively modeled, and efficient reconstruction and real-time rendering of the dynamic three-dimensional scene are achieved.

Description

Technical Field

[0001] The present invention belongs to the field of scene rendering, and particularly relates to a method and device for dynamic three-dimensional scene reconstruction and real-time rendering based on free spatio-temporal Gaussian sputtering. Background Art

[0002] In the existing three-dimensional scene technologies, most methods focus on static scenes or the representation of dynamic scenes with small movements. Some recent dynamic scene rendering methods based on three-dimensional Gaussians can achieve real-time rendering to a certain extent by defining three-dimensional Gaussian basis elements in the standard space and introducing a deformation field to realize the mapping from the space to the observation space. However, these methods have difficulties in optimization when facing dynamic scenes with complex large-scale movements. Specifically, the deformation field needs to establish long-distance spatio-temporal correspondence relationships, resulting in the optimization process being prone to falling into local optima, which affects the rendering quality and speed. In addition, some methods improve the rendering effect by introducing additional numbers of Gaussian basis elements. However, this will lead to a significant increase in storage and computational overheads, limiting the scalability of practical applications. Summary of the Invention

[0003] The purpose of the present invention is to propose a method and device for dynamic three-dimensional scene reconstruction and real-time rendering based on free spatio-temporal Gaussian sputtering in view of the deficiencies of the prior art. By introducing movable Gaussian basis elements at any spatio-temporal position, combining explicit motion functions and time opacity functions, complex dynamic scenes are effectively modeled, and efficient reconstruction and real-time rendering of dynamic three-dimensional scenes are achieved.

[0004] The object of the present invention is achieved by the following technical solutions: A method for dynamic three-dimensional scene reconstruction and real-time rendering based on free spatio-temporal Gaussian basis elements, the method comprising:

[0005] (1) Obtain multi-view dynamic scene videos, extract cross-view feature points for three-dimensional reconstruction, initialize free spatio-temporal Gaussian basis elements, and set optimizable explicit motion functions and time opacity functions for each Gaussian basis element to represent the geometry and appearance of the dynamic three-dimensional scene.

[0006] (2) Based on the observation view, use an efficient rasterization method to sputter and render the Gaussian basis elements at the current moment, and output a high-fidelity dynamic scene image corresponding to the view;

[0007] (3) Jointly optimize the Gaussian basis element parameters using a rendering loss function and a four-dimensional regularization strategy, and simultaneously periodically relocate the low-influence Gaussian basis elements to improve the modeling efficiency.

[0008] Further, in step (1), the specific process of obtaining the multi-view dynamic scene video and initializing the free spatio-temporal Gaussian basis elements is as follows: using the feature point matching algorithm to extract cross-view matching feature points, adopting triangulation to recover the sparse three-dimensional point cloud, and combining the information of each time frame to initialize the position, time parameters, and attributes such as scale, direction, and color of the Gaussian basis elements, forming a description of the initial geometry and appearance of the scene.

[0009] Further, in step (1), the specific process of setting an explicit motion function and a time opacity function for each Gaussian basis element is as follows: assigning a linearly optimizable velocity vector to each Gaussian basis element, and establishing a motion function to describe the spatial position change of the Gaussian basis element in the time dimension; at the same time, setting the time opacity function in the form of a Gaussian distribution, and automatically adjusting the influence range of the Gaussian basis element at different time points according to the time center and duration, so as to reduce redundant representation and improve the flexibility of dynamic scene modeling.

[0010] Further, in step (2), during rendering, the rasterization and sputtering method is adopted to project the Gaussian basis elements at the corresponding position of the current time onto the image plane to achieve real-time dynamic scene rendering.

[0011] Further, in step (3), the four-dimensional regularization strategy is used to suppress the Gaussian basis elements with high opacity to prevent the optimization process from falling into a local optimum; the periodic repositioning strategy migrates the Gaussian basis elements with low influence to the dynamic detail area by comprehensively calculating the spatial gradient and opacity of the Gaussian basis elements, improving the overall expression ability.

[0012] In a second aspect, the present invention also provides a dynamic three-dimensional scene reconstruction and real-time rendering device based on free spatio-temporal Gaussian basis elements, including a memory and one or more processors. Executable code is stored in the memory, and when the processor executes the executable code, any one or more of the above methods are implemented.

[0013] In a third aspect, the present invention also provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the above method for dynamic three-dimensional scene reconstruction and real-time rendering based on free spatio-temporal Gaussian basis elements is implemented.

[0014] In a fourth aspect, the present invention also provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, the above method for dynamic three-dimensional scene reconstruction and real-time rendering based on free spatio-temporal Gaussian basis elements is implemented.

[0015] Advantages of the present invention: The present invention proposes a method for representing Gaussian basis elements in free space-time, which endows the Gaussian basis elements with the freedom to appear at any position in space-time, effectively improving the flexibility of dynamic three-dimensional scene modeling. By introducing explicit motion functions and time opacity functions, redundant representations are reduced, the optimization complexity is lowered, and efficient dynamic three-dimensional scene reconstruction and high-quality real-time rendering without relying on complex deformation fields are achieved. Further combining four-dimensional regularization and periodic repositioning strategies enhances the modeling ability for complex dynamic scenes, ensures high-fidelity rendering while achieving real-time performance, and is applicable to fields such as multi-view dynamic scene reconstruction and virtual reality. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 FIG. is a flowchart of a method for dynamic three-dimensional scene reconstruction and real-time rendering based on free space-time Gaussian sputtering provided by the present invention.

[0017] Figure 2 FIG. is a schematic diagram of the model structure for representing a dynamic scene by free space-time Gaussian sputtering of the present invention.

[0018] Figure 3 FIG. is a structural diagram of a real-time rendering device based on free space-time Gaussian sputtering of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] The following further describes the technical details and principles of the present invention with reference to the accompanying drawings:

[0020] As Figure 1 shown, the present invention proposes a method for dynamic three-dimensional scene reconstruction and real-time rendering based on free space-time Gaussian sputtering, including:

[0021] The present invention optimizes a set of free space-time Gaussian basis elements from multi-view synchronized videos to establish a representation of a dynamic three-dimensional scene; specifically: obtaining multi-view dynamic scene videos, extracting feature points for three-dimensional reconstruction, initializing free space-time Gaussian basis elements, and setting a motion function and a time opacity function for each Gaussian basis element for modeling and rendering of the dynamic three-dimensional scene.

[0022] The optimized free space-time Gaussian basis elements can be used for real-time rendering of a dynamic three-dimensional scene; specifically: rasterizing and sputtering the Gaussian basis elements at the current moment based on the input view to achieve high-fidelity real-time rendering of the dynamic three-dimensional scene, and at the same time combining a rendering loss function and a four-dimensional regularization strategy to periodically reposition the Gaussian basis elements to improve the modeling efficiency.

[0023] As Figure 2 shown, in the free space-time Gaussian basis element model proposed by the present invention, each Gaussian basis element includes the following parameters:

[0024] 1. Position parameter μx and time parameter μ t, representing the spatial position and temporal center of the Gaussian basis element;

[0025] 2. The duration parameter s controls the effective range of the Gaussian basis element in the time dimension;

[0026] 3. The velocity vector v defines the motion trajectory of the Gaussian basis element over time;

[0027] 4. The scale matrix S and rotation matrix R determine the spatial covariance matrix Σ of the Gaussian basis element;

[0028] 5. The opacity σ and spherical harmonic coefficients z are used to control the optical properties and color representation of the Gaussian basis element.

[0029] The covariance matrix Σ of the Gaussian basis element is defined as:

[0030] Σ = RSS T R T

[0031] where T represents the matrix transpose.

[0032] The motion function of the Gaussian basis element is defined as:

[0033] μx(t) = μx + v × (t - μ t )

[0034] where t represents time.

[0035] The influence range of the Gaussian basis element in the time dimension is controlled by the time opacity function, and the specific form is the Gaussian distribution function:

[0036]

[0037] As Figure 3 shown, the specific steps of the real-time rendering and optimization of the dynamic three-dimensional scene proposed by the present invention are as follows:

[0038] 1. According to the current input view, project the Gaussian basis element onto the image plane using the rasterization method, and combine multiple Gaussian basis elements through the opacity σ and spherical harmonic coefficients z to generate a real-time rendering image.

[0039] 2. Adopt rendering loss functions, including L2 loss, SSIM loss, and perceptual loss, compare the real image and the rendering image, and optimize the parameters of the Gaussian basis element through backpropagation.

[0040] 3. To avoid high-opacity basis elements from hindering gradient propagation, the present invention designs a four-dimensional regularization loss function to impose constraints on high-opacity Gaussian basis elements to ensure a stable optimization process.

[0041]

[0042] Among them, N refers to the number of Gaussian basis elements, and σ and σ(t) represent opacity and temporal opacity respectively.

[0043] 4. Meanwhile, the present invention periodically calculates the sampling scores of Gaussian basis elements, comprehensively considers the spatial gradient and opacity index, and migrates the Gaussian basis elements with low influence to the dynamic detail area to achieve efficient modeling.

[0044] Corresponding to the embodiment of the foregoing dynamic three-dimensional scene reconstruction and real-time rendering method based on free spatio-temporal Gaussian basis elements, the present invention also provides an embodiment of a dynamic three-dimensional scene rendering device based on free spatio-temporal Gaussian basis elements.

[0045] See Figure 3 , an embodiment of a dynamic three-dimensional scene rendering device based on free spatio-temporal Gaussian basis elements provided by an embodiment of the present invention includes a memory and one or more processors. Executable code is stored in the memory. When the processor executes the executable code, it is used to implement a dynamic three-dimensional scene reconstruction and real-time rendering method based on free spatio-temporal Gaussian basis elements in the above embodiment.

[0046] An embodiment of a dynamic three-dimensional scene rendering device based on free spatio-temporal Gaussian basis elements provided by the present invention can be applied to any device with data processing capabilities. The any device with data processing capabilities can be a device or apparatus such as a computer, a graphics workstation, a server, or a virtual reality device. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logically meaningful device, it is formed by the processor of any device with data processing capabilities reading the corresponding computer program instructions in the non-volatile memory into the memory for running. From a hardware perspective, as Figure 3 shown, it is a hardware structure diagram of any device with data processing capabilities where a dynamic three-dimensional scene rendering device based on free spatio-temporal Gaussian basis elements provided by the present invention is located. Except for Figure 3 the processor, memory, network interface, and non-volatile memory shown, the any device with data processing capabilities where the device in the embodiment is located usually further includes other hardware modules according to the actual functions of the device, which will not be elaborated here.

[0047] The specific implementation process of the functions and roles of each unit in the above device can be specifically seen in the implementation process of the corresponding steps in the above method, which will not be elaborated here.

[0048] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the descriptions of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. A person of ordinary skill in the art can understand and implement it without creative work.

[0049] An embodiment of the present invention further provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, it implements a method for dynamic three-dimensional scene reconstruction and real-time rendering based on free spatio-temporal Gaussian basis elements in the above embodiments.

[0050] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a Smart Media Card (SMC), an SD card, a Flash Card, etc. equipped on the device. Further, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or will be output.

[0051] The present invention also provides a computer program product, including computer programs / instructions. When the computer programs / instructions are executed by a processor, they implement the above-mentioned method for dynamic three-dimensional scene reconstruction and real-time rendering based on free spatio-temporal Gaussian basis elements.

[0052] The above embodiments are used to explain the present invention, rather than limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims of the present invention fall within the protection scope of the present invention.

Claims

1. A method for dynamic three-dimensional scene reconstruction and real-time rendering based on free-space-time Gaussian sputtering, the method comprising: (1) Obtain multi-view dynamic scene videos, extract cross-view feature points for three-dimensional reconstruction, initialize free-space-time Gaussian basis elements, and set explicit motion functions and time opacity functions that can be optimized for each Gaussian basis element to represent the geometry and appearance of the dynamic three-dimensional scene; (2) Based on the observation view, sputter and render the Gaussian basis elements at the current moment, and output high-fidelity dynamic scene images corresponding to the view; (3) Jointly optimize the parameters of the Gaussian basis elements using a rendering loss function and a four-dimensional regularization strategy, and periodically relocate the Gaussian basis elements with low influence.

2. The method for dynamic three-dimensional scene reconstruction and real-time rendering based on free-space-time Gaussian sputtering according to claim 1, in step (1), the specific process of obtaining multi-view dynamic scene videos and initializing free-space-time Gaussian basis elements is as follows: Use a feature point matching algorithm to extract cross-view matching feature points, use triangulation to recover the sparse three-dimensional point cloud, and combine the information of each time frame to initialize the position, time parameters, and scale, direction, and color attributes of the Gaussian basis elements, forming a description of the initial geometry and appearance of the scene.

3. The method for dynamic three-dimensional scene reconstruction and real-time rendering based on free-space-time Gaussian sputtering according to claim 1, in step (1), the specific process of setting explicit motion functions and time opacity functions for each Gaussian basis element is as follows: Assign a linearly optimizable velocity vector to each Gaussian basis element, and establish a motion function to describe the spatial position change of the Gaussian basis element in the time dimension; at the same time, set the time opacity function in the form of a Gaussian distribution, and automatically adjust the influence range of the Gaussian basis element at different time points according to the time center and duration.

4. The method for dynamic three-dimensional scene reconstruction and real-time rendering based on free-space-time Gaussian sputtering according to claim 1, in step (2), during rendering, rasterization and sputtering methods are used to project the Gaussian basis elements at the corresponding position at the current time onto the image plane to achieve real-time dynamic scene rendering.

5. The method for dynamic three-dimensional scene reconstruction and real-time rendering based on free-space-time Gaussian sputtering according to claim 1, in step (3), the four-dimensional regularization strategy is used to suppress the Gaussian basis elements with opacity close to 1 to prevent the optimization process from falling into a local optimum; the periodic relocation strategy migrates the Gaussian basis elements with opacity close to 0 to the dynamic detail area by comprehensively calculating the spatial gradient and opacity of the Gaussian basis elements, improving the overall expression ability.

6. A dynamic three-dimensional scene reconstruction and real-time rendering device based on free-space Gaussian sputtering, comprising a memory and one or more processors, wherein executable code is stored in the memory, characterized in that, When the processor executes the executable code, it implements a method for dynamic three-dimensional scene reconstruction and real-time rendering based on free-space-time Gaussian sputtering according to any one of claims 1-5.

7. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a method for dynamic three-dimensional scene reconstruction and real-time rendering based on free-space-time Gaussian sputtering according to any one of claims 1-5.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, it implements a method for dynamic three-dimensional scene reconstruction and real-time rendering based on free-space-time Gaussian sputtering according to any one of claims 1-5.

Images