Real-time volume video rendering method, system, device, medium and program based on 3D Gaussian splash

Abstract

The invention discloses a real-time volume video rendering method, system, device, medium and program based on 3D Gaussian splashing, and the method comprises the steps: S1, capturing dynamic point cloud data in real time, and constructing a representation model with 3D Gaussian distribution as a core; s2, projecting the 3D gauss into a 2D image space, sequencing the projected 2D gauss according to depth values, and generating a rendered image; and S3, comparing the rendered image with the reference true value to solve loss so as to update the dynamic point cloud data and the 3D Gaussian parameters, and obtaining a rendered real-time volume video. Through adaptive density control, dynamic LOD control and GPU accelerated rendering, continuous modeling and low-delay and high-throughput real-time rendering of the dynamic scene are realized, the rendering efficiency and the interaction response speed can be remarkably improved, and the problems of low dynamic scene modeling efficiency and insufficient real-time performance in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the field of computer graphics technology, in particular to a real-time volumetric video rendering method, system, device, medium and program based on 3D Gaussian splattering. Background Art

[0002] Volumetric video is an innovative video format that allows viewers to watch from any perspective, significantly enriching the viewing experience. This format is particularly prominent in virtual reality (VR) and augmented reality (AR), particularly in distance learning and live streaming applications, where it provides an unprecedented immersive and interactive experience. This characteristic of volumetric video not only enhances user engagement but also demonstrates enormous potential and application value in a variety of fields, including education and entertainment.

[0003] With the rapid development of virtual reality (VR) and mixed reality (MR) technologies in recent years, volumetric video has become a core technology for immersive content generation, thanks to its ability to dynamically record the complete light field information of real scenes in three-dimensional form. By capturing the geometry, material, and lighting properties of dynamic objects, volumetric video enables realistic rendering from any perspective, providing a highly realistic experience for scenarios such as virtual conferencing, digital twins, and interactive entertainment.

[0004] However, existing volumetric video rendering technologies still face many challenges in practical applications, specifically in the following aspects:

[0005] 1. Efficiency bottleneck of traditional volume representation methods;

[0006] Existing technologies for representing and rendering volumetric video primarily rely on models such as point clouds, voxel grids, or implicit neural radiance fields (NeRF). Point clouds and voxel grids require the storage of large amounts of discrete data, resulting in high memory usage and inefficient dynamic scene updates. While NeRF-based methods can achieve high-fidelity rendering, their reliance on neural network inference requires high computing resources, making it difficult to meet the stringent real-time (>30 FPS) and low power requirements of head-mounted devices (such as Apple Vision Pro).

[0007] 2. Conflict between dynamic scene modeling and real-time rendering;

[0008] Dynamic volumetric videos (such as those of human motion and complex deforming objects) exhibit significant inter-frame discrepancies. Traditional methods require frame-by-frame geometry reconstruction or voxel structure updates, significantly increasing computational latency. Existing algorithms lack adaptability to dynamic scenes, making it difficult to balance rendering quality and performance. For example, while optimization schemes based on optical flow or temporal interpolation can mitigate inter-frame redundancy, they are prone to introducing rendering artifacts due to motion blur or occlusion.

[0009] 3. Hardware limitations of mobile devices;

[0010] While spatial computing devices like the Apple Vision Pro boast powerful sensors and displays, the parallel computing power and memory bandwidth of their computing units (such as the M-series chips) are still not directly compatible with desktop-level volume rendering pipelines. Existing solutions often reduce resolution or simplify geometric details to accommodate mobile devices, but this significantly sacrifices visual immersion and makes it difficult to meet user demands for high-fidelity dynamic content.

[0011] 4. Real-time requirements of interactive applications;

[0012] In VR / MR scenarios, users are extremely sensitive to delays in viewing angle changes and dynamic object interactions (such as gesture manipulation). Traditional volume rendering technology, due to its large data volumes and computational complexity, struggles to achieve real-time updates and multi-view synchronization of dynamic scenes within millisecond latency, limiting the user experience's immersion and interactivity. Summary of the Invention

[0013] The present invention aims to solve existing problems, especially to meet the requirements of dynamic scene rendering in virtual reality (VR) and mixed reality (MR) devices, and to provide a real-time volumetric video rendering method, system, device, medium and program based on 3D Gaussian splattering.

[0014] In order to achieve the above object, the technical solution adopted by the present invention provides a real-time volumetric video rendering method based on 3D Gaussian splattering, comprising:

[0015] S1, captures dynamic point cloud data in real time and builds a representation model based on 3D Gaussian distribution;

[0016] S2, projects the 3D Gaussian into the 2D image space, sorts the 2D Gaussian according to the depth value after projection, and generates a rendered image;

[0017] In S3, the rendered image is compared with the ground truth to calculate the loss, so as to update the dynamic point cloud data and the parameters of the 3D Gaussian to obtain the rendered real-time volumetric video.

[0018] Furthermore, in the initialization of S1, the formula for obtaining the three-dimensional covariance matrix Σ of the 3D Gaussian by combining the scaling transformation S and the rotation transformation R is:

[0019] Σ=RSS T R T .

[0020] Furthermore, in the projection of S2, based on the given view transformation W and the 3D covariance matrix Σ, the formula of the projected 2D covariance matrix Σ′ is:

[0021] Σ′=JWΣW T J T

[0022] where J is the Jacobian matrix of the affine transformation:

[0023]

[0024] Where l′ is the modulus of the point (u0,u1,u2) in three-dimensional space, that is, the Euclidean distance from the origin to the point; the coordinates of the center point of the 3D Gaussian are (u0,u1,u2), and the coordinates of the center point of the 2D Gaussian are (u0 / u2,u1 / u2).

[0025] Furthermore, in the projection of S2, the formula for the opacity α′ of the projected 2D Gaussian is:

[0026]

[0027] where α is the opacity of the 3D Gaussian.

[0028] Furthermore, in the sorting of S2, the 2D image is divided into multiple small blocks, each of which is associated with several projected 2D Gaussians through several identifiers to reduce computational complexity. Then, based on the pixel position x, the depth of the Gaussian is calculated through the view transformation W to form a sorted list N, and finally the final synthesized color C is calculated:

[0029]

[0030] where c i is the learned color, α j ′ is the final opacity of the jth Gaussian for the current pixel, and the final opacity α i ′ is the learned opacity α i With highlight brightness:

[0031]

[0032] where x′ and μ i ′ is projected into the same coordinate system.

[0033] Furthermore, the method according to claim 1 is characterized in that: in S3, the loss function is:

[0034] L=(1-λ)L1+λL D-SSIM

[0035] Where L1 and L D-SSIM is the loss function, and λ is a weighting factor with a value range of [0,1].

[0036] Furthermore, in the pre-processing before S2 projection or in the updating of dynamic point cloud data in S3, adaptive density control is also adopted, that is, the density of 3D Gaussian is controlled by densification and pruning of points.

[0037] Furthermore, in the rendering of S2, smooth rendering processing is adopted to improve visual stability and reduce artifacts, that is, dynamic weighted averaging, motion compensation-based filtering methods or spatial interpolation technology are used to adjust the geometric and color information of adjacent frames to ensure natural transitions during the rendering process.

[0038] Furthermore, in the rendering of S2, multi-threaded parallel calculation of interpolated frames is adopted outside the main rendering thread to reduce the computing load and reduce rendering delay.

[0039] The present invention also provides a real-time volumetric video rendering system, comprising:

[0040] Parameter module, used to capture dynamic point cloud data in real time, obtain 3D Gaussian distribution, and realize continuous modeling of dynamic scenes;

[0041] The rendering module is used to project the 3D Gaussian into the 2D image space, sort the 2D Gaussian according to the depth value after projection, and generate the rendered image;

[0042] The optimization module is used to compare the rendered image with the ground truth value to calculate the loss, so as to update the dynamic point cloud data and the parameters of the 3D Gaussian to obtain the rendered real-time volumetric video.

[0043] The present invention also provides a spatial computing device, comprising a memory and a processor; the memory is used to store a program; the processor is used to execute the program to implement any one of the methods described.

[0044] The present invention also provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, any of the methods described above is implemented.

[0045] The present invention also provides a computer program product, comprising a computer program, wherein the computer program executes any one of the methods when executed by a processor.

[0046] Compared with the existing technology, the present invention realizes continuous modeling and low-latency, high-throughput real-time rendering of dynamic scenes through adaptive density control, dynamic LOD control and GPU accelerated rendering, which can significantly improve rendering efficiency and interactive response speed, and solves the problems of low efficiency and insufficient real-time performance of dynamic scene modeling in existing volumetric video rendering technology. It is suitable for immersive applications such as virtual meetings, real-time interactive entertainment, and dynamic scene reconstruction.

[0047] This invention addresses the inter-frame motion characteristics of dynamic volumetric video by introducing smooth rendering processing and a parallel multi-threaded interpolation strategy to avoid rendering artifacts caused by motion blur and topological changes, while also supporting user gesture interaction and head-mounted spatial movement. This invention achieves a balance between high fidelity and low memory usage on devices such as the Apple Vision Pro, providing an efficient, real-time, and scalable solution for volumetric video presentation in next-generation VR / MR devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1a 、 1b Schematic diagram of snowball splash and two-dimensional Gaussian distribution density function;

[0049] Figure 2 Schematic diagram of 3D Gaussian projection;

[0050] Figure 3 Schematic diagram of image plane partitioning and 3D Gaussian correlation in sorting;

[0051] Figure 4 This is the rendering color effect diagram;

[0052] Figure 5 This is a schematic diagram of the overall pipeline process of an embodiment of the present invention;

[0053] Figure 6 Schematic diagram of obtaining the initial point cloud for SfM;

[0054] Figure 7 、 Figure 8 This is a schematic diagram of the final rendering effect. DETAILED DESCRIPTION

[0055] The present invention will now be further described with reference to the accompanying drawings. One embodiment of the present invention is a real-time volumetric video rendering method based on 3D Gaussian splattering. Combining the adaptability and lightweight advantages of 3D Gaussian representation, an efficient rendering framework for head-mounted spatial computing devices (such as Apple Vision Pro) is proposed. The method primarily includes:

[0056] S1 captures dynamic point cloud data in real time and builds a representation model with 3D Gaussian distribution as the core; dynamically updates Gaussian parameters (position μ, covariance Σ, opacity α, spherical harmonic coefficient c) through a differentiable optimization algorithm to achieve continuous modeling of dynamic scenes.

[0057] In volumetric video, the attributes of a 3D Gaussian include its center position μ, opacity α, three-dimensional covariance matrix Σ (indicating scaling), and color c. c is represented by spherical harmonics to create a view-dependent appearance. All of these attributes are learnable and can be optimized via backpropagation. During initialization, the 3D covariance matrix Σ of the 3D Gaussian is obtained by combining a scaling transformation S and a rotation transformation R:

[0058] Σ=RSS T R T

[0059] Since scaling transformations are always along the coordinate axes, only a 3D vector s is required, while rotations are expressed using quaternions q. Machine learning typically uses gradient descent to optimize parameters, but directly optimizing Σ is difficult to guarantee positive semi-definiteness. Therefore, the approach used is to propagate the gradient to s and q for optimization.

[0060] S2, projects the 3D Gaussian into the 2D image space, sorts the 2D Gaussian according to the depth value after projection, and generates a rendered image.

[0061] In preprocessing, the parameters of the 3D Gaussian (three-dimensional covariance matrix Σ, center position μ, opacity α, color c spherical harmonic function) are optimized to adapt to the given image sequence and perspective. Further, gradient descent or other optimization algorithms can be used to minimize the reconstruction error.

[0062] Preferably, adaptive density control (ADC) may be used to adjust the number and distribution of 3D Gaussians to achieve optimal rendering effects.

[0063] See also Figure 1a Gaussian Splatting is a snowball method, which is like throwing a snowball (3D Gaussian ball) onto a glass plate. When the snowball spreads out, the amount of snow (contribution to the image) at the impact center is the largest, and as the distance from the impact center increases, the amount of snow (contribution) decreases. Figure 1a The image can be associated with Figure 1b The density function of the two-dimensional Gaussian distribution in .

[0064] See also Figure 2 , in the projection of S2, the 3D Gaussian (ellipsoid) is projected into the 2D image space (ellipse) for rendering. Given the view transformation W and the 3D covariance matrix Σ, the projected 2D covariance matrix Σ′ can be calculated using the following formula:

[0065] Σ′=JWΣW T J T

[0066] Where J is the Jacobian matrix of the affine approximation of the projective transformation, which converts a 3D vector into a 2D vector.

[0067] The process of converting a 3D Gaussian to a 2D Gaussian is the key to 3D Gaussian splatting. Traditional volume- and integral-based rendering methods require selecting sampling points to query implicit geometry and accumulating its attribute transformations to calculate pixel color. This requires not only ensuring rendering quality and the continuity of implicit geometry, but also maintaining the fidelity of detail, and errors are inevitable. Therefore, it is often necessary to increase the number of samples, which leads to performance degradation. Since the axial integral of a 3D Gaussian is equivalent to a 2D Gaussian, the need for a large number of samples is mathematically resolved, limiting the amount of computation to the number of Gaussians, a highly efficient innovation. Furthermore, these Gaussians can be quickly rendered in parallel through the rasterization pipeline, further improving efficiency.

[0068]

[0069] in 2X2 variance matrix It can be obtained by omitting the third row and column of the 3X3 matrix V.

[0070] The transformation from camera space to light space is a non-affine transformation, which means that it is impossible to use a single linear transformation matrix to transform the coordinates of all points. This is because the coordinates of each point in the light space are a nonlinear function with its coordinates in the camera space as the independent variable, so there is no universal transformation matrix. Therefore, this embodiment uses an approximation method: the transformation matrix of the entire Gaussian is approximated using the transformation matrix of the center point coordinates of each three-dimensional Gaussian. In this way, the transformation of each Gaussian can be regarded as an affine transformation. This specific transformation matrix is ​​called the Jacobian matrix J:

[0071]

[0072] Where l′ is the modulus of the point (u0, u1, u2) in three-dimensional space (i.e., the Euclidean distance from the origin to the point); finally, for each given three-dimensional Gaussian covariance matrix V″ and its center point coordinates (u0, u1, u2) in world space, the Gaussian covariance V in ray space can be obtained. Since the coordinates are given in world space, they are first multiplied by the rotation and translation matrix W to convert them to camera space coordinates, and then the Jacobian matrix J is obtained from the center point coordinates. The Gaussian covariance in ray space is as follows:

[0073]

[0074] Where T is the translation transformation matrix; D Vis the spatial view transformation projection matrix, D S Space scaling transformation matrix; D N is the illumination-related normal matrix in space. Under orthographic projection, the first two rows and two columns of the 3D Gaussian covariance matrix directly form the 2D Gaussian covariance matrix projected onto the image plane. The center coordinates of the 2D Gaussian are obtained by dividing the camera space coordinates u0 and u1 by the third-dimensional coordinate u2, i.e., the center coordinates are (u0 / u2, u1 / u2). This effectively converts a 3D Gaussian distribution in 3D space into a 2D Gaussian distribution on the image plane.

[0075] The opacity of the projected 2D Gaussian needs to be adjusted according to the opacity and covariance matrix of the 3D Gaussian. Specifically, the opacity α′ of the projected 2D Gaussian can be calculated using the following formula:

[0076]

[0077] Furthermore, the center position μ and color c of the projected 2D Gaussian can be directly obtained from the parameters of the 3D Gaussian.

[0078] In the projection of S2 in some embodiments, the projected 2D Gaussians are sorted according to their depth values ​​so as to correctly handle occlusion relationships in the subsequent rendering process.

[0079] To improve sorting efficiency, spatial partitioning or other acceleration techniques can be used to reduce sorting complexity. When processing images, to reduce the computational cost of performing Gaussian operations on each pixel, this embodiment adopts a different approach: rather than performing precise calculations at the pixel level, it reduces precision to the more macroscopic tile level. The image plane is divided into multiple tiles, and the 2D Gaussians within each tile are then sorted.

[0080] The entire image is first divided into multiple non-overlapping blocks, which are aptly called "bricks" in the original research. As recommended by the original paper, each brick consists of 16 x 16 pixels. Next, it further identifies which "bricks" intersect with a specific Gaussian projection.

[0081] See also Figure 3, considering that a 3D Gaussian projection may cover multiple "bricks", an effective way to deal with it is to copy the 3D Gaussian and assign a unique identifier to each copied 3D Gaussian, that is, the ID of the Tile that intersects with it. In this way, each "brick" is associated with one or more 3D Gaussians, which identify the position and importance of the brick in the image. Through this optimization method, 3DGS can effectively reduce the computational complexity while maintaining the efficiency and accuracy of image processing. In the sorting of S2 in some embodiments, given a pixel position x, the distance to all overlapping 3D Gaussian bodies, that is, the depth of these 3D Gaussian bodies, can be calculated through the view transformation W to form a sorted list N of Gaussian bodies. Then, alpha blending is performed to calculate the final color of the overall image:

[0082]

[0083] where c i is the learned color, α j ′ is the final opacity of the jth Gaussian for the current pixel, and the final opacity α i ′ is the learned opacity α i With highlight brightness:

[0084]

[0085] where x′ and μ i ′ is projected into the same coordinate system.

[0086] See also Figure 4 If the transparency is 1, each ellipsoid is an ellipsoid with a clear boundary. S3 compares the rendered image with the ground truth to calculate the loss, thereby updating the dynamic point cloud data and the parameters of the 3D Gaussian to obtain the rendered real-time volumetric video.

[0087] Further, see Figure 5 In some embodiments, a SfM (Structure from Motion) point cloud is initialized to obtain a large number of 3D Gaussian spheres, and then the points are projected onto the image plane (i.e., splatting) with the help of camera extrinsics, and then differentiable rasterization is used to obtain a rendered image.

[0088] See also Figure 5 After obtaining the rendered image, compare it with the ground truth image to calculate the loss. The loss function is defined as:

[0089] L=(1-λ)L1+λL D-SSIM

[0090] Among them, L1 and L D-SSIM Both are loss functions, λ is a weighting factor with a value range of [0,1];

[0091] It controls the balance between L1 loss and D-SSIM loss:

[0092] When λ is 0, the optimization process relies entirely on L1 loss;

[0093] When λ is 1, the optimization process relies entirely on the D-SSIM loss.

[0094] Then backpropagate along the blue arrows: Figure 5 The blue arrow in the middle points upward, updating the parameters in the 3D Gaussian and sending them downward to the adaptive density control to update the dynamic point cloud data and achieve the final optimization.

[0095] In some embodiments, an adaptive density control method is also used to control the density of the 3D Gaussian.

[0096] In the initialization, 3D Gaussian splashing is initialized from the sparse point cloud generated by SfM or randomly initialized Gaussian, which can be done by directly calling the COLMAP library, and then point densification and pruning are performed to control the density of the 3D Gaussian.

[0097] In point densification, 3D Gaussian splattering adaptively increases the density of Gaussians to better capture the details of the scene, especially in areas where geometric features are missing or the Gaussians are too scattered. Densification is performed after a certain number of iterations (for example, 100 iterations) for Gaussians with large position gradients in the view space (i.e., exceeding a certain threshold). It involves cloning small Gaussians in underreconstructed areas or splitting large Gaussians in overreconstructed areas. For cloning, copies of the Gaussians are created and moved towards the position gradient. For splitting, a large Gaussian is replaced by two smaller Gaussians, reducing their scale by a specific factor. This step aims to find the best distribution and representation of Gaussians in 3D space, enhancing the overall quality of the reconstruction.

[0098] During point pruning, removing redundant or insignificant Gaussians can be considered a regularization process, eliminating Gaussians that are almost transparent (α is below a specified threshold) and those that are too large in world or view space. Furthermore, to prevent the density of Gaussians near the input camera from increasing unreasonably, these Gaussians are set to a value close to 0 after a fixed number of iterations. This step saves computational resources while maintaining the accuracy and effectiveness of the Gaussians.

[0099] In the specific code implementation, this embodiment also utilizes smooth rendering and a parallel multi-threaded frame insertion strategy. During the rendering of dynamic volumetric video, smooth rendering is introduced to address the discontinuity issue of inter-frame transitions, improving visual stability and reducing artifacts. This step smooths and filters the volumetric data during each frame rendering, maintaining temporal coherence while supporting user gesture interaction, enabling real-time movement and rotation.

[0100] Specifically, this embodiment uses dynamic weighted averaging, motion-compensated filtering, or spatial interpolation techniques to adjust the geometry and color information of adjacent frames to ensure a smooth transition during rendering. For example, when rendering each frame, a weighted fusion is performed on the data of the current frame and previous frames to reduce jitter and artifacts caused by rapid motion or interaction.

[0101] During gesture interaction, the present invention combines the device's sensor data to adjust the position and direction of the volumetric video in real time, and performs smoothing during the rendering phase to avoid visual discomfort caused by interactive transformations. Specific implementations include but are not limited to:

[0102] Gesture-driven real-time transformation: Utilizes gesture recognition technology to analyze user input movement and rotation operations and map them to the rendering space for dynamic adjustment.

[0103] Smooth interpolation calculation: Apply smoothing functions (such as exponential smoothing, Kalman filtering, etc.) to the position, rotation, and scale changes generated during the rendering process to reduce the jump problem caused by instantaneous changes;

[0104] Parallel optimization: Leverage multi-threading or GPU parallel computing to efficiently perform smoothing processing in the rendering pipeline, ensuring visual smoothness during interaction.

[0105] To further improve the frame rate and rendering quality of dynamic volumetric videos, the present invention also proposes a parallel multi-threaded interpolation strategy to achieve efficient frame transitions. The core idea of ​​this strategy is to use multi-threaded parallel calculations to interpolate frames outside the main rendering thread to reduce computing load and rendering latency. The specific implementation of this strategy includes:

[0106] Task decomposition and parallel scheduling: Volumetric video data is divided into multiple sub-regions and assigned to different threads for interpolation calculations, ensuring efficient use of computing resources;

[0107] Interpolation calculation: Use optical flow prediction, motion compensation or depth estimation methods to calculate the geometry and color information of the interpolated frame to achieve a natural transition;

[0108] Data synchronization and fusion: Use shared memory or queue mechanism to synchronize data between the main thread and the interpolation thread to ensure the consistency of the final rendering results.

[0109] Finally, by inputting dynamic video, a large number of single-frame point cloud files are output and rendered using the method of the above embodiment. Figure 7 and Figure 8 After wearing Apple Vision Pro, you can clearly see the rendered volumetric video; it also supports movement and rotation of gestures, including a mobile viewing experience in space while wearing glasses.

[0110] This embodiment covers core technical aspects such as dynamic capture of volumetric video data, 3D Gaussian parametric modeling, scene adaptive optimization, and low-latency rendering. It is particularly suitable for immersive application scenarios such as virtual conferencing, real-time interactive entertainment, and dynamic scene reconstruction that require a combination of high fidelity and real-time performance. It provides an innovative solution for dynamic content presentation in next-generation VR / MR devices.

[0111] A second embodiment of the present invention provides a modular solution for real-time volumetric video rendering, including:

[0112] Parameter module, used to capture dynamic point cloud data in real time, obtain 3D Gaussian distribution, and realize continuous modeling of dynamic scenes;

[0113] The rendering module is used to project the 3D Gaussian into the 2D image space, sort the 2D Gaussian according to the depth value after projection, and generate the rendered image;

[0114] The optimization module is used to compare the rendered image with the ground truth value to calculate the loss, so as to update the dynamic point cloud data and the parameters of the 3D Gaussian to obtain the rendered real-time volumetric video.

[0115] A third embodiment of the present invention provides an App for rendering volumetric video on a spatial computing device, such as a head-mounted spatial computing device such as Apple Vision Pro, comprising a memory and a processor; the memory is used to store programs; the processor is used to execute the programs to implement any method step described in the first embodiment of the present invention.

[0116] By executing the method in the first embodiment of the present invention, the rendering efficiency and interactive response speed can be significantly improved, and a balance between high fidelity and low memory usage can be achieved on devices such as Apple Vision Pro, providing an efficient, real-time and scalable solution for volumetric video presentation in next-generation VR / MR devices. The memory may include random access memory, flash memory, read-only memory, programmable read-only memory, non-volatile memory or registers, etc. The processor may be a central processing unit (CPU), etc. Or a graphic processing unit (GPU) memory can store executable instructions. The processor can execute the executable instructions stored in the memory, thereby implementing the various processes described herein.

[0117] It is understood that the memory in this embodiment may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a ROM (Read-Only Memory), a PROM (Programmable ROM), an EPROM (Erasable PROM), an EEPROM (Electrically EPROM), or a flash memory. The volatile memory may be a RAM (Random Access Memory), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as SRAM (Static RAM), DRAM (Dynamic RAM), SDRAM (Synchronous DRAM), DDR SDRAM (Double Data Rate SDRAM), ESDRAM (Enhanced SDRAM), SLDRAM (Synchlink DRAM), and DRRAM (Direct Rambus RAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0118] In some embodiments, the memory stores the following elements, upgrade packages, executable units or data structures, or a subset or extended set thereof: an operating system and applications.

[0119] The operating system includes various system programs, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and handle hardware-based tasks. The application program includes various application programs used to implement various application services. The program implementing the method of the embodiment of the present invention can be included in the application program.

[0120] In an embodiment of the present invention, the processor is configured to execute the method steps provided in the first embodiment by calling a program or instruction stored in a memory, specifically, a program or instruction stored in an application program.

[0121] A fourth embodiment of the present invention further provides a chip for executing the rendering method of the first aspect. Specifically, the chip includes a processor for calling and executing a computer program from a memory, so that a device equipped with the chip executes the method steps of the first embodiment.

[0122] A fifth embodiment of the present invention further provides a computer program product, comprising computer program instructions, which enable a computer to execute the above method steps.

[0123] It will be understood by those skilled in the art that the units and steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

Claims

1. A real-time volumetric video rendering method based on 3D Gaussian splattering, characterized by: S1, captures dynamic point cloud data in real time and builds a representation model based on 3D Gaussian distribution; S2, projects the 3D Gaussian into the 2D image space, sorts the 2D Gaussian according to the depth value after projection, and generates a rendered image; In S3, the rendered image is compared with the ground truth to calculate the loss, so as to update the dynamic point cloud data and the parameters of the 3D Gaussian to obtain the rendered real-time volumetric video.

2. The method according to claim 1, wherein: In the initialization of S1, the formula for the three-dimensional covariance matrix Σ of the 3D Gaussian is obtained by combining the scaling transformation S and the rotation transformation R: S = RSS T R T 。 3. The method according to claim 2, wherein: In the projection of S2, based on the given view transformation W and the 3D covariance matrix Σ, the formula of the projected 2D covariance matrix Σ′ is: Σ′=JWΣW T J T where J is the Jacobian matrix of the affine transformation: Where l′ is the modulus of the point (u0,u1,u2) in three-dimensional space; The coordinates of the center point of the 3D Gaussian are (u0,u1,u2), and the coordinates of the center point of the 2D Gaussian are (u0 / u2,u1 / u2); And / or, in the projection of S2, the formula for the opacity α′ of the projected 2D Gaussian is: where α is the opacity of the 3D Gaussian.

4. The method according to claim 1, wherein: In the sorting of S2, the 2D image is divided into multiple small blocks, each of which is associated with several projected 2D Gaussians through several identifiers; then, based on the pixel position x, the depth of the Gaussian is calculated through the view transformation W to form a sorted list N, and finally the final synthesized color C is calculated: where c i is the learned color, α j ′ is the final opacity of the j-th Gaussian for the current pixel, and the final opacity α i ′ is the learned opacity α i With highlight brightness: where x′ and μ i ′ is projected into the same coordinate system.

5. The method according to claim 1, wherein: In S3, the loss function is: L=(1-λ)L1+λL D-SSIM Where L1 and L D-SSIM is the loss function and λ is the weighting factor.

6. The method according to claim 1, wherein: In the pre-processing before S2 projection or the updating dynamic point cloud data in S3, adaptive density control is also used, that is, the density of 3D Gaussian is controlled by densification and pruning of points; and / or, in the rendering of S2, using smooth rendering processing to improve visual stability and reduce artifacts, that is, using dynamic weighted averaging, motion compensation-based filtering methods or spatial interpolation technology to adjust the geometric and color information of adjacent frames to ensure natural transitions during the rendering process; And / or, in the rendering of S2, multi-thread parallel calculation of interpolated frames is adopted outside the main rendering thread to reduce the computing load and reduce the rendering delay.

7. A real-time volumetric video rendering system, characterized in that include: Parameter module, used to capture dynamic point cloud data in real time, obtain 3D Gaussian distribution, and realize continuous modeling of dynamic scenes; The rendering module is used to project the 3D Gaussian into the 2D image space, sort the 2D Gaussian according to the depth value after projection, and generate the rendered image; The optimization module is used to compare the rendered image with the ground truth value to calculate the loss, so as to update the dynamic point cloud data and the parameters of the 3D Gaussian to obtain the rendered real-time volumetric video.

8. A spatial computing device, characterized in that: The method comprises a memory and a processor; the memory is used to store a program; the processor is used to execute the program to implement the method according to any one of claims 1 to 6.

9. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is performed.

Images