Plug flow volume video rendering method and device based on image grid, chip and medium

Abstract

The invention discloses a plug-flow volume video rendering method and device based on an image grid, a chip and a medium. The method comprises the following steps: S1, acquiring a picture sequence of a dynamic human body from a plurality of visual angles; s2, performing Gaussian splash reconstruction on the first frame of each group by using the point cloud; then modeling is carried out on deformation generated by Gaussian splashing along with time change, and a Gaussian splashing model is obtained; s3, applying a residual entropy loss constraint and a time sequence loss constraint, and performing appearance optimization; and S4, splashing and baking the 3D gauss into a 2D format, and then compressing to form a video. According to the method, dynamic 3D Gaussian splash is regarded as a 2D Gaussian video, motion and appearance attributes are simulated respectively through grouping training from multi-view RGB video input, and finally, a volume video capable of being transmitted in a streaming manner is constructed, so that high-quality dynamic character rendering can be realized on mobile equipment, a high-quality rendering result is realized, and the dynamic character rendering efficiency is improved. Meanwhile, the method is superior to the prior art in quality, rendering speed and storage, and has a wide application prospect.

Description

Technical Field

[0001] The invention relates to computer graphics processing technology, in particular to an image grid-based pushable volume video rendering method, device, chip and medium. Background Art

[0002] In the era of mobile Internet, people can conveniently watch 2D videos on mobile devices anytime and anywhere. Further, watching photorealistic volumetric videos on mobile devices allows users to freely choose viewing angles and provide an immersive experience. However, it is still challenging to stream and render high-quality volumetric videos to mobile devices. The difficulty lies in maintaining high fidelity at a computational complexity suitable for mobile devices while supporting streaming with limited storage space.

[0003] Traditional volumetric video reconstruction usually relies on mesh reconstruction and texture mapping. Although dynamic meshes can support streaming on mobile devices, mesh-based methods are limited in accuracy, especially in areas that are occluded, lack texture, or are difficult to represent with meshes (such as hair). Neural Radiance Fields (NeRF) can avoid the problems caused by geometric reconstruction and achieve photorealistic effects; various methods extend NeRF to dynamic scenes, but they still lack in supporting long sequences and streaming. Some methods can stream radiance fields for each frame, but the high computational overhead makes them impractical on mobile devices. In particular, the recent VideoRF can stream and render dynamic radiance fields on mobile devices, but the rendered results still suffer from blurriness.

[0004] Recently, 3D Gaussian Splatting (3DGS) has achieved remarkable rendering capabilities in terms of speed and quality. Although the original 3DGS only supported static scenes, it has witnessed rapid adoption in dynamic scenes. Many works have extended 3DGS to dynamic scenes and achieved high-quality playback, but these methods do not support long sequences due to the storage overhead of each frame. Animatable methods can handle extended human motions, but the computational overhead is still too large to be applied on mobile devices. On the other hand, due to the advantages of explicit representation, researchers have explored implementing Gaussian rasterization and rendering on various lightweight web platforms and exploiting the effective attribute compression of static scenes. However, their computational and storage overheads are still insufficient to support real-time interactive streaming of dynamic Gaussian points.

[0005] In summary, streaming and rendering dynamic 3DGS for viewing volumetric videos on mobile devices remains an unsolved problem, which requires an efficient training process to generate compact Gaussian representations. In addition, the generated Gaussian assets must also be compatible with the hardware video codec and shader rendering on mobile devices for efficient rendering. Summary of the invention

[0006] The present invention aims to solve the existing problems and provide a pushable volumetric video rendering method, device, chip and medium based on image grid, which are used to realize high-fidelity dynamic character rendering on mobile devices. In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention provides a pushable volumetric video rendering method based on image grid, including

[0007] S1, collects dynamic human body image sequences from multiple perspectives;

[0008] S2, group the image sequences, and use the point cloud to reconstruct the first frame of each group by Gaussian splashing; then model the deformation caused by the Gaussian splashing over time to obtain a Gaussian splashing model of each frame with temporal consistency;

[0009] S3, applies residual entropy loss constraints and timing loss constraints to the Gaussian splash model to optimize the appearance of timing consistency;

[0010] S4 bakes the 3D Gaussian splash into 2D format and then compresses it to form a video.

[0011] In some embodiments, in S1, after the video is captured, frame cutting processing is performed and / or the background is removed to only retain the image of the human foreground.

[0012] In some embodiments, in S2, in the picture sequence, every 20 frames are set as a group, and / or a neural surface reconstruction method based on neural radiation field (NeRF) and signed distance function (SDF) is used to reconstruct and obtain a triangular mesh, and then uniform sampling is performed on the triangular mesh surface to obtain a point cloud.

[0013] In some embodiments, in S2, after the first frame is reconstructed using the point cloud using Gaussian splashing, the number of Gaussian splashing is pruned to reduce disk usage.

[0014] In some embodiments, in S2, a multi-resolution encoded hash grid is used to estimate the position change of each Gaussian point.

[0015] In some embodiments, in S2, motion estimation is implemented by a small multi-layer perceptron MLP:

[0016] Δx=MLP(h(x t-1 ));

[0017] where x t =x t-1 +Δx,x t-1 and x tThey represent the coordinates of the points in the Gaussian splash point cloud at time t-1 and time t respectively, h represents multi-resolution hash coding, MLP represents a small multi-layer perceptron, and Δx represents the offset of the point coordinates.

[0018] In some embodiments, in S2, photometric consistency loss and image quality loss are used to optimize the parameters of the multi-resolution hash grid and the small perceptron, as follows:

[0019] L=(1-λ)L photometric +λL D-SSIM

[0020] Where L photometric is the photometric consistency loss, λL D-SSIM is the structural similarity loss, and λ is the weighted weight. In some embodiments, in S3, the formula of residual entropy loss is:

[0021]

[0022] Where N is the number of points splashed by Gauss; y is the attribute of Gauss, y at the tth frame t ∈[R t , S t , o t , c t , S.H. t ], R t , S t , o t 、c t SH t is the rotation, scale, opacity, color and spherical harmonics at the t-th frame; the quantized Gaussian attribute formula is:

[0023] Δy t =y t -y t-1

[0024]

[0025] in is the random uniform noise imposed on the residual; the formula for the probability mass function of the residual data distribution is:

[0026] In some embodiments, in S3, a timing loss L is imposed on each set of picture sequences. temp The formula is as follows:

[0027]

[0028] Where W and H are the width and height of the image after Gaussian splash mapping, y iThe value of the pixel in the mapped image stores the rotation, scale, opacity, color, and spherical harmonics data in the Gaussian splash model. In summary, the total loss formula in the appearance optimization stage is as follows:

[0029] L=(1-λ)L photometri c+λL D-ssIM +λ e L entropy +λ t L temp

[0030] where λ e and λ t are the weights of the corresponding regularization terms.

[0031] In some embodiments, in S4, the Gaussian splatter point cloud is first Morton sorted according to the position, and then the data carried by the points are mapped one by one to the pixels of the image.

[0032] In some embodiments, it also includes S5, uploading the video to a remote server, and each receiving platform downloads and extracts the Gaussian splash point cloud to achieve cross-platform rendering.

[0033] The present invention also provides a pushable volumetric video rendering system based on an image grid, comprising:

[0034] Image collection module, used to collect multi-view human body images and group them;

[0035] A motion estimation module, for reconstructing the first frame of each group using Gaussian splashing, and then generating Gaussian representations of subsequent frames through motion estimation to obtain a Gaussian splashing model;

[0036] The appearance optimization module applies residual entropy loss constraints and timing loss constraints to the Gaussian splash model to optimize the appearance.

[0037] The baking compression module is used to bake 3D Gaussian splash into 2D format and then compress it into video.

[0038] In some embodiments, a real-time rendering module is also included, which is arranged on a fixed receiving platform or a mobile receiving platform. Each receiving platform downloads the video in the remote server and extracts the Gaussian splash point cloud to achieve cross-platform rendering. The present invention also provides a streaming volumetric video rendering device, including a collection device, a memory and a processor. The collection device is used to collect multi-view human body pictures. The memory includes a computer program that can be run thereon. When the processor executes the computer program, the above-mentioned rendering method is implemented.

[0039] The present invention also provides a chip, comprising one or more processors, for calling and running a computer program from a memory, so that a device equipped with the chip executes the above-mentioned rendering method.

[0040] The present invention also provides a storage medium containing computer executable instructions, and the computer executable instructions are used to execute the above rendering method when executed by a computer processor.

[0041] Compared with the existing technology, the present invention regards dynamic 3D Gaussian splash as 2D Gaussian video, starts from multi-view RGB video input, and performs group training to simulate motion and appearance attributes respectively, and finally constructs a streamable volumetric video. It can achieve high-quality dynamic character rendering on mobile devices and achieve high-quality rendering results. At the same time, it is superior to the existing technology in terms of quality, rendering speed and storage, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of storage and recovery of 3D Gaussian splashing according to the present invention;

[0043] Figure 2 It is a schematic diagram of the overall structure of the present invention;

[0044] Figure 3 Schematic diagram of the effect of an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The present invention will now be further described with reference to the accompanying drawings. Figures 1 to 3 , Figures 1 to 3 What is presented is an embodiment of the present invention, a pushable volumetric video rendering method (V3) based on image grid, which aims to ensure that everyone can seamlessly experience high-quality free viewpoint video (FVV) rendering anywhere and on any device.

[0046] To achieve the above objectives, the following embodiment first models the dynamic Gaussian splash 3DGS sequence as a compact 2D Gaussian video, so that it naturally supports the hardware video codec to achieve efficient streaming and decoding; and then by utilizing shader-based rendering, efficient rendering can be achieved on various portable devices.

[0047] The embodiments mainly include:

[0048] S1, multi-view human action image collection.

[0049] This embodiment uses a dome multi-camera system with a diameter of 8 meters and a height of 5 meters, in which 81 cameras (ZCam) with a resolution of 3840x2160 are used to collect dynamic human body image data; the cameras are dispersed and arranged inside the dome and pointed to the center of the dome, and the existing calibration algorithm is used to obtain the accurate position of the camera. During the acquisition, the performer performs free movements in the center of the dome, and the camera synchronously acquires 3840x2160 resolution at a speed of 30 frames per second. After the acquisition, the video is frame-cut to obtain the corresponding picture, and the existing background removal algorithm is used to obtain a picture that only retains the foreground of the human body.

[0050] S2, fast motion estimation based on multi-resolution hash-coded mesh deformation field. That is, directly use Gaussian splash modeling for each frame independently. Due to its temporal discontinuity, it still occupies a large amount of disk storage after using existing compression methods. Then an efficient grouping optimization strategy is used. The first stage uses hash coding and shallow MLP to learn motion, while the second stage uses residual entropy loss and speed loss to fine-tune other Gaussian properties, so as to quickly obtain time-continuous Gaussian splash.

[0051] For each frame image I in the video, it corresponds to the 3DGS representation of the frame, and each dimension corresponds to a different attribute of the 3DGS. In the rendering process, the mapping function Extract each Gaussian attribute at time t from each pixel in the 2D plane as follows:

[0052]

[0053] Among them I t is the feature image at time t, u,v are the two-dimensional indices in pixel coordinates, R t ,S t ,x t ,o t ,c t ,SH t are the rotation, scale, position, opacity, color and spherical harmonics of the Gaussian spot at time t. As a Gaussian splash 3DGS, each Gaussian point is passed through:

[0054] ∑=RSS T R T ,∑′=JW∑W T J T

[0055] Among them, ∑, ∑′ are the covariance matrices in the world coordinate system and the camera coordinate system, R is the rotation matrix, S is the scale matrix, J is the Jacobian matrix of the affine approximation of the projection transformation, and W is the observation transformation matrix. The projected Gaussian distribution for rendering can be obtained through the covariance matrix, and the color of the pixel on the image plane can be calculated by alpha blending the N-order Gaussian distribution from near to far:

[0056]

[0057] When the calculated Gaussian distribution overlaps a pixel, c i is the view-dependent color of each 3D Gaussian distribution, α i It is the opacity multiplied by the 3D Gaussian distribution and the corresponding covariance matrix ∑′. By utilizing the 2D Gaussian video representation, a high-fidelity and smooth Gaussian splash point cloud can be provided on common devices.

[0058] The dynamic human image sequence is divided into groups of 20 frames, so that topological transformations and sequences of infinite length can be processed. For each group, the first frame is marked as the first frame, and the existing neural surface reconstruction method based on neural radiance field (NeRF) and directed distance function (SDF) is used to reconstruct the triangular mesh, and the point cloud is uniformly sampled on the surface of the triangular mesh to provide a good initialization for Gaussian splashing.

[0059] Furthermore, after the point cloud is used to reconstruct the first frame by Gaussian splashing, the number of Gaussian splashing needs to be pruned to reduce disk usage. In the pruning process of this embodiment, the entire point cloud is first sorted according to transparency, then 30% of the points are deleted, and finally the number of fixed points is only optimized for appearance attributes, and the above steps are repeated until the number of points is less than 100,000, then pruning is stopped.

[0060] Furthermore, in this embodiment, for the Gaussian splashing construction process of each group of key frames, the number of iterations is set to 12,000, and each branch reduction is optimized 2,000 times. When performing motion estimation on subsequent frames, each frame is iterated 2,000 times.

[0061] In order to maintain the continuity of Gaussian splashing between two frames to achieve subsequent compact compression, such as Figure 2 As shown, this embodiment uses a multi-resolution encoded hash grid and a small multi-layer perceptron (MLP) to model the deformation caused by Gaussian splashing over time, thereby ensuring that the Gaussian properties of the same position between adjacent frames are as close as possible to achieve subsequent efficient compression.

[0062] Furthermore, the multi-resolution hash coding grid is set to 16 layers, and the number of features in each layer is 4.

[0063] Specifically, the efficient estimated motion is as follows: Δx = MLP(h(x t-1 ));

[0064] x t =x t-1 +Δx

[0065] x t-1 and x t They represent the coordinates of the points in the Gaussian splash point cloud at time t-1 and time t respectively, h represents multi-resolution hash coding, MLP represents a small multi-layer perceptron, and Δx represents the offset of the point coordinates.

[0066] Furthermore, photometric consistency loss and image quality loss are used to optimize the parameters of the multi-resolution hash grid and small perceptron. The formula is as follows:

[0067] L=(1-λ)L photometric +λL D-SSIM

[0068] Where L photometric is the photometric consistency loss, λL D-SSIM is the structural similarity loss, λ is the weighted weight, which is 0.2 here. Through the above-mentioned fast motion estimation, this embodiment can quickly obtain a Gaussian splash model of each frame with temporal consistency.

[0069] S3, appearance optimization for temporal consistency.

[0070] After S2 performs efficient motion estimation and obtains the Gaussian splash model of multiple frames, the appearance needs to be further optimized to ensure the quality of rendering. Since the video encoder used in the subsequent stage will entropy encode the residual between two frames, residual entropy loss constraints and timing loss constraints are imposed at this stage to ensure that the data between the two frames is as continuous as possible, while reducing the entropy of the residual so that the video encoder can efficiently compress the data.

[0071] Entropy coding, used in video encoders, encodes data based on probability, using fewer bits to encode a value if it occurs more frequently. Therefore, entropy loss moves the data distribution closer to the center, increasing the probability of repeated data values ​​and thus reducing storage.

[0072] After analyzing the properties of Gaussian splash, it is found that the residuals of rotation, scaling, opacity and spherical harmonic coefficients between two frames approximately obey Gaussian distribution. Let y be the property of Gaussian. At the tth frame, y t ∈[R t ,S t ,o t ,c t ,SH t ], qi is the quantization range of different Gaussian attributes. The quantized Gaussian attributes can be calculated by the following formula:

[0073] Δy t =y t -y t-1

[0074]

[0075] in is a random uniform noise added to the residual to simulate the quantization operation, thereby enhancing the robustness of the data to compression during the compression step. Subsequently, the following formula is used:

[0076]

[0077] By calculating the cumulative distribution function CDF of the data distribution between the two frames, the probability mass function PMF of the residual data distribution can be obtained. The final entropy loss can be calculated by summing as follows:

[0078]

[0079] Where N is the number of points in the Gaussian splash. By applying this residual entropy loss, the distribution of the residuals between the two frames is more centered, thereby enhancing the temporal consistency and making it robust under low-ratio compression. In order to further enhance the continuity of the Gaussian splash data, an additional temporal loss is applied in this appearance optimization stage to minimize the data difference between two adjacent frames. Due to the grouping strategy mentioned above, the temporal loss needs to be applied within each group of 20 frames. The formula is as follows:

[0080]

[0081] Where W and H represent the width and height of the image data after Gaussian splash mapping, yi is the value of the pixel in the mapped image, and stores data such as rotation, scaling, opacity, color, and spherical harmonics in the Gaussian splash model.

[0082] In summary, the total loss formula in the appearance optimization stage is as follows:

[0083] L=(1-λ)L photometric +λL D-SSIM +λ e L entropy +λ t L temp

[0084] where λ e and λ t are the weights of the corresponding regularization terms. In this embodiment, e Take 1e-4, λt Take 1e-3. In the appearance optimization process of this embodiment, each frame is iterated 3000 times.

[0085] S4, bakes the Gaussian splash into 2D format and then compresses it to form a video.

[0086] In order to compress or stream the Gaussian splatter using a video codec, the 3D Gaussian splatter needs to be baked into a 2D format. Figure 1 As shown, benefiting from the natural unstructured and disordered nature of the Gaussian splash point cloud structure, unlike the previous voxel grid-based method, this embodiment does not require additional storage of a 3D to 2D mapping table. It only needs to ensure that in each frame of Gaussian splash, different attributes have the same index in the corresponding 2D image, thereby allowing the point cloud structure to be quickly restored from the 2D image without the need for additional storage.

[0087] Specifically, the Gaussian splash point cloud is first Morton sorted according to the position, and then the data carried by the point is mapped one by one to the image pixels, so that the spatial continuity of the image can be enhanced, so that the storage can be reduced after compression using the video codec. During the baking process, since the image storage data type is uint8, the data needs to be quantized.

[0088] Gaussian splash is more sensitive to the quantization loss of point cloud coordinates, so other attributes are quantized in uint8 format, while coordinates are quantized in uint16 format. The high and low bits are stored in two uint8 images respectively, and when using the video codec, lossless compression is always used for the high-bit data, so as to retain the rendering quality of Gaussian splash as much as possible after compression.

[0089] After the baking of Gaussian splash to 2D image format is completed, the data is further compressed using a video codec. Here, the image data of the group of 20 frames mentioned above is compressed using the H.264 standard to obtain the corresponding video and store it on disk.

[0090] Furthermore, it also includes S5, uploading the video to a remote server, and each receiving platform downloads and extracts the Gaussian splash point cloud to achieve cross-platform rendering.

[0091] After obtaining the video of Gaussian splash storage through steps S1-S4, Figure 3 As shown, this embodiment also provides a real-time renderer for multiple platforms, including desktops, laptops, and mobile platforms. When a player receives a 2D Gaussian stream, it can use an outdated hardware video codec for fast decoding and use a shader for real-time rendering.

[0092] First, upload the obtained data video to the remote resource server. For the decoding process in the renderer, first read the data video from the remote server, and use OpenCV to decode the H.264 video to obtain the data image corresponding to each frame. For desktop and notebook platforms, a multi-threaded solution is adopted: Thread 1 is responsible for video stream pulling and decoding, and Thread 2 is responsible for data dequantization and constructing a Gaussian splash point cloud from the data for rendering. On the mobile platform, this embodiment is implemented using the Swift language. In order to maximize the use of hardware resources, the Metal compute shader is used here to convert the image into a Gaussian splash point cloud. For the rendering pipeline, the Metal shader is used to implement Alpha-blending of Gaussian splash, thereby avoiding the requirement of CUDA devices and realizing cross-platform rendering. In this way, high-quality volumetric video streaming can be achieved even on mobile devices.

[0093] Furthermore, the player of this embodiment supports real-time streaming of video streams from the network, real-time decoding and rendering, and supports operations including viewing angle adjustment, progress bar dragging, and playback pause.

[0094] See also Figure 2 A second embodiment of the present invention provides a pushable volumetric video rendering system based on an image grid, based on the method of the above embodiment, comprising:

[0095] Image collection module, used to collect multi-view human body images and group them;

[0096] A motion estimation module, for reconstructing the first frame of each group using Gaussian splashing, and then generating Gaussian representations of subsequent frames through motion estimation to obtain a Gaussian splashing model;

[0097] The appearance optimization module applies residual entropy loss constraints and timing loss constraints to the Gaussian splash model to optimize the appearance.

[0098] The baking compression module is used to bake 3D Gaussian splash into 2D format and then compress it into video.

[0099] Furthermore, it also includes a real-time rendering module, which is arranged on a fixed receiving platform or a mobile receiving platform. Each receiving platform downloads the video in the remote server and extracts the Gaussian splash point cloud to achieve cross-platform rendering. The third embodiment of the present invention also provides a high-fidelity dynamic character rendering device, including a collection device, a memory and a processor. The collection device is used to collect multi-view human body pictures. The memory includes a computer program that can be run thereon. When the processor executes the computer program, the above-mentioned rendering method is implemented.

[0100] The memory may include a random access memory, a flash memory, a read-only memory, a programmable read-only memory, a non-volatile memory or a register, etc. The processor may be a central processing unit (CPU), etc. or a graphic processing unit (GPU). The memory may store executable instructions. The processor may execute the executable instructions stored in the memory to implement the various processes described herein.

[0101] It can be understood that the memory in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a ROM (Read-Only Memory), a PROM (Programmable ROM), an EPROM (Erasable PROM), an EEPROM (Electrically EPROM), or a flash memory. The volatile memory can be a RAM (Random Access Memory), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as SRAM (Static RAM), DRAM (Dynamic RAM), SDRAM (Synchronous DRAM), DDRSDRAM (Double Data Rate SDRAM), ESDRAM (Enhanced SDRAM), SLDRAM (Synchlink DRAM), and DRRAM (Direct Rambus RAM). The memory described herein is intended to include, but is not limited to, these and any other suitable types of memory. In some embodiments, the memory stores the following elements, upgrade packages, executable units, or data structures, or a subset thereof, or an extended set thereof: an operating system and an application.

[0102] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application program includes various application programs, which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application program.

[0103] In the embodiment of the present invention, the processor calls a program or instruction stored in the memory, specifically, a program or instruction stored in an application program, and the processor is used to execute the rendering method steps provided in the first embodiment.

[0104] The fourth embodiment of the present invention further provides a chip for executing the rendering method in the first aspect. Specifically, the chip includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip is used to execute the rendering method in the first embodiment.

[0105] In addition, in a fifth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the rendering method in the first embodiment of the present invention are implemented. For example, the machine-readable storage medium may include, but is not limited to, various known and unknown types of non-volatile memories.

[0106] A sixth embodiment of the present invention further provides a computer program product, including computer program instructions, which enable a computer to execute the above method.

[0107] It will be appreciated by those skilled in the art that the units and steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

Claims

1. A pushable volumetric video rendering method based on an image grid, characterized in that: S1, collects dynamic human body image sequences from multiple perspectives; S2, group the image sequences, and use the point cloud to reconstruct the first frame of each group by Gaussian splashing; then model the deformation caused by the Gaussian splashing over time to obtain the Gaussian splashing model; S3, applies residual entropy loss constraints and timing loss constraints to the Gaussian splash model to optimize the appearance of timing consistency; S4 bakes the 3D Gaussian splash into 2D format and then compresses it to form a video.

2. The method according to claim 1, characterized in that: In S2, motion estimation is achieved through a small multi-layer perceptron MLP: Δx=MLP(h(x t-1 )); where x t =x t-1 +Δx,x t-1 and x t They represent the coordinates of the points in the Gaussian splash point cloud at time t-1 and time t respectively, h represents multi-resolution hash coding, MLP represents a small multi-layer perceptron, and Δx represents the offset of the point coordinates.

3. The method according to claim 1, characterized in that: In S2, photometric consistency loss and image quality loss are used to optimize the parameters of the multi-resolution hash grid and small perceptron. The formula is as follows: L=(1-λ)L photometric +λL D-SSIM Where Lphotometric is the photometric consistency loss, λL D-SSIM is the structural similarity loss, and λ is the weighted weight.

4. The method according to claim 1, characterized in that: In S3, the formula for residual entropy loss is: Where N is the number of points splashed by Gauss; y is the attribute of Gauss, y at the tth frame t ∈[R t ,S t ,o t ,c t ,SH t ], R t , S t , o t 、c t SH t is the rotation, scale, opacity, color and spherical harmonics at the t-th frame; the quantized Gaussian attribute formula is: Δy t =y t -y t-1 in is the random uniform noise imposed on the residual; the formula for the probability mass function of the residual data distribution is:

5. The method according to claim 1, characterized in that: The total loss formula in the appearance optimization stage is as follows: L=(1-λ)L photometric +λL D-SSIM +λ e THE entropy +λ t THE temp where λ e and λ t are the weights of the corresponding regularization terms.

6. The method according to claim 1, characterized in that: It also includes S5, which uploads the video to a remote server, and each receiving platform downloads and extracts the Gaussian splash point cloud to achieve cross-platform rendering.

7. A pushable volumetric video rendering system based on an image grid, characterized in that: Image collection module, used to collect multi-view human body images and group them; A motion estimation module, for reconstructing the first frame of each group using Gaussian splashing, and then generating Gaussian representations of subsequent frames through motion estimation to obtain a Gaussian splashing model; The appearance optimization module applies residual entropy loss constraints and timing loss constraints to the Gaussian splash model to optimize the appearance. The baking compression module is used to bake 3D Gaussian splash into 2D format and then compress it into video.

8. A streaming volumetric video rendering device, characterized in that: It includes an acquisition device, a memory and a processor. The acquisition device is used to acquire multi-view human body pictures. The memory includes a computer program that can be run thereon. When the processor executes the computer program, the above-mentioned rendering method is implemented.

9. A chip, characterized in that: It includes one or more processors, which are used to call and run a computer program from a memory, so that a device equipped with the chip executes the rendering method as described above.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to perform the above rendering method when executed by a computer processor.

Images