Four-dimensional Gaussian video compression and streaming playing method and device based on dynamic point cloud compression and real-time rendering

Abstract

The invention discloses a four-dimensional Gaussian video compression and streaming playing method and device based on dynamic point cloud compression and real-time rendering, and for a four-dimensional Gaussian video expressed by a series of three-dimensional Gaussian point clouds (3DGS), the method designs that a plurality of frames of continuous input point clouds are divided into a plurality of video frame groups according to a time sequence; and expressing the plurality of point clouds in each group as a single point cloud and a motion offset thereof. The Gaussian attribute of the 3DGS point cloud in each group is quantized, and the motion offset in each group is compressed by applying a video coding technology such as H.265 coding. According to the method, the model volume of the 4DGV is greatly reduced, streaming transmission under the public network bandwidth is allowed, and almost lossless video rendering quality is kept. The method can be widely applied to the fields of online free viewpoint volume videos, immersive virtual reality and the like, and high-quality volume videos can be transmitted and played in a streaming manner under the public network bandwidth (about 30Mbps).

Description

Technical Field

[0001] The present invention relates to the fields of computer graphics and 3D vision, and particularly to a four-dimensional Gaussian video compression and streaming playback method and apparatus based on dynamic point cloud compression and real-time rendering. Background Art

[0002] With the progress of computer vision and computer graphics, volumetric video, as an emerging media form, has shown broad prospects in multiple application fields such as remote conferencing and holographic classrooms. The core feature of volumetric video is its ability to encode complex four-dimensional spatio-temporal scene information, thereby providing users with an immersive experience. Therefore, in the development process of online volumetric video systems, achieving high-quality, real-time rendering while adopting a compact representation form to improve transmission efficiency has become a crucial research direction.

[0003] Among them, the three-dimensional reconstruction technology for dynamic scenes has become an important research direction in the graphics and vision communities. In recent years, neural radiance field technology has been developed, which can render high-quality new visual images, but its high computational complexity makes it difficult to achieve real-time rendering. Three-dimensional Gaussian Splatting (3DGS), as an efficient three-dimensional scene reconstruction solution, combines an implicit motion field and uses a neural network to predict the change of Gaussian point attributes over time, enabling the reconstruction and real-time rendering of dynamic scenes.

[0004] Although existing methods can support real-time rendering and provide relatively high visual quality, they still face the following challenges when dealing with long videos or complex motion scenes: (1) Limited scalability: Existing methods based on implicit neural deformation fields have limited representational capabilities due to the fixed model size, making it difficult to effectively handle long videos and complex motions. (2) Surge in storage requirements: The 3DGS method has high storage requirements because it stores a large amount of explicit attributes for each point. When extending 3DGS to four dimensions, the required storage will further increase, thus posing a more serious storage challenge for the representation of dynamic scenes. (3) Limited rendering platforms: Traditional 3DGS rendering is based on the general parallel computing architecture (Compute Unified Device Architecture, CUDA) provided by NVIDIA graphics cards and cannot run on machines equipped with other graphics cards. Therefore, achieving high-quality, low-storage, and streamable (i.e., play while downloading) volumetric videos remains a major challenge. Summary of the Invention

[0005] The object of the present invention is to provide a 4D Gaussian Video (4DGV) compression and streaming playback method based on dynamic point cloud compression and real-time rendering in view of the deficiencies in the prior art. This method has scalability and compactness, can achieve effective compression of 4D Gaussian video, can realize the streaming playback of 4D Gaussian video on consumer-grade graphics cards, and can respond to interactions in real time.

[0006] The object of the present invention is achieved by the following technical solutions: A 4D Gaussian video compression and streaming playback method based on dynamic point cloud compression and real-time rendering, the method includes the following key modules:

[0007] (1) Video grouping representation: Obtain 4D Gaussian video data and group it. According to the time order, several consecutive three-dimensional Gaussian point cloud inputs are divided into multiple video frame groups with the same number of frames. This grouping approach stems from the concept of group of pictures (GOP) in the video standard formulated by the Moving Picture Experts Group. For the consecutive point clouds within a video frame group, the first-frame point cloud within the group is retained, and the motion offsets of the point clouds of the remaining frames relative to the three-dimensional Gaussian points in the first-frame point cloud are calculated. Each video frame group is represented as a combination of a single point cloud and its motion offsets, which facilitates further compression and is also beneficial for video streaming.

[0008] (2) Point cloud compression: Use a variety of compression techniques to compress the first-frame point cloud of each video frame group. For the Gaussian attributes of the three-dimensional Gaussian point cloud, bit quantization technology is adopted to convert some floating-point Gaussian attributes such as rotation and scale into integers. For the high-dimensional features of Gaussian attributes, vector quantization technology is used to compress the high-order spherical harmonic coefficients, thereby achieving effective dimensionality reduction of the three-dimensional Gaussian attributes. For the three-dimensional Gaussian point cloud stored in point cloud format, range Asymmetric Numeral Systems (rANS) coding is used for further compression to obtain a highly compressed binary point cloud file.

[0009] (3) Motion offset compression: Further compress the 4D Gaussian video on the basis of existing video compression techniques. Morton sorting is used for the three-dimensional Gaussian point cloud distributed in three-dimensional space to map the motion offsets of the three-dimensional Gaussian points to the pixels of a two-dimensional image one by one, effectively maintaining the local similarity of the three-dimensional Gaussian point cloud. Then, existing video compression techniques such as H.265 video coding are used to greatly compress the occupied space of the motion offsets (compression ratio < 10%).

[0010] (4) Four - dimensional Gaussian video streaming: For the compressed grouped four - dimensional Gaussian video data, realize streaming transmission and real - time rendering; the specific process is as follows: Adopt multi - thread technology for parallel transmission and rendering. The thread responsible for transmission sequentially receives the video frame group and restores the three - dimensional Gaussian point cloud. The thread responsible for rendering uses the traditional graphics pipeline to transfer the three - dimensional Gaussian point cloud in the form of texture mapping, and calculates the Gaussian splash based on the rendering perspective and 3DGS attributes in the vertex shader and fragment shader to generate the rendered image.

[0011] Further, the input of the present invention is a dynamic scene reconstructed frame by frame in the form of a three - dimensional Gaussian point cloud. Each Gaussian point contains a series of attributes {x i , q i , s i , o i , c i} i∈P , which represent the position, rotation, scale, opacity, and color of the Gaussian point in sequence. The three - dimensional Gaussians between frames should maintain a corresponding relationship to support the calculation of the motion offset of the three - dimensional Gaussian. This is the input data of the present invention.

[0012] Further, every 30 consecutive frames are divided into a group, and the grouped compressed point cloud and its motion offset are compressed.

[0013] Further, the motion offsets to be compressed include the offsets of the position attribute and the rotation attribute, which can be expressed as Δx i,t = x i,t - x i , Δq i,t = q i,t / q i . Since the rendering quality of the three - dimensional Gaussian point cloud is extremely sensitive to the position, when mapping the position offset Δx i,t of the three - dimensional Gaussian point to a two - dimensional image, the floating - point type position offset is quantized to a 16 - bit integer, and the high 8 bits and the low 8 bits are respectively saved to two RGB images. When mapping the rotation offset Δq i,t to a two - dimensional image, the four - dimensional rotation vector is unfolded and saved as a grayscale image. When using video coding compression, lossless compression is adopted for the high 8 - bit image of the position offset, and lossy compression is adopted for the remaining images.

[0014] Further, when compressing the point cloud of the first frame of each group using bit quantization and vector quantization, it is all realized through the quantization - aware training method. It is also possible to directly quantize the trained point cloud, but the effect is not as good as the quantization - aware training method.

[0015] Further, when performing Morton sorting on the three-dimensional Gaussian point cloud, the sorting is based on the position offset of the second frame of point cloud in the group relative to the first frame of point cloud. After quantizing the floating-point type position offset into integer three-dimensional coordinates, the Morton code sorting is calculated. For the pixels of the two-dimensional image, the Morton code sorting is calculated for the pixel coordinates. The three-dimensional point cloud coordinates sorted by the Morton code are mapped one-to-one with the two-dimensional pixel coordinates sorted by the Morton code, satisfying the bijective relationship.

[0016] Further, the streaming is based on the traditional graphics rendering pipeline rather than the Compute Unified Device Architecture (CUDA) of NVIDIA graphics cards. The thread responsible for transmission receives the motion offset encoded as a video and the three-dimensional Gaussian point cloud of the encoded binary file, extracts the motion offset by extracting video frames, and finally calculates to restore the three-dimensional Gaussian point cloud within the video frame group through x i,t = x i + Δx i,t , q i,t = q i ·Δq i,t Calculate to restore the three-dimensional Gaussian point cloud within the video frame group. Calculate the projection of 3DGS on the two-dimensional plane in the vertex shader, and through the hardware rasterization of the graphics card, calculate the exponential part of the Gaussian function and obtain the color and opacity of each pixel in the fragment shader. Finally, obtain the rendered image through alpha blending.

[0017] In a second aspect, the present invention also provides a four-dimensional Gaussian video compression and streaming device based on dynamic point cloud compression and real-time rendering, including a memory and one or more processors. Executable code is stored in the memory, and when the processor executes the executable code, it implements the four-dimensional Gaussian video compression and streaming method based on dynamic point cloud compression and real-time rendering as described above.

[0018] In a third aspect, the present invention also provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, it implements the four-dimensional Gaussian video compression and streaming method based on dynamic point cloud compression and real-time rendering as described above.

[0019] In a fourth aspect, the present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the four-dimensional Gaussian video compression and streaming method based on dynamic point cloud compression and real-time rendering as described above.

[0020] The beneficial effects of the present invention are as follows:

[0021] 1. Draw on the concept of image groups to perform grouped compression processing on four-dimensional Gaussian videos, supporting the streaming transmission of four-dimensional Gaussian videos.

[0022] 2. Use point cloud compression technology and video compression technology respectively to compress the grouped four-dimensional Gaussian videos. At the cost of a basically negligible loss of rendering quality, the storage overhead of four-dimensional Gaussian videos is significantly reduced, allowing them to be transmitted over public network bandwidth.

[0023] 3. Implement three-dimensional Gaussian splashing based on the traditional graphics rendering pipeline, breaking through the limitations of NVIDIA graphics cards on the rendering of three-dimensional Gaussian point clouds, realizing the streaming transmission and real-time playback of four-dimensional Gaussian videos, and being able to respond to interactions in real time, giving an immersive video viewing experience.

[0024] In summary, the present invention provides a method for compressing and streaming playback of four-dimensional Gaussian videos based on dynamic point cloud compression and real-time rendering, which can effectively compress the storage space of streaming volumetric videos, support the streaming playback of volumetric videos on various devices, and is applicable to various application scenarios such as remote immersive communication and holographic classrooms, having broad application prospects and commercial value. Description of the Drawings

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or in the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art:

[0026] Figure 1 It is the overall algorithm flowchart of the present invention.

[0027] Figure 2 It is the flowchart of the streaming transmission and real-time rendering of four-dimensional Gaussian videos in the present invention.

[0028] Figure 3 It is the structural diagram of a device for compressing and streaming playback of four-dimensional Gaussian videos based on dynamic point cloud compression and real-time rendering provided by the present invention. Detailed Embodiments

[0029] The following will further elaborate on the specific embodiments of the present invention in conjunction with the accompanying drawings.

[0030] Figure 1Shows the overall framework of the present invention, that is, a four-dimensional Gaussian video compression and streaming method based on dynamic point cloud compression and real-time rendering, which greatly compresses the four-dimensional Gaussian video represented by three-dimensional Gaussian and supports streaming transmission and playback under public network bandwidth. First, the four-dimensional Gaussian video is divided into multiple groups of video frames with a fixed length G (usually 30) in chronological order. The Gaussian point cloud within each group of video frames is represented as the first-frame point cloud and the motion offset, and different compression techniques are used to compress them respectively, compressing the four-dimensional Gaussian video into a combination of multiple groups of binary point cloud files and motion offset videos. Finally, the four-dimensional Gaussian video is transmitted over the Internet, and after receiving it, it is restored to a three-dimensional Gaussian point cloud, and the image is drawn in real time and the interaction is responded to. The method of the present invention specifically includes the following five steps:

[0031] (1) Video grouping expression: Obtain the four-dimensional Gaussian video data and group it. According to the chronological order, several consecutive three-dimensional Gaussian point cloud inputs (such as 30 frames) are divided into multiple groups of video frames. The four-dimensional Gaussian video data is a dynamic scene reconstructed frame by frame in the form of a three-dimensional Gaussian point cloud. Each Gaussian point contains a series of attributes {x i ,q i ,s i ,o i ,c i} i∈P , which represent the position, rotation, scale, opacity, and color of the Gaussian point in turn. P represents the number of Gaussian points. The three-dimensional Gaussian between frames should maintain a corresponding relationship to support the calculation of the motion offset of the three-dimensional Gaussian. For the continuous point clouds within a group of video frames, the first-frame point cloud within the group is retained, and the motion offset of the point clouds of the remaining frames relative to the three-dimensional Gaussian points in the first-frame point cloud is calculated. Each group of video frames is represented as a combination of a single point cloud and its motion offset {P m ,D m,t} m∈M,t∈G , where M = ceil(F / G) is the number of groups of video frames, and F is the total number of video frames, which is convenient for further compression later and also beneficial to video streaming transmission.

[0032] (2) Point cloud compression: Use a variety of compression techniques to compress the first-frame point cloud P of each group of video frames m . When using bit quantization and vector quantization to compress the first-frame point cloud of each group, it is realized through the quantization-aware training method. It is also possible to directly quantize the trained point cloud, but the effect is not as good as the quantization-aware training method.

[0033] For the high-dimensional features of Gaussian attributes, vector quantization technology is adopted to compress the high-order spherical harmonic coefficients. Specifically, the color parameters in the Gaussian point cloud are expressed as c = {c base ,c rest} by spherical harmonic functions, where c restis the coefficient of the high-order spherical harmonic function, which occupies 94% of the storage of the color parameters but only contributes about 15% of the emitted light radiation energy. Therefore, the present invention selects to use vector quantization to compress c rest substantially. The specific approach is to perform K-Means clustering on the c rest vectors of the Gaussian points to obtain K categories, and calculate the average value of the vectors in each category as the clustering center for storage. During rendering, each Gaussian point uses the corresponding clustering center vector to replace the original c rest for rendering.

[0034] For other parameters {q, s, o, c base} of the Gaussian points, which represent the rotation, scale, opacity, and RGB color (the first-order coefficient of the spherical harmonic function) of the Gaussian points respectively, the present invention adopts a bit quantization method to further save storage space. The goal of bit quantization is to discretize the parameters usually stored using 32-bit floating-point numbers into integer numbers within their value ranges to use fewer bits for storage. Among them, q ∈ (-1, 1), o ∈ (0, 1), c base ∈ (0, 1). The value ranges of these parameters are small enough to be quantized using 8-bit integer numbers. Although s does not have a fixed value range, it is observed in experiments that its value range is usually distributed in the range of (0, 1) times the scene scale. Therefore, the present invention uses 12 bits to discretize and quantize s for storage within the range of [2 -12 , 1] times the scene scale.

[0035] For the three-dimensional Gaussian point cloud stored in point cloud format, range Asymmetric Numeral Systems (rANS) encoding is used for further compression to obtain the Figure 1 binary point cloud file in.

[0036] (3) Motion offset compression: Further compress the four-dimensional Gaussian video based on the existing video compression technology. For the motion offset D m,t , the present invention uses H.265 encoding for compression. The motion offset D m,t in each group of video frames contains the offsets of the positions and rotations of the Gaussian points at each moment within a certain range of time relative to the Gaussian points in the first frame {Δx i,t , Δq i,t} i∈P,t∈G , which can be expressed as Δx i,t = x i,t - x i , Δq i,t = q i,t / q i , x i and q irespectively represent the position and rotation of the current Gaussian point in the first frame of the group, x i,t and q i,t respectively represent the position and rotation of the current Gaussian point in the t-th frame of the group. This type of temporal information is suitable for compression using video coding. First, the present invention sorts the offsets {Δx i,t , Δq i,t} using Morton sorting according to the magnitude of Δx i,t}, obtains the pixel coordinates corresponding to each Gaussian point in the three-dimensional space in the two-dimensional image space, and stores the offsets {Δx i,t , Δq i,t} according to this coordinate to obtain a flattened motion offset map. The size of the motion offset map is H×W×C; H and W are the dimensions of the offset map, and H×W must be greater than the total number of dynamic points to accommodate the motion offsets of all dynamic points; C is the number of channels of the motion offset map. Specifically, C = 10, including six channels of Δx i,t and four channels of Δq i,t . Before encoding the temporal motion offset map into a video, it is also necessary to perform bit quantization on the motion offset map: Since the position of the Gaussian point has a greater impact on the rendering effect, the present invention uses 16-bit quantization for Δx i,t , to obtain two 3-channel 8-bit RGB maps (denoted as high-bit Δx and low-bit Δx); for Δq i,t , 8-bit quantization is used to obtain a 4-channel 8-bit grayscale map (with a shape of H×W×4), and it is spliced into a single-channel 8-bit grayscale map with a shape of 4H×W×1 (denoted as spliced Δq). Then, high-bit Δx, low-bit Δx, and spliced Δq are respectively processed into videos using H.265 encoding, where lossless encoding is used for high-bit Δx, and lossy encoding is used for low-bit Δx and spliced Δq, to obtain three corresponding video files, that is, the motion offset videos in Figure 1 .

[0037] (4) Four-dimensional Gaussian video streaming: As shown in Figure 2 , for the compressed grouped four-dimensional Gaussian video data, streaming transmission and real-time rendering are realized, and multi-threading technology is used for parallel transmission and rendering. The thread responsible for transmission sequentially receives the video frame group and restores the three-dimensional Gaussian point cloud. This thread receives the motion offset video and the binary point cloud file of each video frame group from the Internet on the CPU, extracts the offset map frame by frame from the motion offset video, restores its order according to the Morton order, and restores the integer number to a floating point number (1 in Figure 2 ); performs the inverse operations of rANS encoding, vector quantization, and bit quantization on the binary point cloud file in sequence to restore the first-frame point cloud (2 in Figure 2 ). Calculate the restored motion offset through x i,t = x i + Δx i,t , qi,t = q i ·Δq i,t Added to the first-frame point cloud, the Gaussian point cloud of each frame of the video frame group can be obtained. On the CPU, the Gaussian point cloud is mapped to the texture map in row-major order ( Figure 2 in 3)), and the Gaussian point cloud attributes are passed to the rendering pipeline in the texture map format.

[0038] The thread responsible for rendering processes the response interaction input, calculates the current viewing angle and the video playback progress according to the interaction input (such as dragging the screen or the progress bar) ( Figure 2 in 4)). According to the playback progress, the three-dimensional Gaussian point cloud that should be rendered currently is confirmed in the form of a callback function, and the current frame is drawn using the rendering pipeline in combination with the current viewing angle. The drawing is based on the traditional graphics rendering pipeline rather than the general parallel computing architecture of NVIDIA graphics cards. The current viewing angle and the three-dimensional Gaussian point cloud to be rendered are passed into the GPU in the texture format, and then the three-dimensional Gaussian point cloud is drawn on the GPU. A primitive is assigned to each three-dimensional Gaussian point, and the projection of the three-dimensional Gaussian in the screen space is calculated according to the Gaussian attributes in the vertex shader of the primitive being drawn. The primitive representing the Gaussian enters the hardware rasterization stage, making full use of the speed advantage of GPU acceleration, and quickly obtaining various attributes of the pixels covered by the three-dimensional Gaussian projection through interpolation, including its relative position within the projection, uncalculated color, and opacity. The rasterized fragment shader calculates the exponential part of the Gaussian function to obtain the final color and opacity of the three-dimensional Gaussian projection on each pixel. Finally, through alpha blending from front to back, a realistic rendered image is obtained. Among them, since this rendering essentially belongs to the rendering of semi-transparent objects, the rendering order of the primitives is crucial for the correctness of the rendering result. When the input interaction changes the current viewing angle, the present invention performs a fast bucket sort on all three-dimensional Gaussian points on the CPU, and determines the rendering order of the primitives according to the sorting result.

[0039] The comparative experiment proves the advantages of the present invention compared with the prior art solutions. Table 1 shows the index comparison between the present invention and other prior art solutions on the widely used Neu3DV dataset. The comparison metrics include: Peak Signal-to-Noise Ratio (PSNR) for measuring the rendering quality, and the higher the value, the higher the rendering quality; storage overhead, measured in MBtyes. The data in the table are all from the data published by each technical solution, where "N / A" means that the corresponding data index of the technical solution is not published. It can be seen from Table 1 that compared with the prior art solutions, the present invention simultaneously achieves the highest rendering quality and the smallest storage overhead.

[0040] Table 1. Quantitative comparison between the present invention and the prior art solutions on the Neu3DV dataset

[0041]

[0042] Meanwhile, ablation experiments have demonstrated the necessity of the core steps in the present invention. Table 2 lists the performance of the system on the Neu3DV dataset after stripping different key steps in the present invention, to illustrate the role played by the key steps in the entire system. It can be seen that point cloud quantization compression further reduces the storage overhead without sacrificing the rendering quality, and motion offset compression significantly reduces the storage overhead at the cost of a basically negligible loss in rendering quality.

[0043] Table 2, Influence of the Main Steps in the Present Invention on the Entire System

[0044]

[0045] After testing on multiple platforms, including high-performance servers, personal laptops, and mobile devices, the compressed four-dimensional Gaussian video can be streamed under public network bandwidth conditions (about 30 Mbps). Meanwhile, thanks to the packet design and the multi-threaded working paradigm, the four-dimensional Gaussian video streaming playback method designed in the present invention allows real-time rendering of four-dimensional Gaussian videos at a rendering speed of ≥ 60 FPS, and the final rendering speed is limited by the refresh rate of the device screen.

[0046] Corresponding to the foregoing embodiment of a four-dimensional Gaussian video compression and streaming playback method based on dynamic point cloud compression and real-time rendering, the present invention also provides an embodiment of a four-dimensional Gaussian video compression and streaming playback device based on dynamic point cloud compression and real-time rendering.

[0047] See Figure 3 , an embodiment of a four-dimensional Gaussian video compression and streaming playback device based on dynamic point cloud compression and real-time rendering provided by the embodiment of the present invention includes a memory and one or more processors. Executable code is stored in the memory. When the processor executes the executable code, it is used to implement a four-dimensional Gaussian video compression and streaming playback method in the foregoing embodiment.

[0048] An embodiment of a four-dimensional Gaussian video compression and streaming playback device provided by the present invention can be applied to any device with data processing capabilities, and the any device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware, or by a combination of software and hardware. Taking software implementation as an example, as a logically meaningful device, it is formed by a processor of any device with data processing capabilities reading the corresponding computer program instructions in the non-volatile memory into the memory for operation. From the hardware level, such as Figure 3As shown, it is a hardware structure diagram of any device with data processing capabilities where a four-dimensional Gaussian video compression and streaming playback device based on dynamic point cloud compression and real-time rendering provided by the present invention is located. Except for Figure 3 the shown processor, memory, network interface, and non-volatile memory, any device with data processing capabilities where the device in the embodiment is located usually may further include other hardware according to the actual functions of the any device with data processing capabilities, which will not be elaborated herein.

[0049] The implementation processes of the functions and roles of each unit in the above device are specifically described in detail in the implementation processes of the corresponding steps in the above method, which will not be elaborated herein.

[0050] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0051] The embodiment of the present invention also provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, it implements a four-dimensional Gaussian video compression and streaming playback method in the above embodiment.

[0052] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a Smart Media Card (SMC), an SD card, a Flash Card, etc. equipped on the device. Further, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by the any device with data processing capabilities, and can also be used to temporarily store the data that has been output or will be output.

[0053] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the four-dimensional Gaussian video compression and streaming playback method based on dynamic point cloud compression and real-time rendering. In summary, the present invention provides a four-dimensional Gaussian video compression and streaming playback method based on dynamic point cloud compression and real-time rendering. Its characteristics of high quality and low storage support high-quality volume video reconstruction and online live broadcast. The above embodiments are used to explain the present invention rather than limit the present invention. Any modification and change made to the present invention within the spirit and scope of the claims of the present invention fall within the protection scope of the present invention.

Claims

1. A four-dimensional Gaussian video compression and streaming playback method based on dynamic point cloud compression and real-time rendering, characterized in that, The method includes the following steps: (1) Video grouped expression: Obtain four-dimensional Gaussian video data and group it. According to the time sequence, input several consecutive three-dimensional Gaussian point clouds into multiple video frame groups with the same number of frames; for the consecutive point clouds within a video frame group, retain the first-frame point cloud in the group, and calculate the motion offset of the point clouds in the remaining frames relative to the three-dimensional Gaussian points in the first-frame point cloud. Represent each video frame group as a combination of a single point cloud and its motion offset; (2) Point cloud compression: Compress the first-frame point cloud in each group. For the Gaussian attributes of the three-dimensional Gaussian point cloud, adopt bit quantization technology to convert some floating-point type Gaussian attributes into integers; for the high-dimensional features of the Gaussian attributes, adopt vector quantization technology to compress the high-order spherical harmonic coefficients; for the three-dimensional Gaussian point cloud stored in point cloud format, adopt range asymmetric numeral system rANS coding for further compression to obtain a binary point cloud file; (3) Motion offset compression: Use Morton sorting for the three-dimensional Gaussian point cloud distributed in three-dimensional space, map the motion offsets of the three-dimensional Gaussian points one by one to the pixels of a two-dimensional image, and then use video compression technology to compress the occupied space of the motion offsets; (4) Four-dimensional Gaussian video streaming playback: For the compressed grouped four-dimensional Gaussian video data, implement streaming transmission and real-time rendering; the specific process is as follows: Adopt multi-thread technology for parallel transmission and rendering. The thread responsible for transmission sequentially receives video frame groups and restores the three-dimensional Gaussian point cloud. The thread responsible for rendering uses the graphics rendering pipeline to transfer the three-dimensional Gaussian point cloud in texture mapping format, and calculates Gaussian splashes based on the rendering perspective and 3DGS attributes in the vertex shader and fragment shader to generate a rendered image.

2. The four-dimensional Gaussian video compression and streaming method based on dynamic point cloud compression and real-time rendering according to claim 1, characterized in that Each Gaussian point in the acquired four-dimensional Gaussian video data contains a series of attributes {x i , q i , s i , o i , c i}, i∈P which represent the position, rotation, scale, opacity, and color of the Gaussian point in sequence. The three-dimensional Gaussians between frames should maintain a corresponding relationship to support the calculation of the motion offset of the three-dimensional Gaussian.

3. The four-dimensional Gaussian video compression and streaming method based on dynamic point cloud compression and real-time rendering according to claim 1, characterized in that, Divide every consecutive 30 frames into a group, and group and compress the point cloud and its motion offset.

4. The four-dimensional Gaussian video compression and streaming playback method based on dynamic point cloud compression and real-time rendering according to claim 1, characterized in that, In step (2), adopt a quantization-aware training method to introduce bit quantization and vector quantization.

5. The four-dimensional Gaussian video compression and streaming playback method based on dynamic point cloud compression and real-time rendering according to claim 1, characterized in that, When using Morton sorting for the three-dimensional Gaussian point cloud, the sorting is based on the position offset of the second-frame point cloud in the group relative to the first-frame point cloud; after quantizing the floating-point type position offset into the three-dimensional coordinates of an integer, calculate the Morton code for sorting; for the pixels of the two-dimensional image, calculate the Morton code for sorting the pixel coordinates; the three-dimensional point cloud coordinates sorted by the Morton code and the two-dimensional pixel coordinates sorted by the Morton code are mapped one by one, satisfying the bijective relationship.

6. The four-dimensional Gaussian video compression and streaming method based on dynamic point cloud compression and real-time rendering according to claim 1, wherein The motion offsets to be compressed include the offsets of the position attribute and the rotation attribute; when mapping the position offset of the three-dimensional Gaussian point to the two-dimensional image, quantize the floating-point type position offset into a 16-bit integer, and save the high 8 bits and the low 8 bits to two RGB images respectively; when mapping the rotation offset to the two-dimensional image, expand the four-dimensional rotation vector and save it as a grayscale image; when adopting video coding compression, perform lossless compression on the high 8-bit image of the position offset and perform lossy compression on the remaining images.

7. The four-dimensional Gaussian video compression and streaming method based on dynamic point cloud compression and real-time rendering according to claim 1, characterized in that, The streaming playback is based on the graphics rendering pipeline. The projection of the 3DGS on the two-dimensional plane is calculated in the vertex shader, and through the hardware rasterization of the graphics card, the exponential part of the Gaussian function is calculated in the fragment shader to obtain the color and opacity of each pixel. Finally, the rendered image is obtained through alpha blending.

8. A four-dimensional Gaussian video compression and streaming playback device based on dynamic point cloud compression and real-time rendering, comprising a memory and one or more processors, wherein executable code is stored in the memory, characterized in that, When the processor executes the executable code, it implements a four-dimensional Gaussian video compression and streaming playback method based on dynamic point cloud compression and real-time rendering as described in any one of claims 1-7.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a four-dimensional Gaussian video compression and streaming playback method based on dynamic point cloud compression and real-time rendering as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements a four-dimensional Gaussian video compression and streaming playback method based on dynamic point cloud compression and real-time rendering as described in any one of claims 1-7.

Images