Immersive video coding method and system based on 3DGS

Abstract

The invention provides an immersive video coding method and system based on 3DGS, and the method comprises the steps: carrying out the sparse reconstruction of each frame of multi-view video data, and obtaining an initial point cloud; determining anchor points of three-dimensional Gaussian distribution in each frame; extracting spatial context information of the anchor points through multi-resolution hash coding, and splicing the spatial context information with the features of the anchor points to form fusion features; predicting parameters of three-dimensional Gaussian distribution corresponding to each anchor point through a neural network by using the fusion features and camera parameters, and obtaining a rendered image; calculating the color loss between the rendered image and the original video frame, and optimizing the parameters of the anchor points; performing quantization and entropy coding on the optimized parameters of the anchor points; in combination with the color loss and the coding rate, performing rate distortion optimization on the quantization parameter to generate compressed three-dimensional scene representation; and repeating the steps for each frame of the multi-view video, and finally outputting a compressed immersive video code stream. According to the invention, high-quality representation and efficient compression of the immersive video are realized, and the rate-distortion performance is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an immersive video coding method and system based on 3DGS. Background Technology

[0002] Immersive video, through multi-view acquisition and dynamic scene reconstruction, empowers users to freely switch viewing angles and is widely considered an important direction for the development of digital media. This technology can serve multiple fields such as virtual reality, distance education, and digital twins, but it also places higher demands on the real-time interactive performance and visual realism of the video. Due to the massive data volume, traditional encoding schemes often struggle to achieve an ideal balance between efficient compression and high-fidelity rendering, which has become a key technical bottleneck restricting its large-scale deployment.

[0003] Novel viewpoint synthesis, a core technology for immersive video, can generate images from any observation position based on limited viewpoint information. Early methods (such as depth estimation and optical flow interpolation) are prone to geometric distortion in complex scenes. Neural Radiation Field (NeRF), which has emerged in recent years, achieves high-quality rendering using implicit neural representations, but its reliance on dense sampling and iterative optimization incurs high computational costs, making it difficult to meet real-time interactive requirements. Although some methods based on explicit meshes or voxels have improved rendering speed, they often struggle to balance geometric accuracy with storage costs.

[0004] Against this backdrop, 3D Gaussian Splatting (3DGS) technology has attracted widespread attention in academia due to its explicit point cloud representation and differentiable rendering mechanism. This technology represents a scene as a series of learnable Gaussian primitives, adaptively adjusting their spatial position, scale, and transparency through training, thereby achieving efficient rendering. While maintaining NeRF-level reconstruction quality, 3DGS significantly reduces computational complexity and storage footprint. Experiments show that 3DGS can achieve real-time rendering performance of over 100 frames per second in static scenes, opening up new avenues for real-time interactive applications in immersive video.

[0005] However, raw 3DGS still has certain limitations. The storage space required for single-scene modeling remains relatively large; for example, modeling the Flowers scene in the MipNeRF360 dataset requires 814.2MB, which still poses a challenge for transmission and storage in practical applications. Therefore, further optimizing the bitrate compression efficiency of 3DGS and constructing a dedicated coding framework suitable for immersive videos have become key issues driving the practical application of this technology.

[0006] A search revealed a Chinese patent with patent application number 202510284768.4, which discloses a multi-view video compression method and its application. This method constructs a voxel space based on the sparse point cloud of the first frame of a multi-view video, initializes the static / dynamic features of anchor points, and uses quantization offset and neural network prediction of Gaussian properties for joint optimization, combining rendering loss (Loss1) and bitrate loss (Loss2). It also utilizes hash encoding and a shared MLP to generate quantization offset parameters, and finally obtains the compression result through entropy encoding. However, it lacks explicit multi-resolution hash context modeling, a rate-distortion adaptive optimization mechanism, and does not classify anchor points, resulting in insufficient synergistic optimization between compression efficiency and reconstruction quality. Summary of the Invention

[0007] In view of the deficiencies in the prior art, the purpose of this application is to provide an immersive video coding method and system based on 3DGS.

[0008] A first aspect of this application provides a 3DGS-based immersive video coding method, comprising: Based on the input multi-view video, sparse reconstruction is performed on each frame to obtain the initial point cloud of the scene.

[0009] Based on the spatial location of the initial point cloud, determine the anchor points of the three-dimensional Gaussian distribution in each frame; The spatial context information of the anchor point is extracted by multi-resolution hash encoding and then concatenated with the independent features of the anchor point to form a fused feature. Using the fusion features and camera parameters, the parameters of the three-dimensional Gaussian distribution corresponding to each anchor point are predicted by a neural network, and then the reconstructed rendered image is obtained by differentiable rendering. Calculate the color loss between the rendered image and the corresponding frames of the original multi-view video, and optimize the parameters of the anchor point; The parameters of the optimized anchor points are quantized and entropy encoded. By combining the color loss and entropy coding rate, rate-distortion optimization is performed on the quantized parameters to generate a compressed 3D scene representation; Repeat the above steps for each frame of the multi-view video to finally output a compressed video stream.

[0010] Optionally, each anchor point corresponds to an aggregated representation of one or more three-dimensional Gaussian distributions; Determining the three-dimensional Gaussian distribution through the anchor points includes: Define the parameters for each anchor point, including position, offset, scale, and independent features; Define the parameters for each of the three-dimensional Gaussian distributions, including position, covariance, transparency, and color; The position of each three-dimensional Gaussian distribution is determined based on the position, offset, and scale of each anchor point; Based on the independent features of each anchor point, the relative distance between the current viewpoint and the anchor point, and the current viewpoint direction, the covariance, transparency map, and color of the three-dimensional Gaussian distribution are obtained.

[0011] Optionally, the step of extracting the spatial context information of the anchor point through multi-resolution hash encoding and concatenating it with the independent features of the anchor point to form a fused feature includes: Construct a multi-resolution spatial hash grid to encode three-dimensional spatial locations; For each anchor point, based on its position, lookup and interpolation are performed at each resolution level of the hash grid to extract the corresponding multi-scale spatial context features; The multi-scale spatial context features are concatenated with the independent features of the anchor point to form a fusion feature for subsequent neural prediction.

[0012] Optionally, the quantization and entropy encoding of the optimized anchor point parameters includes: Based on the spatial context information provided by the multi-resolution hash grid, the optimal quantization step size is determined for the anchor point attribute parameters of different spatial regions. Based on the quantization step size, the features, offset, scale, and position parameters of the anchor point are scalar or vector quantized respectively, and the continuous parameter values ​​in the anchor point parameters are mapped to discrete quantization indices. The quantized anchor point position parameters are compressed and encoded using the GPCC standard; Output the entropy-encoded bitstream.

[0013] Optionally, the step of combining the color loss and entropy coding rate to perform rate-distortion optimization on the quantized parameters to generate a compressed 3D scene representation includes: Calculate the color fidelity loss between the reconstructed rendered image and the original input multi-view video frames; Estimate the bitrate loss corresponding to the anchor parameters after quantization and entropy coding, the bitrate loss being based on the actual number of bits generated by coding or obtained through a pre-trained bitrate estimation network; The fidelity loss and the bitrate loss are weighted and summed according to a set weight to form a joint optimization objective function; The joint optimization objective function is minimized using the gradient descent algorithm, and the parameters are dynamically adjusted to ensure reconstruction quality while reducing the final bitrate.

[0014] Optionally, the step of weighting and summing the fidelity loss and the bitrate loss according to a set weight to form a joint optimization objective function includes: ; in: For rendering loss, To control the coefficients for the bit rate and quality tradeoff, For entropy loss, The table normalizes the number of anchor points and dimension for the entropy loss, where N is the total number of anchor points; D a Let K be the feature dimension of the anchor point, and K be the number of Gaussian functions associated with each anchor point. For the quantization loss of binary hash grid, To adjust the pruning intensity at the anchor points, This is the anchor mask loss.

[0015] Optionally, during the initial training rounds, all anchor points are classified, and the classification results are used after the anchor points are quantized. The classification of all anchor points refers to dividing them into three categories—root anchor points, leaf anchor points, and free anchor points—based on feature relevance. Specifically: Based on the spatial distribution and feature similarity of anchor points, a selection algorithm is used to divide two anchor points with similar features into root anchor points and leaf anchor point pairs. Anchor points that fail to be assigned to any pair by the selection algorithm are classified as free anchor points; The root anchor point is used to provide contextual space information for its associated leaf anchor points during entropy encoding. The features of the leaf anchor points are predicted by the features of their respective root anchor points, while the free anchor points are encoded independently.

[0016] A second aspect of this application provides an immersive video coding system based on 3DGS, comprising: Initialization module: Based on the input multi-view video, sparse reconstruction is performed on each frame to obtain the initial point cloud of the scene; 3D Gaussian Anchor Point Construction Module: Based on the spatial location of the initial point cloud, determine the anchor points of the 3D Gaussian distribution in each frame; Feature fusion module: Extracts spatial context information of anchor points through multi-resolution hash encoding and concatenates it with the independent features of anchor points to form fused features; Gaussian point generation and rendering module: Utilizing fused features and camera parameters, the parameters of the 3D Gaussian distribution corresponding to each anchor point are predicted through a neural network, and then the reconstructed rendered image is obtained through differentiable rendering. Optimization module: Calculates the color loss between corresponding frames of the rendered image and the original multi-view video, and optimizes the parameters of the anchor points; Quantization and Encoding Module: Performs quantization and entropy encoding on the parameters of the optimized anchor points; Rate-distortion optimization module: Combines color loss and entropy coding rate to perform rate-distortion optimization on the quantized parameters and generate a compressed 3D scene representation; The compressed bitstream generation module repeats the above steps for each frame of the multi-view video, and finally outputs a compressed video bitstream.

[0017] A third aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the methods described herein.

[0018] A fourth aspect of this application provides an electronic device comprising: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of any of the methods.

[0019] The immersive video coding method based on 3DGS provided in this application adopts a point cloud initialization method based on multi-view video frames and 3D sparse reconstruction to initialize the spatial and attribute information of 3D Gaussian anchor points, which significantly improves the accuracy of the immersive video reconstruction process and the quality of the rendered image. By constructing a multi-resolution hash grid to extract spatial context features, the expressive power of spatial features is enhanced, and the representation performance of 3D Gaussian distribution in complex scenes is improved. By performing adaptive quantization of Gaussian anchor point attributes based on spatial context and combining the correlation between anchor points for context-adaptive arithmetic coding, the compression efficiency is significantly improved, the rate-distortion performance is optimized, and a higher compression efficiency ratio is achieved.

[0020] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0021] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating an immersive video coding method based on 3DGS, according to an exemplary embodiment.

[0022] Figure 2 This is a schematic diagram illustrating a video segmentation and point cloud initialization process according to an exemplary embodiment.

[0023] Figure 3 This is a schematic diagram illustrating an immersive video representation and compression encoding process according to an exemplary embodiment.

[0024] Figure 4 This is a schematic diagram of an anchor context hierarchy according to an exemplary embodiment.

[0025] Figure 5 This is a schematic diagram illustrating a comparison of the rate-distortion performance of a method provided in this disclosure with that of an existing baseline method on a typical immersive video sequence, according to an exemplary embodiment.

[0026] Figure 6 This is a structural block diagram of an immersive video coding system based on 3DGS, according to an exemplary embodiment. Detailed Implementation

[0027] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0028] Under current technological conditions, video coding struggles to efficiently model the 3D space and multi-view correlations of immersive videos, making it difficult to simultaneously meet the three requirements of low bitrate transmission, high-fidelity rendering, and real-time interaction. To address these issues, this application provides a 3DGS-based immersive video coding method.

[0029] Reference Figure 1 As shown in one embodiment of this application, an immersive video coding method based on 3DGS may employ the following steps: S100 performs sparse reconstruction on each frame based on the input multi-view video data to obtain the initial point cloud of the scene; S200, based on the spatial position of the initial point cloud obtained from S100, determines the anchor point of the three-dimensional Gaussian distribution in each frame; S300 extracts the spatial context information of the anchor points obtained from S200 through multi-resolution hash coding, and then concatenates it with the independent features of the anchor points to form a fused feature. S400 uses the fusion features and camera parameters obtained from S300 to predict the parameters of the three-dimensional Gaussian distribution corresponding to each anchor point through a neural network; then, the reconstructed rendered image is obtained through differentiable rendering. S500 calculates the color loss between the rendered image and the input original video frame, and optimizes the parameters of the anchor point; S600 performs quantization and entropy coding on the parameters of the optimized anchor points; The S700 combines color loss and entropy coding rate to optimize the quantization parameters for rate distortion, generating a compressed 3D scene representation. The S800 repeats the above steps for each frame of the multi-view video, and finally outputs a compressed immersive video stream.

[0030] The embodiments described above employ a point cloud initialization method based on multi-view video frames and 3D sparse reconstruction to initialize the spatial and attribute information of 3D Gaussian anchor points, significantly improving the accuracy of the immersive video reconstruction process and the quality of the rendered image. By constructing a multi-resolution hash grid to extract spatial context features, the expressive power of spatial features is enhanced, improving the representation performance of 3D Gaussian distribution in complex scenes. By performing adaptive quantization of Gaussian anchor point attributes based on spatial context and combining the correlation between anchor points for context-adaptive arithmetic coding, compression efficiency is significantly improved, rate-distortion performance is optimized, and a higher compression efficiency ratio is achieved.

[0031] To obtain an initial point cloud as prior knowledge for image reconstruction, some specific embodiments of this application, such as... Figure 2 As shown in step S100, based on the input multi-view video data, sparse reconstruction is performed on each frame to obtain the initial point cloud of the scene. This can be achieved through the following steps: S101: The input multi-view video is segmented to obtain multi-view video frames; S102, perform distortion correction preprocessing on the multi-view video frames obtained in S101 to eliminate the impact of lens distortion on reconstruction accuracy, and obtain distortion-corrected multi-view video frames.

[0032] S103 utilizes the principle of multi-view geometry to extract and match feature points in the distorted multi-view images, establishing correspondences between images.

[0033] S104 calculates the camera parameters and sparse 3D point cloud for each image based on the feature point matching results using the motion reconstruction structure algorithm.

[0034] For example, the sparse reconstruction process of S102 to S104 described above can be implemented using the COLMAP tool.

[0035] The embodiments described above in this application provide high-precision spatial priors for subsequent Gaussian anchor point initialization through the precise sparse reconstruction and screening process based on multi-view geometry, which significantly improves the accuracy and convergence speed of immersive video representation.

[0036] This application constructs an adjustable, optimizable, and compressible 3D Gaussian representation structure for sparse point clouds, using it as the basic unit for rendering, optimization, and encoding. This transforms discrete point clouds into a continuous, smooth, and efficient scene representation. The application employs an anchor point structure to normalize the 3D Gaussian distribution and utilizes a neural network to obtain the 3D Gaussian function distribution. In some specific embodiments of this application, step S200, based on the spatial position of the initial point cloud obtained in S100, determines the anchor points of the 3D Gaussian distribution in each frame, such as... Figure 3 As shown, the following steps may be included: S201 defines the anchor point structure and a three-dimensional Gaussian function.

[0037] The structure of each anchor point includes its location. Offset ,scale Independent features of anchor points .

[0038] Specifically, position refers to the center coordinates of the anchor point in three-dimensional space. Offset is a position offset vector generated from the anchor point using K Gaussian functions. Scale refers to the actual scaling ratio that controls the offset. Anchor point independent features refer to the embedded features of the anchor point's semantic and geometric properties.

[0039] Each three-dimensional Gaussian function distribution includes location parameters. Covariance parameter Transparency parameters With color coefficient .

[0040] S202 initializes the parameters of the anchor point.

[0041] Specifically, the anchor point location can be obtained using a sparse reconstruction method. The remaining offsets, scales, and anchor point-specific features can be initialized using preset rules or a lightweight network.

[0042] S203, determine the position of each three-dimensional Gaussian function distribution based on the position, offset and scale of each anchor point structure; ; S204, based on anchor-point independent features and combined with viewpoint-related information, utilizes a neural network. Decode the complete properties of each 35 Gaussian function.

[0043] Specifically, based on the anchor point features of each anchor point structure and their relative distance Viewpoint and direction Decoding the properties of associated Gaussians: ; Indicates the anchor point's independent feature. The relative distance between the current pixel / viewpoint and the anchor point. Indicates the current viewpoint direction; Indicates color, Represents a rotation quaternion, used to construct the covariance matrix. Indicates scale. Indicates transparency.

[0044] These four properties can be used to determine a three-dimensional Gaussian function, which can then be rendered on a 2D image.

[0045] In the embodiments described above, by defining and generating anchor points and three-dimensional Gaussian functions, overlapping and redundant Gaussian functions can be reduced, thereby achieving model compression; adaptive view rendering can be achieved; and rendering quality and the ability to handle complex scenes can be improved.

[0046] To improve the accuracy of subsequent Gaussian parameter prediction and provide richer structured information for compressed encoding, in some specific embodiments of this application, step S300, extracting the spatial context information of the anchor points obtained in step S200 through multi-resolution hash encoding and concatenating it with the independent features of the anchor points to form fused features, may include the following steps: S301, construct a multi-resolution spatial hash grid for efficient encoding of three-dimensional spatial locations; Specifically, the multi-resolution spatial hash grid is an existing network containing multiple levels of different resolutions. Any 3D coordinate point in space can be used to query this grid model. The query process involves interpolation calculations at different resolution levels to ultimately obtain a comprehensive spatial context feature vector. S302, for each anchor point, based on its spatial coordinates Hash grid built on S301 Searching and interpolation are performed at each resolution level to extract the corresponding multi-scale spatial context features; S303, Combine the multi-scale spatial context features obtained in S302 with the independent features of the anchor point. The features are then concatenated and combined to form a fusion feature representation for subsequent neural prediction, specifically: ; It is worth noting that the multi-scale spatial context features obtained from the multi-resolution spatial hash grid are also used to generate encoding context through neural networks to assist in the encoding and storage of anchor information.

[0047] The fused features obtained in the above embodiments of this application not only include the unique features of the anchor point itself, but also integrate spatial environmental information of different scales around it, thereby improving the expressive power of the fused features and providing a comprehensive basis for subsequent prediction steps. Furthermore, using multi-scale spatial context features to generate encoded context-assisted anchor point information storage can utilize the correlation of spatial information to compress redundant data, improve the efficiency of anchor point information encoding and storage, and simultaneously achieve both the accuracy of scene representation and the compactness of data, providing a foundation for subsequent rendering, optimization, and encoding operations.

[0048] To improve rendering, in some specific embodiments of this application, in step S400, the parameters of the three-dimensional Gaussian distribution corresponding to each anchor point are predicted using a neural network (the neural network used in this application is a multilayer perceptron MLP) by utilizing fused features and camera parameters; then, the reconstructed rendered image is obtained through differentiable rendering, which can be achieved through the following steps: S401, the fused features output by S300 and the camera parameters of the current frame are concatenated to form a joint input; S402 takes the combined input of S401 and feeds it into the MLP, outputting the three-dimensional Gaussian distribution parameters corresponding to the anchor point.

[0049] This process is the same as S203-S204 above, obtaining the Gaussian parameter set. .

[0050] S403, input the Gaussian distribution parameters obtained above into the differentiable renderer. The reconstructed rendered image is generated by the renderer. ,Right now ; in, Indicates the first The location center of a Gaussian distribution and Let represent the rotation and scaling components of its covariance, respectively. Indicates the transparency parameter. This represents the color coefficient.

[0051] Specifically, the differentiable renderer employs a rasterization strategy based on Gaussian splatting, projecting the predicted 3D Gaussian distribution onto the image plane of the current viewpoint one by one, determining its influence area on the pixel grid, and using a front-to-back alpha blending algorithm to fuse the color, transparency, and spatial distribution characteristics of each Gaussian to generate a continuous and differentiable reconstructed image.

[0052] In the embodiments described above, the parameters of the three-dimensional Gaussian distribution of each anchor point are predicted based on the fusion features, and a refined rendered image is obtained based on the predicted parameters.

[0053] To significantly reduce the amount of data and facilitate storage and network transmission, in some specific embodiments of this application, in step S600, the parameters of the anchor points optimized in step S500 are quantized and entropy encoded, which can be achieved by the following steps: S601 adaptively determines the optimal quantization step size for anchor point attribute parameters in different spatial regions based on the spatial context information provided by the multi-resolution hash grid.

[0054] Specifically, the position of each Gaussian anchor point is input into this multi-resolution hash grid, and the output is the spatial context information corresponding to each Gaussian anchor point. This spatial context information is then input into a neural network (here, the neural network refers to a multilayer perceptron (MLP)) to obtain the optimal quantization step size.

[0055] S602, based on the quantization step size obtained in S601, performs scalar or vector quantization on the features, offset, scale and position parameters of the anchor point respectively, and maps the continuous parameter values ​​in the anchor point parameters to discrete quantization indices. Specifically: During the training phase, the quantization formula is as follows: ; During the testing phase, the quantization formula is as follows: ; in , ; These are the quantified features. Features before quantization A randomly generated uniform distribution is used to simulate the deviations that occur during the actual quantization process (this simulation is only performed during the training phase). is the quantization step size, and Round is the rounding step in the quantization operation. It is a hyperparameter representing the initial quantization step size. The spatial context information is represented by a multi-resolution hash grid output.

[0056] S603 further compresses and encodes the anchor point position parameters after S602 quantization using the GPCC standard; the final output will be the bitstream after entropy encoding.

[0057] The embodiments described above in this application balance compression and fidelity through adaptive quantization; adaptive quantization for regions with different representational complexities in the scene helps improve the rate-distortion performance of the model.

[0058] Furthermore, to further reduce data redundancy and improve subsequent compression efficiency, in some specific embodiments of this application, all anchor points are classified during the initial training rounds, and the classification results are used after the anchor points are quantized.

[0059] All anchor points are categorized as follows: The quantized anchor points are classified into three categories based on their feature relevance: root anchor points, leaf anchor points, and free anchor points. Figure 4 As shown, it includes: Based on the spatial distribution and feature similarity of anchor points, a selection algorithm is used to divide two anchor points with similar features into root anchor point-leaf anchor point pairs. Anchor points that fail to be assigned to any pair by the selection algorithm are classified as free anchor points; In this context, the root anchor point provides contextual information for its associated leaf anchor points during entropy encoding. The features of a leaf anchor point can be approximately predicted from the features of its associated root anchor point, while free anchor points are encoded independently. Specifically, let the first... The root anchor point and the first Each leaf anchor point constitutes a root-leaf pair, characterized as follows: and Then the context modeling of the leaf anchor point features is as follows: ; ; in This represents the mean of the leaf anchor feature prediction distribution. For other attribute predictions (including the scale of all anchors, the offset (which is a learnable offset), and root / free anchor features), this is not considered. In all cases, the spatial context information provided by the multi-resolution hash grid is used for attribute distribution estimation. ; The above embodiments of this application reduce data redundancy and improve subsequent compression efficiency by leveraging spatial correlation.

[0060] To find the optimal balance between compression ratio and image quality, this application performs rate-distortion optimization. In some specific embodiments of this application, in step S700, rate-distortion optimization is performed on the quantization parameters by combining color loss and encoding bitrate to generate a compressed 3D scene representation, such as... Figure 4 As shown, the following steps can be taken.

[0061] Based on the bit rate of the encoded anchor features and the rendering quality based on SSIM and L1 loss, including: ; in: To account for rendering loss, structural similarity (SSIM), L1 photometric error, and a Gaussian scale regularization term are considered to ensure the visual fidelity of the reconstructed image.

[0062] This represents an estimate of adjustable bit consumption, obtained by adjusting for entropy loss. After normalization, the normalization factor is obtained. Where N is the total number of anchor points. For each anchor feature dimension, 6 and 3K correspond to the encoding costs of parameters such as position, scale, and rotation, respectively. The quantization loss for binary hash grids is designed to enhance the compactness of feature representations and support efficient encoding and storage. To mitigate anchor point mask loss, a sparsity mechanism is used to suppress the activation of redundant anchor points, further reducing the number of effective anchor points and thus lowering the overall bit rate and storage and transmission overhead.

[0063] This model is trained using rate-distortion optimization to balance the relationship between bitrate and fidelity, and this can be achieved by adjusting the tradeoff parameters. This allows for targeted optimization on either the bitrate or fidelity side.

[0064] The above objective function Rate distortion optimization is achieved through the following mechanisms: Rate control items: Normalize the number of anchor points and dimension of the entropy loss; Distortion control items: Inheriting SSIM and L1 photometric loss ensures rendering quality; Trade-off parameters: Controlling the bit rate-quality tradeoff Adjust the pruning intensity at the anchor points.

[0065] In the embodiments described above, the loss function achieves a dynamic balance between rendering quality and compression ratio by comprehensively considering rendering accuracy, bitrate constraints, and anchor sparsity within a unified framework. By introducing a rendering loss term into the loss function, structural similarity (SSIM) and photometric error (L1) can be measured simultaneously, ensuring that the rendered image generated during optimization maintains consistency with the original video frame in terms of brightness, contrast, and local structure, thereby significantly improving visual fidelity. Compared with traditional schemes that only use a single pixel difference term, this design can achieve higher structural fidelity at the same bitrate. The introduction of an adjustable bitrate constraint term, by quantizing and normalizing the variable bit consumption expectation, achieves controllable constraints on compression efficiency. When the anchor feature dimension or the number of anchors changes, this trade-off term can adaptively adjust the overall bit budget, enabling the network to focus on both quality and compression ratio during optimization, avoiding bitrate waste caused by excessive fidelity. The introduction of an anchor mask term promotes more concentrated and sparse anchors in space, reduces the repeated encoding of redundant information, enhances the distinguishability between anchors, and enables the encoder to significantly reduce redundant storage and bandwidth consumption while maintaining key semantic features.

[0066] Based on the same technical concept, other embodiments of this application provide an immersive video encoding system 100 based on 3DGS, such as... Figure 6 As shown, it includes: Initialization module 110: Based on the input multi-view video, sparse reconstruction is performed on each frame to obtain the initial point cloud of the scene; 3D Gaussian Anchor Point Construction Module 120: Based on the spatial position of the initial point cloud, determine the anchor points of the 3D Gaussian distribution in each frame; Feature fusion module 130: Extracts spatial context information of anchor points through multi-resolution hash encoding and concatenates it with the independent features of anchor points to form fused features; Gaussian point generation and rendering module 140: Utilizes fusion features and camera parameters, predicts the parameters of the three-dimensional Gaussian distribution corresponding to each anchor point through a neural network, and then obtains the reconstructed rendered image through differentiable rendering. Optimization Module 150: Calculates the color loss between the rendered image and the corresponding frames of the original multi-view video, and optimizes the parameters of the anchor point; Quantization and Encoding Module 160: Performs quantization and entropy encoding on the parameters of the optimized anchor points; Rate-distortion optimization module 170: Combines color loss and entropy coding rate to perform rate-distortion optimization on the quantized parameters and generate a compressed 3D scene representation; Module 180 for generating compressed bitstream: Repeat the above steps for each frame of the multi-view video, and finally output the compressed video bitstream.

[0067] The specific implementation techniques of each module / unit in the above examples of this application can be referred to the steps of the immersive video coding method based on 3DGS in the above embodiments, and will not be repeated here.

[0068] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0069] The following examples and comparative examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0070] Comparative Example 1 The TanksAndTemples dataset was used as the test sequence. The TanksAndTemples dataset includes two 3D stereo frame scenes. The rendering performance of the 3DGS-based image representation and compression method provided in this application, as well as HAC, CompGS, and RDO-Gaussian methods, was compared. The evaluation metrics were Peak Signal-to-Noise Ratio (PSNR) and Storage Size. A higher PSNR value indicates higher rendered image quality, and a lower Storage Size value indicates better compression effect.

[0071] The comparison results of rendering performance metrics are shown in Table 1 and Figure 5 As shown: Table 1

[0072] As shown in Table 1, compared with other methods, the method in this application performs best in terms of rendering quality and has the most ideal compression effect. This verifies that the method can improve compression efficiency, optimize rate-distortion performance, and achieve a higher compression efficiency ratio.

[0073] The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.

[0074] Based on the same technical concept, in other embodiments of this application, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the above-described 3DGS-based immersive video encoding method, or to run the above-described 3DGS-based immersive video encoding system.

[0075] Based on the same technical concept, in other embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, can be used to perform the above-described 3DGS-based immersive video coding method, or to run the above-described 3DGS-based immersive video coding system.

[0076] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0077] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0078] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0079] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0080] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0082] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0083] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0084] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. An immersive video coding method based on 3DGS, characterized in that, include: Based on the input multi-view video, sparse reconstruction is performed on each frame to obtain the initial point cloud of the scene; Based on the spatial location of the initial point cloud, determine the anchor points of the three-dimensional Gaussian distribution in each frame; The spatial context information of the anchor point is extracted by multi-resolution hash encoding and then concatenated with the independent features of the anchor point to form a fused feature. Using the fusion features and camera parameters, the parameters of the three-dimensional Gaussian distribution corresponding to each anchor point are predicted by a neural network, and then the reconstructed rendered image is obtained by differentiable rendering. Calculate the color loss between the rendered image and the corresponding frames of the original multi-view video, and optimize the parameters of the anchor point; The parameters of the optimized anchor points are quantized and entropy encoded. By combining the color loss and entropy coding rate, rate-distortion optimization is performed on the quantized parameters to generate a compressed 3D scene representation; Repeat the above steps for each frame of the multi-view video to finally output a compressed video stream.

2. The immersive video coding method based on 3DGS according to claim 1, characterized in that, Each anchor point corresponds to an aggregated representation of one or more three-dimensional Gaussian distributions; Determining the three-dimensional Gaussian distribution through the anchor points includes: Define the parameters for each anchor point, including position, offset, scale, and independent features; Define the parameters for each of the three-dimensional Gaussian distributions, including position, covariance, transparency, and color; The position of each three-dimensional Gaussian distribution is determined based on the position, offset, and scale of each anchor point; Based on the independent features of each anchor point, the relative distance between the current viewpoint and the anchor point, and the current viewpoint direction, the covariance, transparency map, and color of the three-dimensional Gaussian distribution are obtained.

3. The immersive video coding method based on 3DGS according to claim 1, characterized in that, The step of extracting the spatial context information of the anchor point through multi-resolution hash encoding and concatenating it with the independent features of the anchor point to form a fused feature includes: Construct a multi-resolution spatial hash grid to encode three-dimensional spatial locations; For each anchor point, based on its position, lookup and interpolation are performed at each resolution level of the hash grid to extract the corresponding multi-scale spatial context features; The multi-scale spatial context features are concatenated with the independent features of the anchor point to form a fusion feature for subsequent neural prediction.

4. The immersive video coding method based on 3DGS according to claim 1, characterized in that, The process of quantizing and entropy encoding the parameters of the optimized anchor points includes: Based on the spatial context information provided by the multi-resolution hash grid, the optimal quantization step size is determined for the anchor point attribute parameters of different spatial regions. Based on the quantization step size, the features, offset, scale, and position parameters of the anchor point are scalar or vector quantized respectively, and the continuous parameter values ​​in the anchor point parameters are mapped to discrete quantization indices. The quantized anchor point position parameters are compressed and encoded using the GPCC standard; Output the entropy-encoded bitstream.

5. The immersive video coding method based on 3DGS according to claim 1, characterized in that, The step of combining the color loss and entropy coding rate to perform rate-distortion optimization on the quantized parameters and generate a compressed 3D scene representation includes: Calculate the color fidelity loss between the reconstructed rendered image and the original input multi-view video frames; Estimate the bitrate loss corresponding to the anchor parameters after quantization and entropy coding, the bitrate loss being based on the actual number of bits generated by coding or obtained through a pre-trained bitrate estimation network; The fidelity loss and the bitrate loss are weighted and summed according to a set weight to form a joint optimization objective function; The joint optimization objective function is minimized using the gradient descent algorithm, and the parameters are dynamically adjusted to ensure reconstruction quality while reducing the final bitrate.

6. The immersive video coding method based on 3DGS according to claim 5, characterized in that, The step of weighting and summing the fidelity loss and the bitrate loss according to a set weight to form a joint optimization objective function includes: in: For rendering loss, To control the coefficients for the bit rate and quality tradeoff, For entropy loss, The table normalizes the number of anchor points and dimension for the entropy loss, where N is the total number of anchor points; D a Let K be the feature dimension of the anchor point, and K be the number of Gaussian functions associated with each anchor point. For the quantization loss of binary hash grid, To adjust the pruning intensity at the anchor points, This is the anchor mask loss.

7. The immersive video coding method based on 3DGS according to claim 1, characterized in that, During the initial training rounds, all anchor points are classified, and the classification results are used after the anchor points are quantized. The classification of all anchor points refers to dividing them into three categories—root anchor points, leaf anchor points, and free anchor points—based on feature relevance. Specifically: Based on the spatial distribution and feature similarity of anchor points, a selection algorithm is used to divide two anchor points with similar features into root anchor points and leaf anchor point pairs. Anchor points that fail to be assigned to any pair by the selection algorithm are classified as free anchor points; The root anchor point is used to provide contextual space information for its associated leaf anchor points during entropy encoding. The features of the leaf anchor points are predicted by the features of their respective root anchor points, while the free anchor points are encoded independently.

8. An immersive video coding system based on 3DGS, characterized in that, include: Initialization module: Based on the input multi-view video, sparse reconstruction is performed on each frame to obtain the initial point cloud of the scene; 3D Gaussian Anchor Point Construction Module: Based on the spatial location of the initial point cloud, determine the anchor points of the 3D Gaussian distribution in each frame; Feature fusion module: Extracts spatial context information of anchor points through multi-resolution hash encoding and concatenates it with the independent features of anchor points to form fused features; Gaussian point generation and rendering module: Utilizing fused features and camera parameters, the parameters of the 3D Gaussian distribution corresponding to each anchor point are predicted through a neural network, and then the reconstructed rendered image is obtained through differentiable rendering. Optimization module: Calculates the color loss between corresponding frames of the rendered image and the original multi-view video, and optimizes the parameters of the anchor points; Quantization and Encoding Module: Performs quantization and entropy encoding on the parameters of the optimized anchor points; Rate-distortion optimization module: Combines color loss and entropy coding rate to perform rate-distortion optimization on the quantized parameters and generate a compressed 3D scene representation; The compressed bitstream generation module repeats the above steps for each frame of the multi-view video, and finally outputs a compressed video bitstream.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-7.

10. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-7.

Images