Volume video intelligent coding method

Abstract

The invention discloses a volume video intelligent coding method, and the method comprises the steps: a feature enhancement module based on view guidance is used for carrying out the feature enhancement of self-adaptive learning anchor features; the progressive entropy coding module based on context guidance is used for performing efficient compression coding on the enhanced anchor point features; the Gaussian attribute feature decoder is used for obtaining each neural Gaussian attribute; and through a neural Gaussian sputtering rendering technology, the neural Gaussian is mapped to a target visual angle, and a high-quality reconstructed viewpoint image is generated. According to the method, the scene information of each view angle is fused into the feature learning process, and the Gaussian attribute progressive coding strategy is constructed, so that the compression efficiency of the volume video is improved.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and three-dimensional volumetric video coding, and particularly to an intelligent coding method for volumetric video. Background Art

[0002] Compared with traditional 2D videos, volumetric videos can record and present dynamic scenes in all directions, providing users with an immersive experience of freely viewing from any perspective, and having broad application prospects in fields such as virtual reality and augmented reality. However, volumetric videos have a complex three-dimensional data structure, and traditional video coding technologies are difficult to meet the requirements of their efficient compression and transmission. Therefore, it is urgent to study intelligent coding technologies for volumetric videos.

[0003] 2D video coding methods can effectively eliminate the temporal and spatial redundancies of 2D videos through inter-frame prediction and transform coding technologies, achieving good compression performance. However, different from 2D videos, volumetric videos contain multi-domain information in the temporal, spatial, and view domains at the same time. 2D video coding methods have not fully considered the complex coupling relationships among the temporal, spatial, and view domains of volumetric videos, and it is difficult to achieve efficient compression of volumetric videos. In recent years, the three-dimensional Gaussian splatting technology (3D Gaussian Splatting, 3DGS) has shown significant advantages in terms of training speed, rendering efficiency, and synthesis quality. Its explicit representation framework and efficient rendering mechanism provide a new paradigm for three-dimensional scene reconstruction. Based on this breakthrough, researchers have begun to explore methods for volumetric video compression based on 3DGS, that is, to achieve efficient compression of volumetric videos by representing the three-dimensional scene with 3DGS and compressing its attributes.

[0004] However, existing methods usually rely on adaptive learning mechanisms to extract three-dimensional scene features and do not fully utilize the scene information captured from each perspective. In addition, existing methods do not fully utilize the context associations among Gaussian attributes, resulting in ineffective removal of redundant information. Summary of the Invention

[0005] The present invention provides an intelligent coding method for volumetric video. The present invention integrates the scene information of each perspective into the feature learning process and constructs a progressive coding strategy for Gaussian attributes to improve the compression efficiency of volumetric video, as described in detail below:

[0006] An intelligent coding method for volumetric video, the method comprising:

[0007] A view-guided feature enhancement module for enhancing the features of adaptively learned anchor features;

[0008] A context-guided progressive entropy coding module for efficiently compressing and coding the enhanced anchor features;

[0009] A Gaussian attribute feature decoder for obtaining each neural Gaussian attribute;

[0010] Through the neural Gaussian sputtering rendering technique, map the neural Gaussian to the target view and generate a high-quality reconstructed view image.

[0011] Among them, the view-guided feature enhancement module is:

[0012] Use the pre-trained VGG network to extract features from the randomly selected view P to obtain the initial view feature f containing scene information i ;

[0013] Use the UNet network to perform multi-scale feature enhancement on the initial view feature f i to generate the enhanced view feature f e ;

[0014] By performing convolution, pooling, and activation on the enhanced view feature f e , finally transform the enhanced view feature f e into the view-guided feature f c aligned with the three-dimensional Gaussian representation, providing high-semantic scene information for the generation of neural Gaussian attributes.

[0015] Among them, the context-guided progressive entropy encoding module is:

[0016] Construct the context information of the anchor attributes through a binary hash table and perform interpolation calculation based on the anchor coordinates x to obtain the context feature f t ;

[0017] Use the prior feature extraction network to extract the hyper-prior information from the anchor feature f b to generate the hyper-prior feature h f ;

[0018] Concatenate the context feature f t with the hyper-prior feature h f and predict the probability distribution p(f b ) of the anchor feature f b through a multi-layer perceptron MLP.

[0019] Among them, the encoded anchor feature f b is used as prior information to predict the probability distributions o(S) and p(O) of the size Scaling and the coordinate displacement offsets.

[0020] Among them, the Gaussian attribute feature decoder is:

[0021] Combine the obtained view-guided feature f c with the adaptively learned anchor feature f aPerform channel concatenation and send the concatenated features into the MLP network to map and obtain enhanced anchor features f b ;

[0022] Send the obtained enhanced anchor features f b After concatenating with the distance from the anchor to the Gaussian center and the direction vector, send them into different Gaussian attribute prediction networks to decode and obtain the corresponding attributes {col, cov, o} of the neural Gaussian.

[0023] The beneficial effects of the technical solution provided by the present invention are as follows:

[0024] 1. The present invention efficiently encodes Gaussian attributes in an end-to-end manner, effectively improving the compression efficiency of volumetric videos;

[0025] 2. The present invention designs a view-guided feature enhancement module. By deeply fusing two-dimensional view features with three-dimensional Gaussian representations, the scene information of each perspective is embedded into the anchor feature representation, enabling the compression model to obtain richer scene reconstruction clues under the same bitrate constraint, thereby effectively improving the reconstruction quality of volumetric videos;

[0026] 3. The present invention proposes a context-guided progressive entropy coding module. By utilizing the structural similarity relationship between attributes, generating hyperprior features based on the already compressed attributes, and using these to guide the coding process of subsequent attributes, the redundant information between Gaussian attributes is effectively eliminated, thereby improving the compression efficiency of volumetric videos. Description of the Drawings

[0027] Figure 1 It is a flowchart of an intelligent coding method for volumetric videos. Detailed Embodiments

[0028] To make the objectives, technical solutions, and advantages of the present invention clearer, the following further describes the embodiments of the present invention in detail.

[0029] Existing volumetric video coding methods based on 3DGS usually focus on designing efficient entropy models for Gaussian attributes to improve the accuracy of probability modeling, lacking research and exploration on feature enhancement techniques. At the same time, existing methods fail to fully exploit the context associations between Gaussian attributes, resulting in difficulty in effectively removing the structural redundancy between Gaussian attributes.

[0030] To improve the encoding performance of volumetric videos, an embodiment of the present invention proposes a view-guided feature enhancement module. By deeply fusing two-dimensional view features with three-dimensional Gaussian representations, the scene information of each perspective is embedded into the anchor feature representation, enabling the compression model to obtain richer scene reconstruction clues under the same bitrate constraint. In addition, a context-guided progressive entropy coding module is proposed. This module utilizes the structural similarity relationship between attributes to generate hyperprior features based on the compressed attributes and guides the encoding process of subsequent attributes, thereby effectively eliminating redundant information between Gaussian attributes and further enhancing the compression performance.

[0031] The following uses an example to illustrate the specific implementation of an intelligent encoding method for volumetric videos in an embodiment of the present invention.

[0032] I. Constructing an intelligent encoding network for volumetric videos

[0033] The intelligent encoding network for volumetric videos takes a multi-viewpoint image set as input. First, it enhances the features of the adaptively learned anchor features through a view-guided feature enhancement module. Then, it uses a context-guided progressive entropy coding module to perform efficient compression encoding on the enhanced anchor features, significantly reducing the data volume while ensuring the reconstruction quality. Subsequently, it uses a Gaussian attribute feature decoder to obtain each neural Gaussian attribute. Finally, through the neural Gaussian sputtering rendering technique, the neural Gaussian is mapped to the target perspective to generate high-quality reconstructed viewpoint images.

[0034] II. Constructing a view-guided feature enhancement module

[0035] For the view-guided feature enhancement module, first, it uses a pre-trained VGG network to extract features from a randomly selected view P to obtain the initial view feature f containing scene information. i . Subsequently, it uses a UNet network to perform multi-scale feature enhancement on the initial view feature f i to generate an enhanced view feature f with rich detail information e . To further adapt the two-dimensional view features to the three-dimensional Gaussian attribute space, an embodiment of the present invention processes the enhanced view feature f e through convolution, pooling, and activation, etc., and finally converts the enhanced view feature f e into a view-guided feature f aligned with the three-dimensional Gaussian representation c to provide high-semantic scene information for subsequent neural Gaussian attribute generation. The specific formula is as follows:

[0036] f i = VGG(P), f e = UNet(f i ),

[0037] fc = Mapping(f e ).

[0038] III. Constructing a Context-Guided Progressive Entropy Encoding Module

[0039] To achieve accurate probability estimation and minimize the code rate consumption, an embodiment of the present invention designs a context-guided progressive entropy encoding module. The core implementation of this module is as follows:

[0040] First, construct the context information of the anchor attributes through a binary hash table, and perform interpolation calculation based on the anchor coordinate x to obtain the context feature f t . Subsequently, use the prior feature extraction network to extract the hyper-prior information from the anchor feature f b to generate the hyper-prior feature h f .

[0041] After that, cascade the context feature f t with the hyper-prior feature h f , and predict the probability distribution p(f b ) of the anchor feature f b through a multi-layer perceptron MLP:

[0042] f t = Hash(x), h f = Hyper(f b ),

[0043] {μ f , σ f} = MLP f (Concat(f t , h f ))

[0044]

[0045] f where Hash(·) represents hash feature extraction. Hyper(·) represents hyper-prior feature extraction. MLP f (·) represents the probability distribution prediction network, which is used to obtain the mean μ f and the variance σ . represents the normal distribution

[0046] To effectively utilize the context correlation characteristics between anchor attributes, an embodiment of the present invention uses the encoded anchor feature f b as the prior information to predict the probability distributions p(S) and p(O) of the size Scaling and the coordinate displacement offsets, and the specific formula is as follows:

[0047] {μs , σ s , μ o , σ o} = MLP s (f b ),

[0048]

[0049] Finally, according to the probability distribution of the estimated anchor attributes, calculate the code rate consumption required for encoding. The specific formula is as follows:

[0050] R f = E(-logp(f b ) - logp(h f )),

[0051] R s = E(-logp(S)), R o = E(-logp(O)).

[0052] IV. Constructing a Gaussian Attribute Feature Decoder

[0053] To effectively utilize the high-semantic scene information in the view features, first concatenate the obtained view-guided feature f c with the adaptively learned anchor feature f a at the channel level, and send the concatenated feature into the MLP network to map and obtain the enhanced anchor feature f b . Subsequently, to achieve accurate prediction of Gaussian attributes, concatenate the obtained enhanced anchor feature f b with the distance from the anchor to the Gaussian center and the direction vector, and then send them into different Gaussian attribute prediction networks to decode the corresponding attributes {col, cov, o} of the neural Gaussian:

[0054] f b = MLP a (Concat(f a , f c )),

[0055] {col, cov, o} = MLP n (f b ).

[0056] Among them, Concat(·) represents the channel concatenation operation, and {col, cov, o} represent the color, covariance, and opacity of the Gaussian respectively. MLP a (·) and MLP n (·) represent the networks for feature fusion and Gaussian attribute prediction respectively.

[0057] V. Training the Volume Video Intelligent Coding Network

[0058] In the embodiment of the present invention, the rate-distortion loss function L is used to perform end-to-end joint optimization on the proposed intelligent volume video coding network. This loss function consists of two parts: a distortion term and a bitrate consumption term. The distortion term includes the SSIM loss L SSIM between the reconstructed image and the original image and the reconstruction loss L reg based on the L1 norm; the bitrate consumption term is composed of the sum of the bitrate consumptions of the anchor features, dimensions, and coordinate offsets. The specific calculation formula of the rate-distortion loss function L is as follows:

[0059] L = λ(R f + R s + R o ) + L SSIM + L reg ,

[0060] where λ is a hyperparameter that controls the bitrate consumption and the reconstruction loss.

[0061] After the training is completed, only the network parameters of the anchor attribute and the Gaussian attribute feature decoder need to be saved. Then, using the neural Gaussian sputtering rendering technology, the neural Gaussian can be mapped to the target view to generate a high-quality reconstructed view image, thereby realizing the efficient compression coding of the volume video.

[0062] In the embodiment of the present invention, except for the special descriptions of the models of each device, the models of other devices are not limited, and any device that can perform the above functions can be used.

[0063] Those skilled in the art can understand that the drawings are only schematic diagrams of a preferred embodiment. The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0064] The above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An intelligent encoding method for volumetric video, characterized in that, The method includes: A view-guided feature enhancement module for enhancing the features of adaptively learned anchor points; A context-guided progressive entropy coding module for efficiently compressing and coding the enhanced anchor point features; A Gaussian attribute feature decoder for obtaining each neural Gaussian attribute; Through the neural Gaussian sputtering rendering technique, map the neural Gaussian to the target view to generate a high-quality reconstructed viewpoint image.

2. The volume video intelligent encoding method according to claim 1, characterized in that, The feature enhancement module is: Extract features from the randomly selected view P using the pre-trained VGG network to obtain the initial view feature f containing scene information i ; Use the UNet network to perform multi-scale feature enhancement on the initial view feature f i to generate the enhanced view feature f e ; By performing convolution, pooling, and activation on the enhanced view feature f e and finally transforming the enhanced view feature f e into a view-guided feature f aligned with the three-dimensional Gaussian representation c to provide high-semantic scene information for neural Gaussian attribute generation.

3. The intelligent volume video encoding method according to claim 1, wherein The progressive entropy coding module is: Construct the context information of the anchor attributes through a binary hash table, and perform interpolation calculations based on the anchor coordinate x to obtain the context feature f t ; Extract the hyper-prior information from the anchor feature f using the prior feature extraction network to generate the hyper-prior feature h b f ;​ The context feature f t is cascaded with the hyperprior feature h f to perform feature concatenation, and the probability distribution p(f b ) of the anchor feature f is predicted through a multi-layer perceptron MLP b .

4. A method for intelligent encoding of volumetric video according to claim 3, wherein, Use the encoded anchor feature f b as prior information to predict the probability distributions p(S) and p(O) of the size Scaling and the coordinate displacement offsets 5. A method for intelligent encoding of volumetric video according to claim 1, wherein, The Gaussian attribute feature decoder is: The obtained view guiding feature f c is cascaded with the anchor feature f of adaptive learning a at the channel level, and the cascaded feature is fed into the MLP network to map and obtain the enhanced anchor feature f b ; The obtained enhanced anchor feature f b After concatenating with the distance from the anchor to the Gaussian center and the direction vector, it is fed into different Gaussian attribute prediction networks, and the corresponding attributes {col, cov, o} of the neural Gaussian are decoded.

Images