An end-to-end optical field image coding method

Abstract

This invention discloses an end-to-end optical field image coding method, comprising: utilizing multi-domain features of the optical field and their correlations to mine large disparity angle information in the EPI domain to obtain a complete disparity range angle feature representation, and modeling the correlation between features to remove redundant information between features; using the extracted angle domain features and EPI domain features as inputs to mine large disparity range angle information in the EPI domain to obtain a complete disparity range angle feature representation; utilizing the intrinsic correlation between spatial and complete disparity range angle features to remove redundant information between features; using a rate-distortion loss function as the loss function, and employing a variable bitrate training method to obtain a variable bitrate model that can be tested for different λ values. This invention fully mines large disparity angle information in the EPI domain to obtain a complete disparity range angle feature representation; and removes redundant information between features by modeling the intrinsic correlation between space and angle.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and image coding, and in particular to an end-to-end optical field image coding method. Background Technology

[0002] With the rapid development of light field imaging technology, light field images have been widely used in augmented reality, free viewpoint rendering, 3D reconstruction, and other fields. However, the dense viewpoints and high resolution of light field images pose significant challenges to storage and transmission. Therefore, researchers are dedicated to studying light field image coding methods to improve the storage and transmission efficiency of light field images.

[0003] Traditional light field image coding methods can be mainly divided into transform-based and pseudo-video sequence-based methods. Transform-based methods rely on the inherent correlation of light field images and use different transform coding techniques to decorrelate the light field images to eliminate redundant information. These methods have achieved certain results in reducing spatial redundancy, but as the angular resolution of the light field increases, the coding efficiency is difficult to improve further. To address this issue, some researchers have proposed light field image coding methods based on pseudo-video sequences. This method arranges light field sub-aperture images into a pseudo-video sequence and uses traditional video coding techniques to compress the pseudo-video sequence, effectively improving coding efficiency. However, pseudo-video sequence-based methods usually require cumbersome preprocessing steps, such as sub-aperture image sorting and pseudo-video sequence generation, which increases the complexity of the entire coding process.

[0004] Thanks to the powerful feature extraction and nonlinear representation capabilities of deep learning, end-to-end light field image coding methods based on deep learning achieve comparable coding performance to traditional methods without complex preprocessing steps. Tong et al. proposed a deep learning-based end-to-end light field image coding method that utilizes dilated and expanded convolutions to decouple spatial and angular information in light field images, removing redundancy and thus improving coding efficiency. Zhong et al. proposed a data structure-adaptive 3D convolutional encoder that upsamples and downsamples light field images along angular or spatial resolutions, avoiding artifacts caused by direct encoding and achieving better coding efficiency. Existing methods typically utilize spatial or angular features within a limited parallax range to compress light field images, failing to fully exploit large parallax angular information in the epipolar (EPI) domain, thus hindering effective encoding of large parallax light field images. Furthermore, existing methods lack exploration and utilization of correlations between multi-domain features, making it difficult to effectively remove redundant information. Therefore, the performance of end-to-end light field image coding needs further improvement. Summary of the Invention

[0005] Existing light field image coding methods typically only utilize spatial or finite disparity range angular features, failing to fully exploit the large disparity angular information unique to the EPI domain of light field images. Furthermore, existing methods lack effective utilization of the intrinsic correlations between multi-domain features, making it difficult to effectively remove redundancy between features. This invention proposes an end-to-end light field image coding method that fully exploits the large disparity angular information in the EPI domain to obtain a complete disparity range angular feature representation. Simultaneously, by modeling the intrinsic correlations between spatial and complete disparity range angular features, redundancy between features is effectively removed, thereby improving the coding performance of light field images. Details are described below:

[0006] An end-to-end light field image coding method, the method comprising:

[0007] By utilizing the multi-domain features of the light field and their correlations, we can mine large disparity angle information in the EPI domain to obtain a complete disparity range of angle feature representation, and model the correlation between features to remove redundant information between features.

[0008] Using the extracted angular domain features and EPI domain features as input, we mine angular information with a large disparity range in the EPI domain to obtain a complete angular feature representation of the disparity range.

[0009] By leveraging the inherent correlation between spatial and complete parallax range angular features, redundant information between features is removed;

[0010] Using the rate-distortion loss function as the loss function and employing a variable bitrate training method, a variable bitrate model that can be tested for different λ values ​​is obtained.

[0011] The input of the end-to-end light field image coding network is the original light field image, and the output is the compressed reconstructed light field image. At the coding end, the current coded image is first mapped to the feature space using the feature extraction module to obtain the light field image spatial features, the angular features with a limited parallax range, and the EPI features.

[0012] The method uses a spatial angle joint transformation module to remove redundant information between spatial and complete parallax range angle features; a feature fusion module composed of convolutional and activation layers is used to obtain a latent feature representation, and entropy encoding and decoding are performed on the latent representation; the reconstructed latent representation is processed by a decoding end to obtain a compressed and reconstructed light field image, wherein the decoding end consists of a spatial angle joint inverse transformation module and a reconstruction module.

[0013] The angular features for obtaining the complete disparity range are represented as follows:

[0014] The input local angular features f A and EPI domain features f E Feature f is generated using two residual blocks. A 'and fE Then, a mutual attention mechanism is applied to model the similarity relationship between features, and after channel cascading, a nonlinear transformation is performed to finally obtain the common feature f. EA The calculation formula is as follows:

[0015]

[0016] f A '=R(f A ),f E '=R(f E )

[0017] Among them, F L (·) denotes the mutual attention function, h L (·) consists of two stacked convolutional layers and one activation layer. Representing the cascade of channel dimensions, R(·) consists of two layers of residual blocks; according to f EA From f E 'Mining complete features with large parallax angle information in the middle, and combining the complete features with f A Through cascaded fusion, a complete angular feature representation of the parallax range is obtained. The specific calculation process is as follows:

[0018]

[0019] Among them, g L (·) consists of one convolutional layer and one activation layer.

[0020] The method of utilizing the inherent correlation between spatial and complete parallax range angular features to remove redundant information between features is as follows:

[0021]

[0022]

[0023] Among them, R s (·) and R a (·) consists of two convolutional layers and one activation layer, F c (·) consists of one convolutional layer and one activation layer;

[0024]

[0025]

[0026] Among them, the input space features f S Angular features with complete parallax range After TMS-S and TMS-A, the spatial features with compact information are obtained respectively. and angular feature representation

[0027] The beneficial effects of the technical solution provided by this invention are:

[0028] 1. This invention fully utilizes the multi-domain features of the light field and their correlations to fully mine the large parallax angle information in the EPI domain to obtain a complete parallax range of angle feature representation;

[0029] 2. This invention removes redundant information between features by modeling the intrinsic correlation between space and angle. Attached Figure Description

[0030] Figure 1 A flowchart of an end-to-end optical field image coding method;

[0031] Figure 2 This is a schematic diagram of the spatial angle joint transformation module. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0033] I. Constructing an end-to-end light field image coding network framework

[0034] To improve the compression efficiency of light field images, this invention designs an end-to-end light field image coding network. This network utilizes multi-domain features and their correlations to obtain complete angular feature representations and reduce redundancy between features. Specifically, by exploring the large disparity angular information contained in the EPI domain, a complete disparity range angular feature representation is obtained, thereby improving the coding performance of large disparity light field images. Furthermore, the inherent correlation between spatial and angular features is further utilized to remove redundant information between features, improving the coding efficiency of the end-to-end network.

[0035] The end-to-end light field image coding network takes the original light field image as input and outputs a compressed reconstructed light field image. At the encoding end, a feature extraction module first maps the current encoded image to a feature space, obtaining spatial features of the light field image, angular features within a finite disparity range, and EPI (epidial polar image) features. Then, an EPI-based angle completion module obtains an angular feature representation with a complete disparity range. Next, a spatial-angle joint transformation module effectively removes redundant information between the spatial and complete disparity range angular features. Afterward, a feature fusion module consisting of convolutional and activation layers obtains a latent feature representation, which is then entropy encoded and decoded. Finally, the reconstructed latent representation is decoded at the decoding end to obtain a compressed reconstructed light field image, where the decoding end consists of a spatial-angle joint inverse transformation module and a reconstruction module.

[0036] II. Design of an EPI-based perspective completion module

[0037] Existing angle domain features only contain angle information within a limited parallax range. To fully extract angle information from light field images and thus improve the encoding performance of large parallax light field images, this invention proposes an angle completion module based on EPI. This module takes the extracted existing angle domain features and EPI domain features as input, and obtains a complete angle feature representation of the parallax range by mining angle information within the large parallax range in the EPI domain.

[0038] First, the input local angular features f A and EPI domain features f E Feature f is generated using two residual blocks. A 'and f E Then, a mutual attention mechanism is applied to model the similarity relationship between features, and after channel cascading, a nonlinear transformation is performed to finally obtain the common feature f. EA The calculation formula is expressed as follows:

[0039]

[0040] f A '=R(f A ),f E '=R(f E )

[0041] Among them, F L (·) denotes the mutual attention function. h L (·) consists of two stacked convolutional layers and one activation layer. Representing the cascade of channel dimensions, R(·) consists of two layers of residual blocks.

[0042] Finally, according to f EA From f E 'Mining complete features with large parallax angle information in the middle, and combining the complete features with f A Through cascaded fusion, a complete angular feature representation of the parallax range is obtained. The specific calculation process is as follows:

[0043]

[0044] Among them, g L (·) consists of one convolutional layer and one activation layer.

[0045] III. Design of a spatial angle joint transformation module

[0046] This invention presents a spatial angle joint transformation module. This module effectively removes redundant information between features by making full use of the inherent correlation between spatial and complete parallax range angle features, thereby improving the coding efficiency of the end-to-end light field image coding network.

[0047] The spatial angle joint transformation module designed in this embodiment of the invention consists of multiple transformation sub-modules (TMS), based on spatial features f S Angular features with complete parallax range As input, the spatial features designed to output information are compact. and angular features The TMS is divided into a spatial transformation submodule (TMS-S) and an angle transformation submodule (TMS-A). Two TMS-S modules and two TMS-A modules are connected alternately to remove redundant information from spatial features and angle features with complete parallax range, respectively.

[0048] Taking TMS-A as an example, it contains two parallel paths. First, the input spatial features... and angular features After processing with residual blocks and convolutional layers, spatial features of the same size are generated. and angular features Subsequently, these two feature paths are concatenated to model the intrinsic correlation between spatial and angular features, resulting in redundant features. Finally, the features Subtract redundant features This yields a compact set of angular features. The above process can be described as follows:

[0049]

[0050]

[0051] Among them, R s (·) and R a (·) consists of two convolutional layers and one activation layer, F c (·) consists of one convolutional layer and one activation layer.

[0052] The structure of TMS-S is similar to that of TMS-A, aiming to output compact spatial features. Taking TMS-S connected after TMS-A as an example, its formula is expressed as follows:

[0053]

[0054]

[0055] Among them, the input space features f SAngular features with complete parallax range After TMS-S and TMS-A, the spatial features with compact information are obtained respectively. and angular feature representation The designed spatial angle joint transformation module eliminates redundancy between features by modeling the intrinsic correlation between space and angle, which can effectively improve the compression performance of the end-to-end light field image compression network.

[0056] IV. Training an end-to-end light field image compression network

[0057] This invention uses the rate-distortion loss function L = R + λD as the loss function, where D(·) represents the mean squared error, R represents the bitrate, and λ represents the hyperparameter for adjusting the bitrate. The training phase employs a variable bitrate training method, consisting of three stages. The first stage trains the neural network until convergence to obtain a single bitrate model; the second stage fine-tunes the single bitrate model to a variable bitrate model capable of multi-bitrate testing; the third stage further improves coding performance by changing the quantization method of the entropy model, ultimately obtaining a variable bitrate model that can be tested for different λ values.

[0058] During testing, the trained variable bitrate model is loaded, and the light field images in the test set are used as input to obtain the bitrate and reconstruction results required for compressed light field images under different λ values.

[0059] For example, in this embodiment of the invention, the rate-distortion loss function L = R + λD is used as the loss function, where D(·) represents the mean squared error, R represents the bit rate, and λ represents the hyperparameter for adjusting the bit rate, with values ​​set to {0.0003, 0.0006, 0.001, 0.002, 0.004}. The training phase employs a variable bit rate training method, divided into three stages. The first stage trains the neural network for 30 epochs to obtain a single bit rate model with λ = 0.002; the second stage fine-tunes the single bit rate model with λ = 0.002 to a variable bit rate model capable of multi-bit rate testing; the third stage further improves coding performance by changing the quantization method of the entropy model, ultimately obtaining a variable bit rate model capable of testing five different λ values.

[0060] During testing, the trained variable bitrate model is loaded, and the light field images in the test set are used as input to obtain the bitrate and reconstruction results required for the compressed light field images under 5 different λ values.

[0061] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0062] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An end-to-end optical field image coding method, characterized in that, The method includes: The input is the original light field image, and the output is the compressed reconstructed light field image. At the encoding end, the current encoded image is first mapped to the feature space using the feature extraction module to obtain the light field image spatial features, angular features with a limited parallax range, and EPI features. Using the extracted angular features and EPI features as input, we mine angular information with a large disparity range in the EPI domain to obtain a complete angular feature representation of the disparity range. By leveraging the inherent correlation between spatial and complete parallax range angular features, redundant information between features is removed; Using the rate-distortion loss function as the loss function and employing a variable bitrate training method, we obtain results that can be tested for different... Variable bit rate model; Load the trained variable bitrate model, take the light field images from the test set as input, and obtain different... The bitrate and reconstruction results required for compressed light field images; The method of utilizing the inherent correlation between spatial and complete parallax range angular features to remove redundant information between features is as follows: ； ； in, The characteristic of redundancy and It consists of two convolutional layers and one activation layer. For spatial features, As an angular feature, This refers to the compact angle features obtained after removing redundant angle features from the angle features. Indicates based on spatial characteristics and the compact angle features of information The obtained spatial redundancy features This refers to the compact spatial features obtained after removing the redundant spatial features from the spatial features. It consists of one convolutional layer and one activation layer; ； ； Among them, input space features Angular features with complete parallax range Spatial features Angular features with complete parallax range Residual blocks and convolutional layers are applied to generate spatial features of the same size. and angular features After passing through the angle transformation submodule TMS-A and the spatial transformation submodule TMS-S, compact angular and spatial feature representations are obtained, respectively.

2. The end-to-end optical field image coding method according to claim 1, characterized in that, The method uses a spatial angle joint transformation module to remove redundant information between spatial and complete parallax range angle features; a feature fusion module composed of convolutional and activation layers is used to obtain a latent feature representation, and entropy encoding and decoding are performed on the latent representation; the reconstructed latent representation is processed by the decoding end to obtain a compressed and reconstructed light field image, wherein the decoding end consists of a spatial angle joint inverse transformation module and a reconstruction module.

3. The end-to-end optical field image coding method according to claim 1, characterized in that, The angular features for obtaining the complete disparity range are represented as follows: Input local angular features and EPI domain features Features are generated using two residual blocks. and Then, a mutual attention mechanism is applied to model the similarity relationship between features, and after channel cascading, a nonlinear transformation is performed to finally obtain the shared features. The calculation formula is as follows: ； , ； in, Represents the mutual attention function. It consists of two convolutional layers and one activation layer stacked together. Cascading representing the channel dimension, It consists of two layers of residual blocks; according to from Mining completion features with large parallax angle information, and combining completion features with... By cascading fusion, a complete angular feature representation of the parallax range is obtained. The specific calculation process is as follows: ； in, It consists of one convolutional layer and one activation layer.

Images