A scalable light field image encoding method

Abstract

The application discloses a kind of hierarchical light field image encoding methods, comprising: enhancement layer coding network utilizes the enhancement layer coding strategy based on domain-specific reference to carry out interlayer prediction and residual compression to spatial feature, and angle feature is generated at decoding end, and the light field image is encoded;Interlayer spatial prediction module is constructed, the co-location and local self-similarity within reference spatial feature are modeled simultaneously, to implicitly reconstruct high-frequency details in high-resolution light field representation, predict spatial latent representation;Spatial-angle context-aware reconstruction module is constructed, long-distance and short-distance spatial-angle correlation is explored in long-range and short-range context-aware branch respectively to model light field image geometry and preserve local details, and the light field image of enhancement layer is reconstructed.The application encodes light field image of different spatial resolution in an end-to-end manner, and improves the encoding performance of spatial hierarchical light field image.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and image coding, and in particular to a scalable light field image coding method. Background Technology

[0002] Light field images simultaneously record spatial and angular information of 3D scenes and are widely used in depth estimation, 3D reconstruction, and virtual reality. However, light field images have a massive data volume, posing significant challenges to data storage and transmission. To effectively improve the coding efficiency of light field images, the international image / video coding standards organizations JPEG and MPEG have carried out a series of standardization efforts for light field image coding in the new standard sets JPEG Pleno and MPEG-I. However, to adapt to different application scenarios in actual transmission, such as network heterogeneity, user needs, and the diversity of terminal devices, light field image coding methods need to have scalable functions, such as scalable quality, scalable angle, and scalable spatial resolution. In particular, spatial scalability can adaptively decode a single encoded bitstream into a suitable subset of the bitstream, thereby providing different spatial resolution versions of the same light field image. Therefore, research on spatially scalable light field image coding is of great significance.

[0003] With the increasing application of light field images in various fields, researchers have begun to focus on spatially scalable light field image coding methods to improve the coding efficiency of light field images with different spatial resolutions. Typically, these methods utilize a base layer and multiple enhancement layers to perform hierarchical coding of light field images with different spatial resolutions to adapt to different application scenarios in actual transmission. The base layer aims to encode the lowest resolution version of the original light field image and provide reference information for adjacent enhancement layers to assist in their coding. The enhancement layers generate prediction images through inter-layer prediction techniques, and then only need to encode the residual between the light field image to be coded and the prediction light field image, thus achieving a reduction in bit rate. For example, Garrote et al. proposed a spatially scalable light field image coding method based on discrete wavelet transform, which uses the JPEG2000 standard to encode the discrete wavelet transform coefficients. Li et al. used a parallax-based interpolation and inpainting algorithm to generate a reference image and utilized the improved High Efficiency Video Coding (HEVC) standard to encode the prediction residual of the enhancement layer. Amirpour et al. proposed a spatially scalable light field image coding method based on pseudo-video sequences. This method is designed based on the Versatile Video Coding (VVC) standard and generates reference images for predictive coding through learning-based image super-resolution techniques. However, these methods based on traditional coding standards fail to jointly optimize all coding modules end-to-end, which limits further improvements in the performance of spatially scalable light field image coding. Summary of the Invention

[0004] This invention provides a scalable light field image coding method. This invention further improves the coding efficiency of spatially scalable light field images through an end-to-end coding approach, as detailed below:

[0005] A scalable light field image coding method, the method comprising:

[0006] The base layer coding network is responsible for encoding the light field image with the lowest spatial resolution, and uses a spatial-angular feature decoupling module to extract spatial and angular features from the reconstructed spatial-angular features to generate reference features for the enhancement layer;

[0007] The enhancement layer coding network uses a domain-specific reference-based enhancement layer coding strategy to perform inter-layer prediction and residual compression of spatial features, and generates angular features at the decoding end to encode the light field image;

[0008] A spatial prediction module is constructed between layers, and the inherent colocalization and local self-similarity of reference spatial features are modeled to implicitly reconstruct high-frequency details in high-resolution light field representations and predict latent spatial representations.

[0009] A spatial-angle context-aware reconstruction module is constructed. By exploring long-range and short-range spatial-angle correlations in long-range and short-range context-aware branches respectively, the geometric structure of the light field image is modeled and local details are preserved, and the light field image of the enhancement layer is reconstructed.

[0010] The enhancement layer coding network includes: a spatial feature extractor, an encoder, an inter-layer spatial prediction module, an entropy coding module, a decoder, an angle generation module, and a space-angle context-aware reconstruction module.

[0011] At the encoding end, the spatial feature extractor is used to extract features from the light field image x. i Extracting spatial features s i The encoder is used to extract spatial features s i Nonlinearly transforming into a compact spatial latent representation The interlayer spatial prediction module is used to reconstruct spatial features from adjacent layers. To obtain the predicted spatial latent representation The entropy coding module is used to compress the spatial latent representation of the prediction of the i-th enhancement layer. and the latent representation of the original space The residual r between i .

[0012] At the decoding end, the residual of the i-th enhancement layer is decoded. and predicted spatial latent representation Corresponding addition yields the reconstructed spatial latent representation The decoder is used to reconstruct the spatial latent representation of the i-th enhancement layer. Inverse transformation to reconstructed spatial features The angle generation module is used to reconstruct angle features from the base layer. Generate the angular features of the i-th enhancement layer The spatial-angle context-aware reconstruction module is used to fuse spatial features. and angular features Reconstructing high-quality light field images

[0013] The interlayer spatial prediction module consists of three dual-stream feature aggregation blocks. Each dual-stream feature aggregation block models co-locality and local self-similarity in the co-locality and local self-similarity exploration branches to predict the spatial latent representation.

[0014] Furthermore, the space-angle context-aware reconstruction module captures long-range space-angle context information in the long-range context-aware branch to model the geometry of the light field image; and captures short-range space-angle context information in the short-range context-aware branch to preserve the local details of the light field image.

[0015] The beneficial effects of the technical solution provided by this invention are:

[0016] 1. This invention uses an end-to-end hierarchical encoding method to encode light field images with different spatial resolutions, thereby improving the encoding performance of spatially hierarchical light field images. This invention designs an enhancement layer encoding strategy based on domain-specific references, which uses reference features to predictively encode spatial features to effectively remove inter-layer spatial redundancy, and generates angle features at the decoding end to avoid direct encoding of angle features, thereby improving the encoding performance of the enhancement layer.

[0017] 2. This invention designs an inter-layer spatial prediction module, which obtains accurate spatial prediction results by fully exploring the inherent co-location self-similarity and local self-similarity of reference spatial features, thereby effectively removing inter-layer spatial redundancy;

[0018] 3. This invention designs a spatial-angle context-aware reconstruction module. By fully mining long-distance and short-distance spatial-angle context information, it effectively models the geometric structure of the light field image while preserving local details, thereby reconstructing a high-quality light field image. Attached Figure Description

[0019] Figure 1 This is a flowchart of a scalable light field image encoding method. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0021] To overcome the shortcomings of existing technologies, this invention provides a scalable light field image coding method, the specific steps of which are as follows:

[0022] I. Constructing the Basic Layer Coding Network

[0023] The base layer coding network consists of an end-to-end light field image coding network proposed by Tong et al. and a spatial-angular feature decoupling module. The base layer takes a 4x downsampled version x0 of the original input light field image as input, encodes the extracted spatial-angular features through the light field image coding network proposed by Tong et al., and finally reconstructs the light field image at the decoding end. Furthermore, the base layer utilizes a spatial-angular feature decoupling module to decouple the reconstructed spatial-angular features into spatial and angular features. Then, the reconstructed features from different domains are used as domain-specific reference features to assist the encoding of the enhancement layer.

[0024] II. Constructing an Enhanced Layer Coding Network Based on Domain-Specific References

[0025] Because spatial and angular information are highly coupled in light field images, it is difficult to learn an efficient light field representation in the coupled domain. Therefore, the enhancement layer uses the previous coding layer as a reference layer, employs a domain-specific reference-based enhancement layer coding strategy to perform inter-layer prediction and residual compression of spatial features, and generates angular features at the decoding end, thereby improving the coding performance of the enhancement layer. The enhancement layer network consists of a spatial feature extractor, encoder, inter-layer spatial prediction module, entropy coding module, decoder, angle generation module, and space-angle context-aware reconstruction module.

[0026] The i-th (i = 1, 2) enhancement layer (EL-i) first uses a spatial feature extractor to extract features from the light field image x. i Extracting spatial features s i Then, the encoder is used to extract the spatial features s i Nonlinearly transforming into a compact spatial latent representation Then, the designed inter-layer spatial prediction module uses the spatial features reconstructed from adjacent layers. As input, the output is the predicted spatial latent representation. The formula is expressed as follows:

[0027]

[0028] in, This represents the operation of the designed inter-layer spatial prediction module. Then, the i-th enhancement layer compresses the predicted spatial latent representation through an entropy coding module. and the latent representation of the original space The residual r between i This effectively removes spatial redundancy between adjacent layers.

[0029] At the decoding end, the i-th enhancement layer decodes the residual. and predicted spatial latent representation Corresponding addition yields the reconstructed spatial latent representation Then, the i-th enhancement layer uses the decoder to reconstruct the spatial latent representation. Inverse transformation to reconstructed spatial features Meanwhile, since the angular features of the i-th enhancement layer are not encoded into a binary bitstream, the angular features of the i-th enhancement layer are reconstructed from the base layer by the angle generation module. Generate the angle features of the current layer Ultimately, the designed spatial-angle context-aware reconstruction module effectively integrates spatial features. and angular features Output high-quality reconstructed light field images The formula is expressed as follows:

[0030]

[0031] in, This indicates the operation of the designed spatial-angle context-aware reconstruction module. In this embodiment of the invention, the angle generation module consists of a sub-pixel convolutional layer, a convolutional layer, and a channel attention module.

[0032] III. Constructing an Interlayer Spatial Prediction Module

[0033] Considering the characteristics of the reference space Inherent co-localization self-similarity reflects the correlation between adjacent pixels in the same view space features, while reference space features The inherent local self-similarity reflects the correlation between related pixels in different view space features. Therefore, an inter-layer spatial prediction module was constructed to implicitly reconstruct high-frequency details in the high-resolution light field representation by simultaneously modeling co-locality and local self-similarity in the reference spatial features, thereby predicting the latent spatial representation.

[0034] The constructed inter-layer spatial prediction module consists of three two-stream feature aggregation blocks. Each two-stream feature aggregation block contains a co-locality self-similarity exploration branch and a local self-similarity exploration branch to model co-locality and local self-similarity, respectively. Furthermore, an adaptive feature fusion unit aggregates the spatial context information of the learned features from the two branches. Taking one of the two-stream feature aggregation blocks as an example, the j-th (j = 0, 1, 2) two-stream feature aggregation block uses spatial features... As input, the output is refined spatial features.

[0035] Specifically, in the co-localization self-similarity exploration branch, co-localized pixels are extracted by dilated convolutional layers to obtain contextual information from features in the same view space, thereby obtaining compact features. The formula is expressed as follows:

[0036]

[0037] Among them, R col (·) represents a residual block consisting of two dilated convolutional layers and one ReLU activation layer. Each dilated convolutional layer has a kernel size of 3×3, a stride of 1, and a dilation rate of X∈{8,4,2}. col The (·) indicates a downsampling operation, consisting of a dilated convolutional layer with a stride of 2 and an expansion rate of X, and a generalized divisor normalization (GDN) layer. The local self-similarity exploration branch replaces the dilated convolutional layer with a 3×3 kernel size convolutional layer, mining texture information from different view space features through similar operations to obtain compact features. Finally, to aggregate complementary spatial context information, the adaptive feature fusion unit selectively fuses compact features from the two branches by assigning different spatial weights, predicting a refined spatial feature representation. Therefore, after three consecutive two-stream feature aggregation blocks, the predicted spatial latent representation is output, thereby eliminating spatial redundancy between adjacent layers.

[0038] IV. Constructing a Spatial-Angle Context-Aware Reconstruction Module

[0039] Considering that nonlocal geometric modeling and local detail preservation are helpful for reconstructing high-fidelity light field images, a space-angle context-aware reconstruction module was constructed. This module explores long-range and short-range space-angle correlations in long-range and short-range context-aware branches, respectively, to improve the reconstruction quality of the light field images.

[0040] Specifically, the long-range context-aware branch aims to capture long-range spatial-angular contextual information to model the geometry of light field images. First, spatial features of the input are fused through convolutional layers. and angular features Obtain shallow spatial-angular features Then, for features Downsampling is performed to obtain spatial-angular features Where C represents the channel dimension, H d and W d Let these represent the height dimension and the width dimension, respectively. To fully explore long-distance spatial-angular correlations across different spatial dimensions, the following parameters are used: [Image of horizontal and vertical grating scanning directions]. Expanding into sequences of patches yields directional features. Where D = H d W d / P 2 Let P represent the depth dimension, and P×P represent the block resolution. Then, in order to... This method effectively models long-range space-angle dependencies in different directions and utilizes two weighted multi-scale feature extraction blocks to obtain enhanced features. The formula is expressed as follows:

[0041]

[0042] in, This represents a multi-scale feature extraction block operation. Within each multi-scale feature extraction block, to more effectively explore long-range spatial-angular correlations in the depth dimension, two depthwise separable 3D convolutional layers with different dilation rates process the input features in parallel. Then, the outputs of the two depthwise separable 3D convolutional layers are fused using 3D convolutional layers, and the dimension of the fused features is transformed back to the original C×H. d ×W d Dimensions, as the final output. In obtaining... Then, refined features are generated using fusion and residual join operations. Through the Upsampling to obtain long-range context The formula is expressed as follows:

[0043]

[0044] in, The symbols represent channel cascading, h(·) represents a convolutional layer, and u(·) represents a sub-pixel convolutional layer. In this embodiment of the invention, the block size P is set to 8.

[0045] Meanwhile, the short-range context-aware branch aims to capture short-range spatial-angular contextual information to preserve local details of the light field image. Specifically, it uses two stacked residual blocks to pair features. Perform nonlinear transformations to generate short-range context. After modeling the spatial-angular correlation between long-distance and short-distance distances, and Corresponding addition produces deep spatial-angular features. Finally, through convolutional layers from High-quality light field images of the current enhancement layer network were reconstructed. In this embodiment of the invention, each residual block consists of two convolutional layers and one activation layer.

[0046] V. Constructing an end-to-end spatially hierarchical light field image coding network

[0047] An end-to-end spatially scalable light field image coding network is constructed, which encodes different spatial resolution versions of the input light field image through a base layer network and two enhancement layers network. The base layer first outputs the reconstructed light field image with the lowest spatial resolution. Then, the two enhancement layers output a reconstructed light field image with higher spatial resolution. In this embodiment of the invention, the rate-distortion loss function L is used to train the entire network for 40 epochs, as expressed in the following formula:

[0048]

[0049] Where k represents the layer index. and D represents the code rate of the latent representation of the coding base layer and the residual of the enhancement layer, respectively. k (·) indicates the reconstruction distortion calculated based on the mean square error. λ k As a hyperparameter, it controls the rate-distortion tradeoff of the corresponding layer. To accommodate different bit rates, λ is set. k The four different combinations are {(250, 150, 75), (500, 300, 150), (1000, 600, 300), (2000, 1200, 600)}, where the values ​​in each combination correspond to the base layer, the first enhancement layer, and the second enhancement layer, respectively.

[0050] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0051] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A scalable light field image encoding method characterized by, The method comprises: The base layer encoding network is responsible for encoding the light field image with the lowest spatial resolution, and extracting spatial features and angle features from the reconstructed spatial-angle features using a spatial-angle feature decoupling module to generate reference features of the enhancement layer; The enhancement layer encoding network uses a domain-specific reference-based enhancement layer encoding strategy to perform inter-layer prediction and residual compression on the spatial features, and generates angle features at the decoding end; An inter-layer spatial prediction module is constructed to model the co-location and local self-similarity within the reference spatial features to implicitly reconstruct high-frequency details in the high-resolution light field representation and predict the spatial latent representation; A spatial-angle context-aware reconstruction module is constructed to capture long-distance spatial-angle context information in a long-range context-aware branch to model the geometric structure of the light field image, and to capture short-distance spatial-angle context information in a short-range context-aware branch to preserve the local details of the light field image, and to reconstruct the light field image of the enhancement layer; The enhancement layer encoding network comprises a spatial feature extractor, an encoder, an inter-layer spatial prediction module, an entropy encoding module, a decoder, an angle generation module, and a spatial-angle context-aware reconstruction module; At the encoding end, The spatial feature extractor is configured to extract spatial features from a light field image ;​ The encoder is configured to extract spatial features non-linearly to a compact spatial latent representation ; the inter-layer spatial prediction module is configured to obtain the predicted spatial latent representation from spatial features reconstructed from a neighboring layer ;​ The entropy coding module is configured to compress the spatial latent representation of the i-th enhancement layer prediction and the residual between the original spatial latent representation ;​ At the decoding end, residual of the i-th enhancement layer decoded and the predicted spatial latent representation corresponding addition results in a reconstructed spatial latent representation ; the decoder is configured to transform the reconstructed spatial latent representation of the i-th enhancement layer into a reconstructed spatial feature ; The angle generation module is configured to generate the angle feature of the i-th enhancement layer from the angle feature reconstructed from the base layer ;​ The spatial-angular context-aware reconstruction module is used to fuse spatial features and angular features to reconstruct a high-quality light field image .

2. The hierarchical light field image encoding method of claim 1, wherein, The inter-layer spatial prediction module is composed of three double-flow feature aggregation blocks, each of which explores co-location and local self-similarity in the branch to implicitly reconstruct high-frequency details in the high-resolution light field representation and predict the spatial latent representation.

Images