360-degree image compression method and device for adaptive latitude perception transform coding

Abstract

The invention discloses a 360-degree image compression method and a 360-degree image compression device for adaptive latitude perception transform coding, which have obvious advantages in code rate saving and can effectively solve the problem of distortion redundancy of ERP (Enterprise Resource Planning) images. The method comprises the following steps: (1) designing a self-adaptive latitude sensing module; (2) constructing a multi-scale gated convolutional neural network; (3) transforming spatial features guided by the modulation importance feature activation graph; and (4) constructing a learned 360-degree image compression overall framework.

Description

Technical Field

[0001] The present invention relates to the technical field of image compression, and in particular to a 360° image compression method of adaptive latitude perceptual transform coding, and a 360° image compression device of adaptive latitude perceptual transform coding. Background Art

[0002] 360° images, also known as panoramic images, omnidirectional images, or spherical images, cover scenes with a viewing angle of 360°×180°, unlike 2D images that only cover a limited plane. Observers can view images from different angles as if they were standing at the center of the 360° image. In recent years, with the rapid development of virtual reality (VR), 360° images, as a carrier of VR, provide an immersive experience and have become increasingly popular. This has led to a substantial increase in the number of 360° images, and these images require higher resolutions to ensure quality. The large number and ultra-high resolution of 360° images put tremendous pressure on storage and transmission. Therefore, there is an urgent need to develop an efficient 360° image compression technology to alleviate this pressure.

[0003] Over the past decade, 360° images have been represented in various formats and subsequently compressed using traditional codecs such as HEVC and VVC. Typically, 360° images are represented as equirectangular projection (ERP) format images, which are the most widely used format due to their ease of subsequent processing. However, ERP images gradually become distorted from the equator to the poles due to oversampling. Several methods have been proposed to alleviate these problems, including sampling density correction methods, reprojection methods, and perceptual compression methods. However, these methods have achieved limited success. In addition, these traditional codecs use nonlinear transform coding, making it difficult to achieve optimal rate-distortion performance.

[0004] In recent years, due to the powerful learning ability of deep neural networks, planar image compression based on traditional codec architecture (usually including three steps of transformation, quantization and entropy modeling) has achieved excellent performance in an end-to-end manner and has surpassed traditional codecs such as VVC. These end-to-end compression methods are more effective in modeling structural information and locally smooth planar images. However, for 360° images, in order to utilize end-to-end planar image compression methods, the spherical image must first be projected onto a plane. Regardless of the projection method used, this will inevitably introduce distortion or destroy the structural content of the image. Therefore, directly applying these methods to the projected ERP image will result in lower compression efficiency. Considering the characteristics of ERP images and the advantages of end-to-end planar image compression, Li et al. first proposed a learning-based end-to-end 360° image compression method. This method uses a neural network-based latitude adaptive coding scheme in the quantization and entropy modeling steps to assign different codewords to different regions according to the latitude on the sphere. In addition to quantization and entropy modeling, the reason why end-to-end planar image compression has achieved great success compared to traditional codec methods is that it utilizes powerful nonlinear transform coding based on super priors. These nonlinear transform codes convert the image into a more compact latent representation and directly predict the entropy of the latent representation through a super prior, thereby achieving excellent rate-distortion performance. Based on this, the present invention considers removing redundancy from the perspective of nonlinear transform coding based on a super prior. Summary of the invention

[0005] In order to overcome the defects of the prior art, the technical problem to be solved by the present invention is to provide a 360° image compression method of adaptive latitude-aware transform coding, which has significant advantages in bit rate saving and can effectively solve the distortion redundancy problem of ERP images.

[0006] The technical solution of the present invention is: the 360° image compression method of the adaptive latitude-aware transform coding comprises the following steps:

[0007] (1) Design an adaptive latitude perception module: First, select four rectangular convolution kernels of appropriate sizes: 5×11, 3×9, 5×7, and 7×7 to capture the distortion redundancy of different latitude regions. Use a 3×3 convolution kernel with expansion rates of (2,5), (1,4), (2,3), and (3,3) to achieve the effect of the rectangular convolution kernel, and use deep convolution for feature extraction.

[0008] (2) Construct a multi-scale gated convolutional neural network: First, the normalized features are divided into three features of equal size by channel, and then they are input into the deep convolutional network in three parallel branches. The sizes of these convolution kernels are 3×3, 5×5, and 7×7, respectively, to explore multi-scale representations; then the feature representations extracted at multiple scales are concatenated and channel fused through convolution; finally, the GELU function is applied to the fused features as a gating mechanism and multiplied element-wise with the original features;

[0009] (3) Transformation modulation of spatial features guided by importance feature activation maps: First, an importance feature activation network is used, which consists of three residual blocks and a sigmoid activation function, to generate an importance feature activation map m. Then, m is used as an external condition for the spatial feature transformation to generate affine parameters α and β for each element of the latent representation y, which is quantized as

[0010] On the other hand, y is passed to the entropy model network to learn the latent representation The probability distribution parameters μ and σ;

[0011] (4) Constructing the overall framework of learning 360° image compression: In order to ensure stable training, layer normalization is embedded before each module. Through convolution downsampling and transposition upsampling, ALAB is progressively inserted into the analysis transform and synthesis transform. The analysis transform gradually and adaptively perceives the distortion redundancy of different latitudes, while the synthesis transform gradually promotes the generation of the original distorted ERP image.

[0012] The potential representation y is modeled as a Gaussian distribution. In order to estimate the mean μ and variance σ of the y distribution well, the entropy model composed of the hyper-prior and the context model is used for estimation. Based on the analysis transformation, the synthesis transformation and the entropy model, the overall framework of the learned 360° image compression is constructed.

[0013] For end-to-end training.

[0014] The present invention designs an adaptive latitude distortion perception module to perceive distortion and redundancy at different latitudes, and a multi-scale gated convolutional neural network to suppress the feature redundancy brought by the adaptive latitude perception module and extract multi-scale rich contextual information. The adaptive latitude perception module and the multi-scale convolutional gated neural network are combined to construct an adaptive latitude perception block, and a spatial feature transformation guided by an importance feature activation map is designed to modulate latent variables and allocate the bit rate to important areas. Through the designed module and the existing entropy model, the entire learned 360° image compression framework is constructed. Therefore, this method has significant advantages in bit rate saving and can effectively solve the distortion redundancy problem of ERP images.

[0015] A 360° image compression device with adaptive latitude-aware transform coding is also provided, the device comprising:

[0016] Adaptive latitude perception module, which is configured to first select four rectangular convolution kernels of appropriate size 5

[0017] ×11, 3×9, 5×7, and 7×7 are used to capture the distortion redundancy in different latitude regions. A 3×3 convolution kernel with dilation rates of (2,5), (1,4), (2,3), and (3,3) is used to achieve the effect of a rectangular convolution kernel, and deep convolution is used for feature extraction.

[0018] The multi-scale gated convolutional neural network module is configured to first divide the normalized features into three features of equal size by channel, and then input them into the deep convolutional network in three parallel branches. The sizes of these convolution kernels are 3×3, 5×5, and 7×7 respectively, which are used to explore multi-scale representations; then the feature representations extracted at multiple scales are concatenated and channel fused by convolution; finally, the GELU function is applied to the fused features as a gating mechanism and multiplied element-wise with the original features;

[0019] A transform modulation module configured to first use an importance feature activation network comprising three residual blocks and a sigmoid activation function to generate an importance feature activation map m;

[0020] Then, using m as the external condition for the spatial feature transformation, affine parameters α and β are generated for each element of the latent representation y, which is quantized as On the other hand, y is passed to the entropy model network to learn the latent representation The probability distribution parameters μ and σ;

[0021] The overall framework of learned 360° image compression: its configuration is to embed layer normalization before each module for training stability, and ALAB is progressively inserted into the analysis transform and synthesis transform through convolution downsampling and transposed upsampling. The analysis transform gradually and adaptively perceives the distortion redundancy of different latitudes, while the synthesis transform gradually promotes the generation of original distorted ERP images;

[0022] The latent representation y is modeled as a Gaussian distribution. In order to make a good estimate of the mean μ and variance σ of the y distribution, the entropy model composed of the hyper-prior and the context model is used for estimation. Based on the analysis transformation, the synthesis transformation and the entropy model, a learning 360° image compression overall framework is constructed for end-to-end training. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1The flowchart of step (1) ALAM of the 360° image compression method of adaptive latitude-aware transform coding according to the present invention is shown. It represents the depth convolution with the convolution kernel size of 3×3 and the expansion rate of (m,n), Convk×k represents the convolution with the convolution kernel size of k×k, and C represents concatenation.

[0024] Figure 2 The flowchart of step (2) MCGFNN of the 360° image compression method of adaptive latitude-aware transform coding according to the present invention is shown, wherein DConvk×k represents a deep convolution with a convolution kernel size of k×k, and C represents concatenation.

[0025] Figure 3 The flowchart of step (3) IFAN of the 360° image compression method of adaptive latitude-aware transform coding according to the present invention is shown. Wherein, Conv(k,s) represents a convolution with a convolution kernel size of k×k and a step size of s, and m represents the generated important feature activation map.

[0026] Figure 4 A flow chart showing the network structure of the SFT of step (3) of the 360° image compression method of adaptive latitude-aware transform coding according to the present invention is shown.

[0027] Figure 5 A flow chart showing the network structure of ALAB in step (4) of the 360° image compression method of adaptive latitude-aware transform coding according to the present invention.

[0028] Figure 6 The overall network structure of step (4) of the 360° image compression method of adaptive latitude-aware transform coding according to the present invention is shown. Wherein, Conv(m,n) / TConv(m,n) represents convolution / transposed convolution with a convolution kernel of m×m and a step size of n, Q represents quantization, AD represents an arithmetic encoder, AE represents an arithmetic decoder, μ and σ represent the estimated mean and variance of y, and H, W and N represent the height, width and number of channels of the feature map, respectively. DETAILED DESCRIPTION

[0029] The 360° image compression method of the adaptive latitude-aware transform coding comprises the following steps:

[0030] (1) Figure 1 As shown in Figure 1, we design an adaptive latitude awareness module (ALAM): first select four rectangular convolution kernels of appropriate sizes: 5×11, 3×

[0031] 9, 5×7, and 7×7 are used to capture the distortion redundancy in different latitude regions. A 3×3 convolution kernel with dilation rates of (2,5), (1,4), (2,3), and (3,3) is used to achieve the effect of a rectangular convolution kernel, and deep convolution is used for feature extraction;

[0032] (2) Figure 2 As shown in the figure, a multi-scale convolutional gated feedforward neural network (MCGFNN) is constructed to suppress redundant information and enhance feature representation: first, the normalized features are divided into three features of equal size by channel, and then they are input into the deep convolutional network in three parallel branches. The sizes of these convolution kernels are 3×3, 5×5, and 7×7, respectively, to explore multi-scale representations; then the feature representations extracted at multiple scales are concatenated and channel fused by convolution; finally, the GELU function is applied to the fused features as a gating mechanism and multiplied element-wise with the original features;

[0033] (3) Transformation modulation importance feature activation map guided spatial features: Figure 3 As shown in Figure 2, we first use the Importance Feature Activation Network (Importance Feature Activation Network,

[0034] IFAN), the network includes three residual blocks and a sigmoid activation function to generate the important feature activation map m; Figure 4 As shown, m is then used as the external condition of the spatial feature transform (SFT) to generate affine parameters α and β for each element of the latent representation y, and y is quantized as On the other hand, y is passed to the entropy model network to learn the latent representation The probability distribution parameters μ and σ;

[0035] (4) Construct the overall framework of learning 360° image compression: Integrate the ALAM designed in step 1 and the MCGFNN designed in step 2 into a Transformer-like block, such as Figure 5As shown in the figure, this block is called Adaptive Latitude Aware Block (ALAB); for training stability, layer normalization is embedded before each module, and ALAB is progressively inserted into the analysis transform and synthesis transform through convolution downsampling and transposition upsampling. The analysis transform gradually and adaptively perceives the distortion redundancy of different latitudes, while the synthesis transform gradually promotes the generation of the original distorted ERP image; the potential representation y is modeled as a Gaussian distribution. In order to make a good estimate of the mean μ and variance σ of the y distribution, the entropy model composed of the hyper-prior and the context model is used for estimation; based on the analysis transform, the synthesis transform and the entropy model, the overall framework of 360° image compression learning is constructed.

[0036] (like Figure 6 ), for end-to-end training.

[0037] The present invention designs an adaptive latitude distortion perception module to perceive distortion and redundancy at different latitudes, and a multi-scale gated convolutional neural network to suppress the feature redundancy brought by the adaptive latitude perception module and extract multi-scale rich contextual information. The adaptive latitude perception module and the multi-scale convolutional gated neural network are combined to construct an adaptive latitude perception block, and a spatial feature transformation guided by an importance feature activation map is designed to modulate latent variables and allocate the bit rate to important areas. Through the designed module and the existing entropy model, the entire learned 360° image compression framework is constructed. Therefore, this method has significant advantages in bit rate saving and can effectively solve the distortion redundancy problem of ERP images.

[0038] Preferably, in step (1),

[0039]

[0040] Where X represents the input features, represents a depthwise convolution with a kernel size of 3×3 and a dilation rate of (m,n); then, in order to select an appropriate convolution kernel for each latitude and extract relevant features, a latitude adaptive selection mechanism is introduced.

[0041] Preferably, in step (1), the latitude adaptive selection mechanism includes:

[0042] First, the features extracted by different dilated convolutions are summed:

[0043]

[0044] Then, these summed features F add Perform channel and height-based average pooling P avg (·)

[0045] and the maximum pooling Pmax (·) to enhance their representation capabilities:

[0046] L favg =P avg (F add ), L fmax =P max (F add ) (6)

[0047] Among them, L favg and L fmax They are the feature descriptions after average pooling and maximum pooling along the channel and height directions respectively; in order to achieve information interaction between different feature descriptions and reduce complexity, the pooled features are concatenated and a convolution with a convolution kernel size of 1×1 is used to convert these pooled features into low-dimensional features:

[0048] L fam =Conv1×1([L favg ,L fmax ]) (7)

[0049] Among them, [·] represents the concatenation operation; then, N different convolutions with kernel size of 1×1 are used to extract multi-stream features to capture the dependencies between different dimensions:

[0050]

[0051] Among them, k is equal to 1, 2, 3 and 4, and the multi-stream features are spliced ​​along the channel dimension. Then, the Softmax operation is performed on the spliced ​​features in the channel dimension to obtain each feature at a specific latitude h The attention weight of this process is expressed as:

[0052]

[0053] in, represents the attention score of the kth feature at a certain latitude h, and the final feature is weighted by the corresponding adaptive latitude attention weight; then, the weighted feature passes through a convolution layer to smooth the features of different latitudes:

[0054]

[0055] Preferably, in step (2), the convolution kernel is used to explore multi-scale representation:

[0056] X3=DConv 3×3 (X l ) (11)

[0057] X5=DConv 5×5 (Xl ) (12)

[0058] X7=DConv 7×7 (X l ) (13)

[0059] Then, the feature representations extracted at multiple scales are concatenated and channel fused through convolution; finally, the fused features are subjected to the GELU function as a gating mechanism and element-wise multiplication with the original features; this exploration and integration process is expressed as follows:

[0060] X′=Conv 1x1 ([X3,X5,X7]) (14)

[0061] X″=GELU(X′)·Conv 1x1 (X l ) (15)

[0062] Among them, [·] indicates channel-by-channel splicing.

[0063] Preferably, in step (3), m is used as an external condition for spatial feature transformation to generate affine parameters α and β for each element of the potential representation y. This process is expressed as follows:

[0064]

[0065] in, It is the characteristic before modulation.

[0066] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes the steps of the above-mentioned embodiment method, and the storage medium can be: ROM / RAM, disk, CD, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes a 360° image compression device with adaptive latitude-aware transform coding, which is usually represented in the form of functional modules corresponding to the steps of the method. The device includes:

[0067] Adaptive latitude perception module, which is configured to first select four rectangular convolution kernels of appropriate size 5

[0068] ×11, 3×9, 5×7, and 7×7 are used to capture the distortion redundancy in different latitude regions. A 3×3 convolution kernel with dilation rates of (2,5), (1,4), (2,3), and (3,3) is used to achieve the effect of a rectangular convolution kernel, and deep convolution is used for feature extraction.

[0069] The multi-scale gated convolutional neural network module is configured to first divide the normalized features into three features of equal size by channel, and then input them into the deep convolutional network in three parallel branches. The sizes of these convolution kernels are 3×3, 5×5, and 7×7 respectively, which are used to explore multi-scale representations; then the feature representations extracted at multiple scales are concatenated and channel fused by convolution; finally, the GELU function is applied to the fused features as a gating mechanism and multiplied element-wise with the original features;

[0070] A transform modulation module configured to first use an importance feature activation network comprising three residual blocks and a sigmoid activation function to generate an importance feature activation map m;

[0071] Then, using m as the external condition for the spatial feature transformation, affine parameters α and β are generated for each element of the latent representation y, which is quantized as On the other hand, y is passed to the entropy model network to learn the latent representation The probability distribution parameters μ and σ;

[0072] The overall framework of learned 360° image compression: its configuration is to embed layer normalization before each module for training stability, and ALAB is progressively inserted into the analysis transform and synthesis transform through convolution downsampling and transposed upsampling. The analysis transform gradually and adaptively perceives the distortion redundancy of different latitudes, while the synthesis transform gradually promotes the generation of original distorted ERP images;

[0073] The potential representation y is modeled as a Gaussian distribution. In order to estimate the mean μ and variance σ of the y distribution well, the entropy model composed of the hyper-prior and the context model is used for estimation. Based on the analysis transformation, the synthesis transformation and the entropy model, the overall framework of the learned 360° image compression is constructed.

[0074] For end-to-end training.

[0075] Preferably, in the adaptive latitude perception module,

[0076]

[0077]

[0078] Where X represents the input features, represents a depthwise convolution with a kernel size of 3×3 and a dilation rate of (m,n); then, in order to select an appropriate convolution kernel for each latitude and extract relevant features, a latitude adaptive selection mechanism is introduced.

[0079] Preferably, in the adaptive latitude perception module, the latitude adaptive selection mechanism includes:

[0080] First, the features extracted by different dilated convolutions are summed:

[0081]

[0082] Then, these summed features F add Perform channel and height-based average pooling P avg (·)

[0083] and the maximum pooling P max (·) to enhance their representation capabilities:

[0084] L favg =P avg (F add ), L fmax =P max (F add ) (6)

[0085] Among them, L favg and L fmax They are the feature descriptions after average pooling and maximum pooling along the channel and height directions respectively; in order to achieve information interaction between different feature descriptions and reduce complexity, the pooled features are concatenated and a convolution with a convolution kernel size of 1×1 is used.

[0086] Convert these pooled features into low-dimensional features:

[0087] L fam =Conv1×1([L favg ,L fmax ]) (7)

[0088] Among them, [·] represents the concatenation operation; then, N convolution kernels with different sizes of 1×1 are used.

[0089] Convolution is used to extract multi-stream features to capture dependencies between different dimensions:

[0090]

[0091] Among them, k is equal to 1, 2, 3 and 4, and the multi-stream features are spliced ​​along the channel dimension. Then, the Softmax operation is performed on the spliced ​​features in the channel dimension to obtain each feature at a specific latitude h The attention weight of this process is expressed as:

[0092]

[0093] in, represents the attention score of the kth feature at a certain latitude h, and the final feature is weighted by the corresponding adaptive latitude attention weight; then, the weighted feature passes through a convolution layer to smooth the features of different latitudes:

[0094]

[0095] Preferably, in the multi-scale gated convolutional neural network module, the convolution kernel is used to explore multi-scale representation:

[0096] X3=DConv 3×3 (X l ) (11)

[0097] X5=DConv 5×5 (X l ) (12)

[0098] X7=DConv 7×7 (X l ) (13)

[0099] Then, the feature representations extracted at multiple scales are concatenated and channel fused through convolution; finally, the fused features are subjected to the GELU function as a gating mechanism and element-wise multiplication with the original features; this exploration and integration process is expressed as follows:

[0100] X′=Conv 1x1 ([X3,X5,X7]) (14)

[0101] X″=GELU(X′)·Conv 1x1 (X l ) (15)

[0102] Among them, [·] indicates channel-by-channel splicing.

[0103] Preferably, in the transformation modulation module, m is used as an external condition for spatial feature transformation to generate affine parameters α and β for each element of the potential representation y. This process is expressed as follows:

[0104]

[0105] in, It is the characteristic before modulation.

[0106] The experimental method of the present invention is described in detail below.

[0107] 1. Experimental data: A total of 19,790 high-quality 360° images were collected from the Flickr sharing website. This dataset is called the LIC360 dataset, of which 19,590 images are used for the training set and 200 images are used for the test set. In addition, the ODI-SR and SUN360 datasets, each containing 100 high-resolution images with a resolution of 1024×512, are also used as test sets.

[0108] 2. Metrics: The present invention uses Bits Per Pixel (BPP), Weighted-to-Spherically-uniform Peak Signal-to-Noise Ratio (WS-PSNR) and Weighted-to-Spherically-uniform Peak Signal-to-Noise Ratio (WS-SSIM) to compare the rate-distortion performance of the present invention with different projection methods and learning methods. In order to quantitatively evaluate the performance, the BD-rate of each method is calculated, and its WS-PSNR BPP curve is used to compare with the benchmark WS-PSNR BPP curve. The WS-PSNR BPP benchmark curve is set by VVC (the BD-rate of VVC is 0), and the reference software of VVC is VTM-9.1.

[0109] 3. Experimental setup: The present invention uses the Compress AI framework to implement the designed 360° image compression model. Several different models were trained from scratch to represent different levels of image quality, and each model corresponds to a specific lambda. Specifically, the designed model was trained using lambda in the range of {0.0018, 0.0035, 0.0067, 0.0130, 0.0250, 0.0483} to optimize WMSE. Unlike the commonly used learning optimization loss in planar image compression, the use of MSE is inappropriate due to the spherical characteristics of 360° images. Therefore, WMSE is used as the optimized loss function. Similarly, WSSIM is also used as the optimized loss function, and the range of lambda is {2.40, 4.58, 8.73, 16.64}. All models were trained for 200 cycles using the Adam optimizer with a batch size of 4. The initial learning rate was set to 1×10 -4 , and after 180 cycles it drops to 1×10 -5 The number of channels N and M in all our models are set to 128 and 320, respectively.

[0110] 4. Comparison with different compression methods: The method of the present invention is compared with the learning-based 360° image compression method proposed by Li22. Due to the limited number of learning-based 360° image compression methods, the method of the present invention is also compared with several learning-based planar image compression methods, including Ballé18, Minnen18, and Cheng20. In order to highlight the gap between the learning-based compression method and the traditional codec, the present invention also adds the traditional codec methods BPG and VVC to the comparison.

[0111] As shown in Table 3, the method of the present invention achieves 8.96%, 3.75% and 3.16% BD-rate savings on the LIC360, ODI-SR and SUN360 datasets respectively compared with VVC. As shown in Table 1, compared with the learning-based 360° image compression method proposed by Li22, which increases the BD-rate by 26.83% on the LIC360 dataset, the method of the present invention obviously far exceeds Li22 in terms of bit rate savings. In addition, compared with other methods relative to VVC, the method of the present invention is the only method that achieves bit rate savings on the LIC360 dataset, indicating that the present invention can effectively solve the distortion redundancy problem of ERP images.

[0112] In order to fairly evaluate the performance of various methods, the present invention also provides a comparison of encoding time, decoding time, number of parameters, GMACs and BD-rate of different methods on the LIC360 dataset in Table 1. The configuration of the test machine is Intel i9-10850K CPU and NVIDIA GTX 4090 GPU. Since VVC and BPG use traditional codecs for intra-frame prediction and do not provide parameters and GMACs. The present invention is similar to the Minnen18 method in terms of the number of parameters and GMACs, but has a significant advantage in bit rate saving. Compared with the learned 360° image compression method Li22, the present invention reduces GMACs by nearly six times and the number of parameters is also halved. Since the present invention uses a serial autoregressive entropy model like Minnen18 and Cheng20, the encoding and decoding time is relatively long. However, compared with the complex intra-frame prediction mode of VVC, the encoding time of the present invention is relatively low. Li22 and Ballé18 do not use this serial autoregressive entropy model, so the encoding and decoding time is shorter. The present invention has relatively low GMACs and parameter numbers, and is the only method that achieves BD-rate savings, proving the high efficiency of the present invention.

[0113] Table 1 Comparison of different test results on the LIC360 dataset

[0114] method Encoding time (s)↓ Decoding time (s)↓ Parameter quantity↓ Calculation amount↓ BD-rate(%)↓ VVC 129.752 0.057 - - 0.00 BPG 0.215 0.111 - - 21.13 Ballé 18 0.045 0.051 11.82 101.69 49.27 Minnen18 3.364 6.660 25.50 124.06 25.64 Cheng20 3.357 6.654 29.63 554.16 21.89 Li22 0.460 0.613 52.63 772.84 26.83 Method of the present invention 3.326 6.596 24.08 125.43 -8.96

[0115] 5. Ablation experiment: In order to verify the effectiveness of different modules of the present invention, the present invention conducted ablation experiments on nonlinear transformation, ALAM, MCGFNN and m respectively.

[0116] Effectiveness of nonlinear transformation: The nonlinear transformation of the present invention is compared with that of the Cheng20 method. For fair comparison, the Cheng20 model was retrained using WMSE as the loss function. In the present invention, the important feature activation map m was removed and the entropy model of the present invention was ensured to be consistent with that of Cheng20. The results of the ablation experiment are shown in Table 2. It can be seen that the present invention outperforms Cheng20 (WMSE) on the LIC360 and ODI-SR datasets. Although the performance on the SUN360 dataset is slightly inferior, as shown in Table 1, Cheng20 requires more GMACs and parameters than the present invention. Generally, nonlinear transformation consists of upsampling / downsampling layers and attention layers. In terms of upsampling and downsampling, Cheng20 uses a residual block, while the method of the present invention uses a 3×3 convolutional layer. Therefore, compared with Cheng20, the present invention has simpler upsampling and downsampling layers, which further proves the effectiveness of ALAB proposed in the present invention in nonlinear transformation.

[0117] Effectiveness of ALAM: In these experiments, ALAM is replaced by a single convolutional layer with a kernel size of 3×3 and a dilation rate of (3,3). The results of the ablation experiments are shown in Table 3. Obviously, the absence of ALAM leads to a degradation of rate-distortion performance. ALAM achieves 4.93%, 6.01%, and 8.51% BD-rate savings on the LIC360, ODI-SR, and SUN360 datasets, respectively, which proves its effectiveness in 360° image compression.

[0118] Effectiveness of MCGFNN: The present invention also evaluates the effectiveness of the proposed MCGFNN. MCGFNN is replaced by FFN, as shown in Table 3. It can be observed that MCGFNN improves the rate-distortion performance and achieves BD-rate savings of 1.86%, 3.17% and 3.14% on the LIC360, ODI-SR and SUN360 datasets, respectively. However, FFN fails to simultaneously utilize multi-scale knowledge. In contrast, the MCGFNN of the present invention explores and combines multi-scale cues in the forward pass and suppresses the redundant features generated by the fusion of multiple rectangular dilated convolutions, thereby significantly improving the performance.

[0119] Effectiveness of m: The present invention conducts ablation experiments to verify the effectiveness of m. The results of the ablation experiments are shown in Table 3. As can be seen from Table 3, the absence of m will lead to a decrease in rate-distortion performance. With the help of m, 0.75%, 0.67% and 1.51% BD-rate savings are achieved on the LIC360, ODI-SR and SUN360 datasets, respectively.

[0120] Table 2 Nonlinear transformation ablation experiments on LIC360, ODI-SR and SUN360 datasets

[0121]

[0122]

[0123] Table 3 Ablation experiments of different modules on LIC360, ODI-SR and SUN360 datasets

[0124]

[0125] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the protection scope of the technical solution of the present invention.

Claims

1. A 360° image compression method based on adaptive latitude-aware transform coding, characterized in that: The method comprises the following steps: (1) Design an adaptive latitude perception module: First, select four rectangular convolution kernels of appropriate sizes: 5×11, 3×9, 5×7, and 7×7 to capture the distortion redundancy of different latitude regions. Use a 3×3 convolution kernel with expansion rates of (2,5), (1,4), (2,3), and (3,3) to achieve the effect of the rectangular convolution kernel, and use deep convolution for feature extraction. (2) Construct a multi-scale gated convolutional neural network: First, the normalized features are divided into three features of equal size by channel, and then they are input into the deep convolutional network in three parallel branches. The sizes of these convolution kernels are 3×3, 5×5, and 7×7, respectively, to explore multi-scale representations; then the feature representations extracted at multiple scales are concatenated and channel fused through convolution; finally, the GELU function is applied to the fused features as a gating mechanism and multiplied element-wise with the original features; (3) Transformation modulation of spatial features guided by importance feature activation maps: First, an importance feature activation network is used, which consists of three residual blocks and a sigmoid activation function, to generate an importance feature activation map m. Then, m is used as an external condition for the spatial feature transformation to generate affine parameters α and β for each element of the latent representation y, which is quantized as On the other hand, y is passed to the entropy model network to learn the latent representation The probability distribution parameters μ and σ; (4) Constructing an overall framework for learning 360° image compression: In order to ensure training stability, layer normalization is embedded before each module. ALAB is progressively inserted into the analysis transform and synthesis transform through convolution downsampling and transposition upsampling. The analysis transform gradually and adaptively perceives the distortion redundancy of different dimensions, while the synthesis transform gradually promotes the generation of the original distorted ERP image. The latent representation y is modeled as a Gaussian distribution. In order to make a good estimate of the mean μ and variance σ of the y distribution, the entropy model composed of the hyper-prior and the context model is used for estimation. Based on the analysis transform, synthesis transform and entropy model, an overall framework for learning 360° image compression is constructed for end-to-end training.

2. The 360° image compression method of adaptive latitude-aware transform coding according to claim 1, characterized in that: In the step (1), Where X represents the input features, represents a depthwise convolution with a kernel size of 3×3 and a dilation rate of (m,n); then, in order to select an appropriate convolution kernel for each latitude and extract relevant features, a latitude adaptive selection mechanism is introduced.

3. The 360° image compression method of adaptive latitude-aware transform coding according to claim 2, characterized in that: In the step (1), the latitude adaptive selection mechanism includes: First, the features extracted by different dilated convolutions are summed: Then, these summed features F add Perform channel and height-based average pooling P avg (·) and the maximum pooling P max (·) to enhance their representation capabilities: L favg =P avg (F add ), L fmax =P max (F add ) (6) Among them, L favg and L fmax They are the feature descriptions after average pooling and maximum pooling along the channel and height directions respectively; in order to achieve information interaction between different feature descriptions and reduce complexity, the pooled features are concatenated and a convolution with a convolution kernel size of 1×1 is used to convert these pooled features into low-dimensional features: L fam =Conv1×1([L favg ,L fmax ]) (7) Among them, [·] represents the concatenation operation; then, N different convolutions with kernel size of 1×1 are used to extract multi-stream features to capture the dependencies between different dimensions: Among them, k is equal to 1, 2, 3 and 4, and the multi-stream features are spliced ​​along the channel dimension. Then, the Softmax operation is performed on the spliced ​​features in the channel dimension to obtain each feature at a specific latitude h The attention weight of this process is expressed as: in, represents the attention score of the kth feature at a certain latitude h, and the final feature is weighted by the corresponding adaptive latitude attention weight; then, the weighted feature passes through a convolution layer to smooth the features of different latitudes:

4. The 360° image compression method of adaptive latitude-aware transform coding according to claim 3, characterized in that: In step (2), the convolution kernel is used to explore multi-scale representation: X3=DConv 3×3 (X l ) (11) X5=DConv 5×5 (X l ) (12) X7=DConv 7×7 (X l ) (13) Then, the feature representations extracted at multiple scales are concatenated and channel fused through convolution; finally, the fused features are subjected to the GELU function as a gating mechanism and element-wise multiplication with the original features; this exploration and integration process is expressed as follows: X′=Conv 1x1 ([X3,X5,X7]) (14) X″=GELU(X′)·Conv 1x1 (X l ) (15) Among them, [·] indicates channel-by-channel splicing.

5. The 360° image compression method of adaptive latitude-aware transform coding according to claim 3, characterized in that: In step (3), m is used as the external condition of the spatial feature transformation to generate affine parameters α and β for each element of the potential representation y. This process is expressed as follows: in, It is the characteristic before modulation.

6. A 360° image compression device with adaptive latitude-aware transform coding, characterized in that: The device includes: The adaptive latitude perception module is configured by first selecting four rectangular convolution kernels of appropriate sizes: 5×11, 3×9, 5×7, and 7×7 to capture the distortion redundancy of different latitude regions, and then using a 3×3 convolution kernel with dilation rates of (2,5), (1,4), (2,3), and (3,3) to achieve the effect of the rectangular convolution kernel, and using deep convolution for feature extraction; The multi-scale gated convolutional neural network module is configured to first divide the normalized features into three features of equal size by channel, and then input them into the deep convolutional network in three parallel branches. The sizes of these convolution kernels are 3×3, 5×5, and 7×7 respectively, which are used to explore multi-scale representations; then the feature representations extracted at multiple scales are concatenated and channel fused by convolution; finally, the GELU function is applied to the fused features as a gating mechanism and multiplied element-wise with the original features; The transform modulation module is configured to first use an important feature activation network consisting of three residual blocks and a sigmoid activation function to generate an important feature activation map m; then, m is used as an external condition for the spatial feature transformation to generate affine parameters α and β for each element of the latent representation y, which is quantized as On the other hand, y is passed to the entropy model network to learn the latent representation The probability distribution parameters μ and σ; The overall framework of learned 360° image compression: its configuration is to embed layer normalization before each module for stable training. Through convolution downsampling and transposed upsampling, ALAB is progressively inserted into the analysis transform and synthesis transform. The analysis transform gradually and adaptively perceives the distortion redundancy of different latitudes, while the synthesis transform gradually promotes the generation of the original distorted ERP image; the potential representation y is modeled as a Gaussian distribution. In order to make a good estimate of the mean μ and variance σ of the y distribution, the entropy model composed of the hyper-prior and context model is used for estimation; based on the analysis transform, synthesis transform and entropy model, the overall framework of learned 360° image compression is constructed for end-to-end training.

7. The 360° image compression device of adaptive latitude-aware transform coding according to claim 6, characterized in that: In the adaptive latitude perception module, Where X represents the input features, represents a depthwise convolution with a kernel size of 3×3 and a dilation rate of (m,n); then, in order to select an appropriate convolution kernel for each latitude and extract relevant features, a latitude adaptive selection mechanism is introduced.

8. The 360° image compression device of adaptive latitude-aware transform coding according to claim 7, characterized in that: In the adaptive latitude perception module, the latitude adaptive selection mechanism includes: First, the features extracted by different dilated convolutions are summed: Then, these summed features F add Perform channel and height-based average pooling P avg (·) and the maximum pooling P max (·) to enhance their representation capabilities: L favg =P avg (F add ), L fmax =P max (F add ) (6) Among them, L favg and L fmax They are the feature descriptions after average pooling and maximum pooling along the channel and height directions respectively; in order to achieve information interaction between different feature descriptions and reduce complexity, the pooled features are concatenated and a convolution with a convolution kernel size of 1×1 is used to convert these pooled features into low-dimensional features: L fam =Conv1×1([L favg ,L fmax ]) (7) Among them, [·] represents the concatenation operation; then, N different convolutions with kernel size of 1×1 are used to extract multi-stream features to capture the dependencies between different dimensions: Among them, k is equal to 1, 2, 3 and 4, and the multi-stream features are spliced ​​along the channel dimension. Then, the Softmax operation is performed on the spliced ​​features in the channel dimension to obtain each feature at a specific latitude h The attention weight of this process is expressed as: in, represents the attention score of the kth feature at a certain latitude h, and the final feature is weighted by the corresponding adaptive latitude attention weight; then, the weighted feature passes through a convolution layer to smooth the features of different latitudes:

9. The 360° image compression device of adaptive latitude-aware transform coding according to claim 8, characterized in that: In the multi-scale gated convolutional neural network module, the convolution kernel is used to explore multi-scale representations: X3=DConv 3×3 (X l ) (11) X5=DConv 5×5 (X l ) (12) X7=DConv 7×7 (X l ) (13) Then, the feature representations extracted at multiple scales are concatenated and channel fused through convolution; finally, the fused features are subjected to the GELU function as a gating mechanism and element-wise multiplication with the original features; this exploration and integration process is expressed as follows: X′=Conv 1x1 ([X3,X5,X7]) (14) X″=GELU(X′)·Conv 1x1 (X l ) (15) Among them, [·] indicates channel-by-channel splicing.

10. The 360° image compression device of adaptive latitude-aware transform coding according to claim 9, characterized in that: In the transformation modulation module, m is used as the external condition of the spatial feature transformation to generate affine parameters α and β for each element of the potential representation y. This process is expressed as follows: in, It is the characteristic before modulation.

Images