360-degree image compression method and device using adaptive latitude-aware transform coding

Abstract

The application discloses a 360-degree image compression method and device based on adaptive latitude-aware transform coding, has obvious advantages in code rate saving, and can effectively solve the distortion redundancy problem of ERP images.The method comprises the following steps: (1) designing an adaptive latitude-aware module; (2) constructing a multi-scale gated convolutional neural network; (3) guiding spatial features by transform modulation importance feature activation maps; and (4) constructing a learned overall framework of 360-degree image compression.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image compression, and in particular to a 360° image compression method based on adaptive latitude-aware transform coding and a 360° image compression device based on adaptive latitude-aware transform coding. BACKGROUND

[0002] A 360° image, also known as a panoramic image, omnidirectional image or spherical image, is different from a 2D image which only covers a limited plane, in that it covers a scene with a viewing angle of 360°x180°. An observer can view the image from different angles as if he / she were standing at the center position of the 360° image. In recent years, with the rapid development of virtual reality (VR), 360° images as a carrier of VR provide an immersive experience and have become increasingly popular. This has led to a significant increase in the number of 360° images, and these images need to have a high resolution in order to ensure quality. The huge number and ultra-high resolution of 360° images have put a huge pressure on storage and transmission, and therefore there is an urgent need to develop an efficient 360° image compression technology to alleviate this pressure.

[0003] In the past decade, 360° images have been represented in various formats and then compressed using traditional codecs such as HEVC and VVC. Typically, 360° images are represented as equirectangular projection (ERP) format images, which are the most widely used format due to their ease of subsequent processing. However, ERP images gradually exhibit distortion from the equator to the poles due to oversampling. Some methods have been proposed to alleviate these problems, including sampling density correction methods, re-projection methods, and perceptual compression methods. However, these methods have achieved limited success. In addition, these traditional codecs use non-linear transform coding, making it difficult to achieve the best rate-distortion performance.

[0004] In recent years, due to the powerful learning ability of deep neural networks, the planar image compression based on traditional coding and decoding architecture (usually including three steps of transformation, quantization and entropy modeling) has achieved excellent performance in an end-to-end manner, and has surpassed traditional coding and decoding such as VVC. These end-to-end compression methods are more effective in modeling structural information and locally smooth planar images. However, for 360° images, in order to utilize the end-to-end planar image compression method, the spherical image must be projected onto a plane first. No matter what projection method is used, this will inevitably introduce distortion or destroy the structural content of the image. Therefore, directly applying these methods to the projected ERP image will result in lower compression efficiency. Considering the characteristics of ERP images and the advantages of end-to-end planar image compression, Li et al. first proposed a learning-based end-to-end 360° image compression method. This method uses a neural network-based latitude adaptive coding scheme in the quantization and entropy modeling steps, which allocates different code words to different regions according to the latitude on the sphere. In addition to quantization and entropy modeling, the reason why end-to-end planar image compression has achieved great success compared to traditional coding and decoding methods is that it utilizes powerful nonlinear transform coding based on hyperprior. These nonlinear transform coding converts the image into a more compact latent representation and directly predicts the entropy of the latent representation through hyperprior, thereby achieving excellent rate-distortion performance. Based on this, the present application considers from the perspective of nonlinear transform coding based on hyperprior to remove redundancy. SUMMARY

[0005] To overcome the defects of the prior art, the technical problem solved by the present application is to provide an adaptive latitude-aware transform coding 360° image compression method which has significant advantages in code rate saving and can effectively solve the distortion redundancy problem of ERP images.

[0006] The technical solution of the present application is: this adaptive latitude-aware transform coding 360° image compression method includes the following steps:

[0007] (1) Design an adaptive latitude-aware module: first select four rectangular convolution kernels of appropriate size 5x11, 3x9, 5x7, 7x7 to capture the distortion redundancy of different latitude regions, use 3x3 convolution kernels with dilation rates of (2, 5), (1, 4), (2, 3) and (3, 3) to achieve the effect of rectangular convolution kernels, and use deep convolution for feature extraction;

[0008] (2) Constructing a multi-scale gated convolutional neural network: first, the normalized features are divided into three equal-sized features by channel, then they are input into the deep convolutional network in three parallel branches, and the size of the convolution kernel is 3*3, 5*5, 7*7, which is used to explore multi-scale representation; then the multi-scale extracted feature representation is spliced and fused by convolution; finally, the fused features are applied to the GELU function as the gating mechanism, and are multiplied by the original features element by element;

[0009] (3) Transform modulation importance feature activation map guided spatial feature: first, an importance feature activation network is used, which includes three residual blocks and a sigmoid activation function to generate an importance feature activation map m; then, using m as the external condition of spatial feature transformation, affine parameters alpha and beta are generated for each element of the latent representation y, which is quantized as

[0010] On the other hand, y is transmitted to the entropy model network to learn the probability distribution parameters mu and sigma of the latent representation ;

[0011] (4) Constructing a learning 360° image compression overall framework: in order to train stably, layer normalization is embedded before each module, ALAB is gradually inserted into the analysis transformation and synthesis transformation through convolution downsampling and transposed upsampling, the analysis transformation gradually and adaptively perceives the distortion redundancy of different latitudes, and the synthesis transformation gradually promotes the generation of the original distorted ERP image;

[0012] The latent representation y is modeled as a Gaussian distribution, in order to well estimate the mean mu and variance sigma of the y distribution, an entropy model composed of hyper-prior and context model is used for estimation; based on the analysis transformation, synthesis transformation and entropy model, a learning 360° image compression overall framework is constructed,

[0013] for end-to-end training.

[0014] The present application designs an adaptive latitude distortion perception module to perceive distortion and redundancy at different latitudes, a multi-scale gated convolutional neural network to suppress the feature redundancy brought by the adaptive latitude perception module and extract multi-scale rich context information, combines the adaptive latitude perception module and the multi-scale convolutional gated neural network to construct an adaptive latitude perception block, designs an importance feature activation map guided spatial feature transformation to modulate the latent variable and allocate code rate to important areas, and through the designed module and the existing entropy model, an entire learning 360° image compression framework is constructed, so that the method has significant advantages in code rate saving and can effectively solve the distortion redundancy problem of ERP images.

[0015] An adaptive latitude-aware transform coding 360° image compression device is also provided, the device comprising:

[0016] an adaptive latitude-aware module configured to first select four rectangular convolution kernels 5 of appropriate size to capture the distortion redundancy of different latitude regions, using 3x3 convolution kernels with dilation rates of (2, 5), (1, 4), (2, 3) and (3, 3) to achieve the effect of rectangular convolution kernels, and using deep convolution for feature extraction;

[0017] an adaptive latitude-aware module configured to first select four rectangular convolution kernels 5 of appropriate size to capture the distortion redundancy of different latitude regions, using 3x3 convolution kernels with dilation rates of (2, 5), (1, 4), (2, 3) and (3, 3) to achieve the effect of rectangular convolution kernels, and using deep convolution for feature extraction;

[0018] a multi-scale gated convolutional neural network module configured to first divide the normalized features by channel into three features of equal size, then input them into a deep convolutional network in three parallel branches, the sizes of the convolution kernels being 3x3, 5x5 and 7x7 respectively, for exploring multi-scale representations; then concatenate the multi-scale extracted feature representations and fuse them by convolution; finally apply a GELU function as a gating mechanism to the fused features and element-wise multiply them with the original features;

[0019] a transform modulation module configured to first use an importance feature activation network, which includes three residual blocks and a sigmoid activation function, to generate an importance feature activation map m;

[0020] Then, using m as an external condition for spatial feature transformation, affine parameters a and b are generated for each element of the latent representation y, which is quantized as On the other hand, y is transmitted to an entropy model network to learn the probability distribution parameters μ and σ of the latent representation

[0021] The learned 360° image compression overall framework is configured to be stable for training, with layer normalization embedded before each module, and ALAB progressively inserted into the analysis transform and synthesis transform by convolution downsampling and transposed upsampling, the analysis transform gradually and adaptively perceiving the distortion redundancy of different latitudes, while the synthesis transform gradually promoting the generation of the original distorted ERP image;

[0022] The latent representation y is modeled as a Gaussian distribution, and in order to estimate the mean μ and variance σ of the y distribution well, an entropy model composed of a hyper-prior and a context model is used for estimation; based on the analysis transform, the synthesis transform and the entropy model, a learned 360° image compression overall framework is constructed for end-to-end training. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 ​The flow chart of step (1) ALAM of the adaptive latitude aware transform coding 360° image compression method according to the present application is shown. Wherein, represents a deep convolution with a convolution kernel size of 3x3 and an expansion rate of (m, n), Convk×k represents a convolution with a convolution kernel size of kxk, and C represents concatenation.

[0024] Figure 2 The flow chart of step (2) MCGFNN of the adaptive latitude aware transform coding 360° image compression method according to the present application is shown. Wherein, DConvk×k represents a deep convolution with a convolution kernel size of kxk, and C represents concatenation.

[0025] Figure 3 The flow chart of step (3) IFAN of the adaptive latitude aware transform coding 360° image compression method according to the present application is shown. Wherein, Conv(k, s) represents a convolution with a convolution kernel size of kxk and a step size of s, and m represents the generated importance feature activation map.

[0026] Figure 4 The flow chart of the network structure of step (3) SFT of the adaptive latitude aware transform coding 360° image compression method according to the present application is shown.

[0027] Figure 5 The flow chart of the network structure of step (4) ALAB of the adaptive latitude aware transform coding 360° image compression method according to the present application is shown.

[0028] Figure 6 The overall network structure of step (4) of the adaptive latitude aware transform coding 360° image compression method according to the present application is shown. Wherein, Conv(m, n) / TConv(m, n) represents a convolution with a convolution kernel of mxm and a step size of n / transpose convolution, Q represents quantization, AD represents an arithmetic encoder, AE represents an arithmetic decoder, μ and σ represent the estimated mean and variance of y, H, W and N represent the height, width and channel number of the feature map, respectively. DETAILED DESCRIPTION

[0029] The adaptive latitude aware transform coding 360° image compression method comprises the following steps:

[0030] (1) As shown in Figure 1 , an adaptive latitude aware module (ALAM) is designed: first, four rectangular convolution kernels with appropriate sizes 5x11, 3x

[0031] 9, 5×7, and 7×7 are used to capture distortion redundancy in different latitudinal regions. 3×3 convolution kernels with dilation rates of (2,5), (1,4), (2,3), and (3,3) are used to achieve the effect of rectangular convolution kernels, and depthwise convolution is used for feature extraction.

[0032] (2) Figure 2 As shown, a multi-scale convolutional-gated feedforward neural network (MCGFNN) is constructed to suppress redundant information and enhance feature representation: First, the normalized features are divided into three equal-sized features according to channels, and then they are input into a deep convolutional network in three parallel branches. The kernel sizes of these convolutions are 3×3, 5×5, and 7×7, respectively, to explore multi-scale representation; then, the feature representations extracted from the multi-scale are concatenated and channel fusion is performed through convolution; finally, the GELU function is applied as a gating mechanism to the fused features, and element-wise multiplication is performed with the original features.

[0033] (3) Spatial features guided by the activation map of the transformation modulation importance feature: such as Figure 3 As shown, the Importance Feature Activation Network is first used.

[0034] The network (IFAN) consists of three residual blocks and a sigmoid activation function to generate an activation map m of importance features; for example... Figure 4 As shown, then, using m as an external condition for the Spatial Feature Transform (SFT), affine parameters α and β are generated for each element of the latent representation y, and y is quantized as... On the other hand, y is fed into the entropy model network to learn the latent representation. The probability distribution parameters μ and σ;

[0035] (4) Constructing a learning-based 360° image compression framework: Integrating the ALAM designed in step one and the MCGFNN designed in step two into a Transformer-like block, such as... Figure 5As shown, the block is called an adaptive latitude aware block (ALAB); in order to train stability, layer normalization is embedded in front of each module, ALAB is gradually inserted into the analysis transform and the synthesis transform through convolution downsampling and transposed upsampling, the analysis transform gradually and adaptively perceives the distortion redundancy of different latitudes, and the synthesis transform gradually promotes the generation of the original distorted ERP image; the latent representation y is modeled as a Gaussian distribution, in order to well estimate the mean μ and variance σ of the y distribution, an entropy model composed of a hyper-prior and a context model is used for estimation; based on the analysis transform, the synthesis transform and the entropy model, a learning 360° image compression overall framework is constructed

[0036] (as shown in FIG. 1), which is used for end-to-end training. Figure 6

[0037] The present application designs an adaptive latitude distortion perception module to perceive distortion and redundancy of different latitudes, a multi-scale gated convolutional neural network to suppress feature redundancy brought by the adaptive latitude perception module and extract multi-scale rich context information, combines the adaptive latitude perception module and the multi-scale convolutional gated neural network to construct an adaptive latitude perception block, designs an importance feature activation map guided spatial feature transformation to modulate the latent variable and allocate code rate to important areas, and through the designed module and the existing entropy model, an entire learning 360° image compression framework is constructed, so that the method has significant advantages in code rate saving and can effectively solve the distortion redundancy problem of ERP images.

[0038] Preferably, in the step (1),

[0039]

[0040] wherein X represents an input feature, represents a deep convolution with a convolution kernel size of 3x3 and an expansion rate of (m, n); then, in order to select a proper convolution kernel for each latitude and extract relevant features, a latitude adaptive selection mechanism is introduced.

[0041] Preferably, in the step (1), the latitude adaptive selection mechanism comprises:

[0042] First, the features extracted by different expansion convolutions are summed up:

[0043]

[0044] Then, the summed features F add are subjected to channel and height-based average pooling P avg (·)

[0045] and maximum pooling P​max (·), to enhance their representation ability:

[0046] L favg = P avg (F add ), L fmax = P max (F add ) (6)

[0047] where L favg and L fmax are the average-pooled and max-pooled feature descriptions along the channel and height directions, respectively; to realize information interaction between different feature descriptions and reduce complexity, the pooled features are concatenated and converted into low-dimensional features using a convolution with a 1×1 kernel size:

[0048] L fam = Conv1×1([L favg , L fmax ]) (7)

[0049] where [·] denotes the concatenation operation; subsequently, different convolutions with a 1×1 kernel size are used to extract multi-stream features to capture the dependency between different latitudes:

[0050]

[0051] where k is equal to 1, 2, 3, and 4, the multi-stream features are concatenated along the channel dimension, and then a Softmax operation is performed on the concatenated features in the channel dimension to obtain the attention weight of each feature at a specific latitude h, which is represented as:

[0052]

[0053] where represents the attention score of the kth feature at a certain latitude h, and the final feature is weighted by the corresponding adaptive latitude attention weight; then, the weighted feature passes through a convolution layer to smooth the features at different latitudes:

[0054]

[0055] Preferably, in step (2), the convolution kernel is used to explore multi-scale representations:

[0056] X3 = DConv 3×3 (X l ) (11)

[0057] X5 = DConv 5×5 (Xl ) (12)

[0058] X7=DConv 7×7 (X l ) (13)

[0059] Then, the multi-scale extracted feature representation is spliced and channel fusion is performed through convolution; finally, the GELU function is applied to the fused feature as a gating mechanism, and element-wise multiplication is performed with the original feature; this exploration and integration process is represented as follows:

[0060] X'=Conv 1x1 ([X3,X5,X7]) (14)

[0061] X''=GELU(X')·Conv 1x1 (X l ) (15)

[0062] Where [·] represents channel splicing.

[0063] Preferably, in step (3), using m as an external condition for spatial feature transformation, affine parameters a and b are generated for each element of the latent representation y, and the process is represented as follows:

[0064]

[0065] Where, is the feature before modulation.

[0066] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes the steps of the above-mentioned embodiment method, and the storage medium can be ROM / RAM, a magnetic disk, an optical disk, a memory card, etc. Therefore, corresponding to the method of the present application, the present application also simultaneously includes a 360° image compression device for adaptive latitude-aware transform coding. The device is usually represented in the form of a functional module corresponding to each step of the method. The device includes:

[0067] An adaptive latitude-aware module configured to first select four rectangular convolution kernels 5

[0068] of appropriate size to capture distortion redundancy in different latitude regions, use 3x3 convolution kernels with dilation rates of (2, 5), (1, 4), (2, 3) and (3, 3) to achieve the effect of rectangular convolution kernels, and use deep convolution for feature extraction;

[0069] The multi-scale gated convolutional neural network module is configured to first divide the normalized features into three equal-sized features according to channels, and then input them into a deep convolutional network in three parallel branches. The kernel sizes of these convolutions are 3×3, 5×5, and 7×7, respectively, to explore multi-scale representations. Then, the feature representations extracted from the multi-scale are concatenated and channel fusion is performed through convolution. Finally, the fused features are applied to the GELU function as a gating mechanism and multiplied element-wise with the original features.

[0070] The transform modulation module is configured to first use an importance feature activation network, which includes three residual blocks and a sigmoid activation function, to generate an importance feature activation map m.

[0071] Then, using m as an external condition for the spatial feature transformation, affine parameters α and β are generated for each element of the latent representation y, and y is quantized as... On the other hand, y is fed into the entropy model network to learn the latent representation. The probability distribution parameters μ and σ;

[0072] The overall framework for learning 360° image compression is configured to stabilize training by embedding layer normalization before each module. ALAB is progressively inserted into the analytical and synthetic transformations through convolutional downsampling and transpose upsampling. The analytical transformation gradually and adaptively perceives the distortion redundancy in different dimensions, while the synthetic transformation progressively promotes the generation of the original distorted ERP image.

[0073] The latent representation y is modeled as a Gaussian distribution. To accurately estimate the mean μ and variance σ of the y distribution, an entropy model composed of a prior and a context model is used. Based on analytical transform, synthetic transform, and the entropy model, a learned 360° image compression framework is constructed.

[0074] Used for end-to-end training.

[0075] Preferably, in the adaptive latitude sensing module,

[0076]

[0077]

[0078] Where X represents the input feature. This represents a depthwise convolution with a kernel size of 3×3 and an inflation rate of (m,n). Then, in order to select an appropriate kernel for each dimension and extract relevant features, a dimension adaptive selection mechanism is introduced.

[0079] Preferably, in the adaptive latitude sensing module, the latitude adaptive selection mechanism includes:

[0080] First, the features extracted by different dilated convolutions are summed:

[0081]

[0082] Then, these summed features F add Perform average pooling P based on channel and height avg (·)

[0083] and max pooling P max (·) to enhance their expressive power:

[0084] L favg =P avg (F add ), L fmax =P max (F add (6)

[0085] Among them, L favg and L fmax These are the feature descriptions after average pooling and max pooling along the channel and height directions, respectively. To enable information interaction between different feature descriptions and reduce complexity, the pooled features are concatenated and then convolved using a 1×1 kernel.

[0086] Convert these pooled features into low-dimensional features:

[0087] L fam =Conv1×1([L favg ,L fmax (7)

[0088] Where [·] denotes a concatenation operation; subsequently, N convolutional kernels of size 1×1 with different...

[0089] Convolution is used to extract multi-stream features to capture dependencies between different dimensions:

[0090]

[0091] Where k equals 1, 2, 3, and 4, the multi-stream features are concatenated along the channel dimension. Then, a softmax operation is performed on the concatenated features along the channel dimension to obtain each feature at a specific dimension h. Attention weights, this process is represented as:

[0092]

[0093] in, This represents the attention score of the k-th feature at a certain dimension h. The final feature is weighted by the corresponding adaptive dimension attention weights. Then, the weighted feature is passed through a convolutional layer to smooth the features across different dimensions.

[0094]

[0095] Preferably, in the multi-scale gated convolutional neural network module, the convolutional kernel is used to explore multi-scale representations:

[0096] X3=DConv 3×3 (X l (11)

[0097] X5 = DConv 5×5 (X l (12)

[0098] X7=DConv 7×7 (X l (13)

[0099] Then, the feature representations extracted from multiple scales are concatenated and channel fusion is performed through convolution; finally, the GELU function is applied as a gating mechanism to the fused features, and element-wise multiplied with the original features; this exploration and integration process is represented as follows:

[0100] X′=Conv 1x1 ([X3,X5,X7]) (14)

[0101] X″=GELU(X′)·Conv 1x1 (X l (15)

[0102] In this context, [·] indicates splicing by channel.

[0103] Preferably, in the transform modulation module, m is used as an external condition for the spatial feature transformation to generate affine parameters α and β for each element of the potential representation y. This process is represented as follows:

[0104]

[0105] in, These are characteristics before modulation.

[0106] The experimental methods of this invention are described in detail below.

[0107] 1. Experimental Data: A total of 19,790 high-quality 360° images were collected from the Flickr sharing website. This dataset is called the LIC360 dataset, with 19,590 images used as the training set and 200 images used as the test set. In addition, the ODI-SR and SUN360 datasets were also used, each containing 100 high-resolution images with a resolution of 1024×512, as the test set.

[0108] 2. Metrics: This invention uses Bits Per Pixel (BPP), Weighted-to-Spherically-uniform Peak Signal-to-Noise Ratio (WS-PSNR), and Weighted-to-Spherically-uniform Peak Signal-to-Noise Ratio (WS-SSIM) to compare the rate-distortion performance of this invention with different projection and learning methods. To quantitatively evaluate performance, the BD-rate of each method was calculated, and its WS-PSNR BPP curve was compared with a benchmark WS-PSNR BPP curve. The WS-PSNR BPP benchmark curve was set by VVC (VVC's BD-rate is 0), and the reference software for VVC is VTM-9.1.

[0109] 3. Experimental Setup: This invention uses the Compress AI framework to implement the designed 360° image compression model. Several different models were trained from scratch to represent different image quality levels, each corresponding to a specific lambda. Specifically, a lambda in the range {0.0018, 0.0035, 0.0067, 0.0130, 0.0250, 0.0483} was used to train the designed model to optimize WMSE. Unlike the learning optimization loss commonly used in planar image compression, MSE is unsuitable due to the spherical nature of 360° images. Therefore, WMSE was used as the optimization loss function. Similarly, WSSIM was also used as the optimization loss function, with a lambda range of {2.40, 4.58, 8.73, 16.64}. All models were trained using the Adam optimizer for 200 epochs with a batch size of 4. The initial learning rate was set to 1×10⁻⁶. -4 And after 180 cycles, it decreased to 1×10 -5 In all our models, the number of channels N and M are set to 128 and 320, respectively.

[0110] 4. Comparison with Different Compression Methods: The method of this invention was compared with the learning-based 360° image compression method proposed by Li22. Since the number of learning-based 360° image compression methods is limited, the method of this invention was also compared with several learning-based planar image compression methods, including Ballé18, Minnen18, and Cheng20. To highlight the difference between learning-based compression methods and traditional codecs, the traditional codec methods BPG and VVC were also included in the comparison.

[0111] As shown in Table 3, the method of this invention achieves BD-rate savings of 8.96%, 3.75%, and 3.16% compared to VVC on the LIC360, ODI-SR, and SUN360 datasets, respectively. As shown in Table 1, compared to the learning-based 360° image compression method proposed by Li22, which increases the BD-rate by 26.83% on the LIC360 dataset, the method of this invention clearly surpasses Li22 in terms of bitrate savings. Furthermore, compared to other methods relative to VVC, the method of this invention is the only one that achieves bitrate savings on the LIC360 dataset, demonstrating that this invention can effectively solve the distortion and redundancy problem of ERP images.

[0112] To fairly evaluate the performance of various methods, Table 1 also provides a comparison of encoding time, decoding time, number of parameters, GMACs, and BD-rate for different methods on the LIC360 dataset. The test machine was configured with an Intel i9-10850K CPU and an NVIDIA GTX 4090 GPU. Since traditional codecs using intra-frame prediction in VVC and BPG do not provide parameters and GMACs, this invention is similar to the Minnen18 method in terms of the number of parameters and GMACs, but has a significant advantage in bitrate savings. Compared to the learned 360° image compression method Li22, this invention reduces GMACs by nearly six times and halves the number of parameters. Because this invention, like Minnen18 and Cheng20, uses a serial autoregressive entropy model, the encoding and decoding times are relatively long. However, compared to the complex intra-frame prediction mode of VVC, the encoding time of this invention is relatively low. Li22 and Ballé18 do not employ this serial autoregressive entropy model, and therefore have shorter encoding and decoding times. This invention has relatively low GMACs and parameter count, and is the only method to achieve BD-rate savings, demonstrating the efficiency of the method.

[0113] Table 1 Comparison of different test results on the LIC360 dataset.

[0114] Method Encoding time (s) ↓ Decoding time (s) ↓ Parameter amount ↓ Computational amount ↓ BD-rate (%) ↓ VVC 129.752 0.057 - - 0.00 BPG 0.215 0.111 - - 21.13 Ballé 18 0.045 0.051 11.82 101.69 49.27 Minnen 18 3.364 6.660 25.50 124.06 25.64 Cheng 20 3.357 6.654 29.63 554.16 21.89 Li 22 0.460 0.613 52.63 772.84 26.83 The method of the present invention 3.326 6.596 24.08 125.43 -8.96

[0115] 5. Ablation Experiment: In order to verify the effectiveness of different modules of the present invention, ablation experiments were conducted on nonlinear transformation, ALAM, MCGFNN and m respectively.

[0116] Effectiveness of Nonlinear Transformation: The nonlinear transformation of this invention is compared with that of the Cheng20 method. For a fair comparison, the Cheng20 model was retrained using WMSE as the loss function. In this invention, important feature activation maps m are removed, and the entropy model of this invention is kept consistent with that of Cheng20. The results of the ablation experiments are shown in Table 2. It can be seen that this invention outperforms Cheng20 (WMSE) on the LIC360 and ODI-SR datasets. Although it performs slightly worse on the SUN360 dataset, as shown in Table 1, Cheng20 requires more GMACs and parameters compared to this invention. Typically, nonlinear transformations consist of upsampling / downsampling layers and attention layers. In terms of upsampling and downsampling, Cheng20 uses residual blocks, while the method of this invention uses 3×3 convolutional layers. Therefore, compared with Cheng20, this invention has simpler upsampling and downsampling layers, which further proves the effectiveness of the proposed ALAB in nonlinear transformations.

[0117] Effectiveness of ALAM: In these experiments, ALAM was replaced with a single convolutional layer with a kernel size of 3×3 and a dilation rate of (3,3). The results of the ablation experiments are shown in Table 3. Clearly, the absence of ALAM leads to a decrease in rate-distortion performance. ALAM achieved BD-rate savings of 4.93%, 6.01%, and 8.51% on the LIC360, ODI-SR, and SUN360 datasets, respectively, demonstrating its effectiveness in 360° image compression.

[0118] Effectiveness of MCGFNN: This invention also evaluates the effectiveness of the proposed MCGFNN. Replacing MCGFNN with FFN is shown in Table 3. It can be observed that MCGFNN improves rate-distortion performance, achieving BD-rate savings of 1.86%, 3.17%, and 3.14% on the LIC360, ODI-SR, and SUN360 datasets, respectively. FFN, however, fails to utilize multi-scale knowledge simultaneously. In contrast, the MCGFNN of this invention explores and incorporates multi-scale cues during the pre-propagation process and suppresses redundant features generated by the fusion of multiple rectangular dilated convolutions, thereby significantly improving performance.

[0119] Effectiveness of m: Ablation experiments were conducted to verify the effectiveness of m. The results of the ablation experiments are shown in Table 3. As can be seen from Table 3, the absence of m leads to a decrease in rate-distortion performance. With the addition of m, BD-rate savings of 0.75%, 0.67%, and 1.51% were achieved on the LIC360, ODI-SR, and SUN360 datasets, respectively.

[0120] Table 2 Nonlinear transformation ablation experiments on the LIC360, ODI-SR, and SUN360 datasets.

[0121]

[0122]

[0123] Table 3 Ablation experiments on different modules of the LIC360, ODI-SR, and SUN360 datasets.

[0124]

[0125] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A 360° image compression method using adaptive latitude-aware transform coding, characterized in that: The method includes the following steps: (1) Design of the adaptive latitude perception module ALAM: First, select four rectangular convolution kernels of appropriate size 5×11, 3×9, 5×7 and 7×7 to capture the distortion redundancy of different latitude regions. Use 3×3 convolution kernels with dilation rates of (2,5), (1,4), (2,3) and (3,3) to achieve the effect of rectangular convolution kernels. Use depthwise convolution for feature extraction. (2) Constructing a multi-scale gated convolutional neural network MCGFNN: First, the normalized features are divided into three equal-sized features according to the channels. Then, they are input into the deep convolutional networks in three parallel branches. The kernel sizes of these convolutions are 3×3, 5×5, and 7×7, respectively, to explore multi-scale representations. Then, the feature representations extracted from the multi-scale are concatenated and channel fusion is performed through convolution. Finally, the GELU function is applied to the fused features as a gating mechanism and element-wise multiplied with the original features. (3) Spatial features guided by the transformation modulation importance feature activation map: First, an importance feature activation network, consisting of three residual blocks and a sigmoid activation function, is used to generate an importance feature activation map m; then, m is used as an external condition for the spatial feature transformation to generate affine parameters α and β for each element of the latent representation y, and y is quantized as... On the other hand, y is fed into the entropy model network to learn the latent representation. The probability distribution parameters μ and σ; (4) Constructing a learning-based 360° image compression framework: Integrating ALAM designed in step (1) and MCGFNN designed in step (2) into an adaptive dimensionality-aware block ALAB; To ensure training stability, layer normalization is embedded before each module. ALAB is progressively inserted into the analytical transformation and synthetic transformation through convolutional downsampling and transpose upsampling. The analytical transformation gradually and adaptively perceives the distortion redundancy of different dimensions, while the synthetic transformation gradually promotes the generation of the original distorted ERP image; The latent representation y is modeled as a Gaussian distribution. In order to estimate the mean μ and variance σ of the y distribution well, an entropy model composed of the prior and context model is used for estimation; Based on the analytical transformation, synthetic transformation and entropy model, a learning-based 360° image compression framework is constructed for end-to-end training.

2. The 360° image compression method with adaptive latitude-aware transform coding according to claim 1, characterized in that: In step (1), Where X represents the input feature. This represents a depthwise convolution with a kernel size of 3×3 and an inflation rate of (m,n). Then, in order to select an appropriate kernel for each dimension and extract relevant features, a dimension adaptive selection mechanism is introduced.

3. The 360° image compression method with adaptive latitude-aware transform coding according to claim 2, characterized in that: In step (1), the latitude adaptive selection mechanism includes: First, the features extracted by different dilated convolutions are summed: Then, these summed features F add Perform average pooling P based on channel and height avg (·) and maximum pooling P max (·) to enhance their expressive power: L favg =P avg (F add ), L fmax =P max (F add ) (6) Among them, L favg and L fmax These are the feature descriptions after average pooling and max pooling along the channel and height directions, respectively. To enable information exchange between different feature descriptions and reduce complexity, the pooled features are concatenated, and a 1×1 convolution kernel is used to transform these pooled features into low-dimensional features. L fam <Conv1×1([L favg ,L fmax ]) (7) Where [·] denotes a concatenation operation; subsequently, N different convolutions with kernel size 1×1 are used to extract multi-stream features to capture dependencies between different dimensions: Where k equals 1, 2, 3, and 4, the multi-stream features are concatenated along the channel dimension. Then, a softmax operation is performed on the concatenated features along the channel dimension to obtain each feature at a specific dimension h. Attention weights, this process is represented as: in, This represents the attention score of the k-th feature at a certain dimension h. The final feature is weighted by the corresponding adaptive dimension attention weights. Then, the weighted feature is passed through a convolutional layer to smooth the features across different dimensions.

4. The 360° image compression method with adaptive latitude-aware transform coding according to claim 3, characterized in that: In step (2), the convolution kernel is used to explore multi-scale representations: X3=DConv 3×3 (X l ) (11) X5 = DConv 5×5 (X l ) (12) X7=DConv 7×7 (X l ) (13) Then, the feature representations extracted from multiple scales are concatenated and channel fusion is performed through convolution; finally, the GELU function is applied as a gating mechanism to the fused features, and element-wise multiplied with the original features; this exploration and integration process is represented as follows: X′=Conv 1x1 ([X3,X5,X7]) (14) X″=GELU(X′)·Conv 1x1 (X l ) (15) In this context, [·] indicates splicing by channel.

5. The 360° image compression method with adaptive latitude-aware transform coding according to claim 3, characterized in that: In step (3), m is used as an external condition for spatial feature transformation to generate affine parameters α and β for each element of the potential representation y. This process is represented as follows: in, These are characteristics before modulation.

6. A 360° image compression device with adaptive latitude-sensing transform coding, characterized in that: The device includes: The adaptive latitude perception module is configured to first select four rectangular convolution kernels of appropriate sizes, 5×11, 3×9, 5×7, and 7×7, to capture the distortion redundancy of different latitude regions. It uses 3×3 convolution kernels with dilation rates of (2,5), (1,4), (2,3), and (3,3) to achieve the effect of rectangular convolution kernels, and uses depthwise convolution for feature extraction. The multi-scale gated convolutional neural network module is configured to first divide the normalized features into three equal-sized features according to channels, and then input them into a deep convolutional network in three parallel branches. The kernel sizes of these convolutions are 3×3, 5×5, and 7×7, respectively, to explore multi-scale representations. Then, the feature representations extracted from the multi-scale are concatenated and channel fusion is performed through convolution. Finally, the fused features are applied to the GELU function as a gating mechanism and multiplied element-wise with the original features. The transform modulation module is configured to first use an importance feature activation network, comprising three residual blocks and a sigmoid activation function, to generate an importance feature activation map m; then, using m as an external condition for the spatial feature transformation, affine parameters α and β are generated for each element of the latent representation y, which is quantized as... On the other hand, y is fed into the entropy model network to learn the latent representation. The probability distribution parameters μ and σ; The learned 360° image compression framework is configured to stabilize training by embedding layer normalization before each module. ALAB is progressively inserted into the analytic and synthetic transformations through convolutional downsampling and transpose upsampling. The analytic transformation gradually and adaptively perceives distortion redundancy at different dimensions, while the synthetic transformation progressively facilitates the generation of the original distorted ERP image. The latent representation y is modeled as a Gaussian distribution. To accurately estimate the mean μ and variance σ of the y distribution, an entropy model composed of a prior and a context model is used. Based on the analytic transformation, synthetic transformation, and entropy model, the learned 360° image compression framework is constructed for end-to-end training.

7. The 360° image compression device with adaptive latitude-sensing transform coding according to claim 6, characterized in that: In the adaptive latitude sensing module Where X represents the input feature. This represents a depthwise convolution with a kernel size of 3×3 and an inflation rate of (m,n). Then, in order to select an appropriate kernel for each dimension and extract relevant features, a dimension adaptive selection mechanism is introduced.

8. The 360° image compression device with adaptive latitude-sensing transform coding according to claim 7, characterized in that: The adaptive latitude sensing module includes a latitude adaptive selection mechanism: First, the features extracted by different dilated convolutions are summed: Then, these summed features F add Perform average pooling P based on channel and height avg (·) and maximum pooling P max (·) to enhance their expressive power: L favg =P avg (F add ), L fmax =P max (F add ) (6) Among them, L favg and L fmax These are the feature descriptions after average pooling and max pooling along the channel and height directions, respectively. To enable information exchange between different feature descriptions and reduce complexity, the pooled features are concatenated, and a 1×1 convolution kernel is used to transform these pooled features into low-dimensional features. L fam <Conv1×1([L favg ,L fmax ]) (7) Where [·] denotes a concatenation operation; subsequently, N different convolutions with kernel size 1×1 are used to extract multi-stream features to capture dependencies between different dimensions: Where k equals 1, 2, 3, and 4, the multi-stream features are concatenated along the channel dimension. Then, a softmax operation is performed on the concatenated features along the channel dimension to obtain each feature at a specific dimension h. Attention weights, this process is represented as: in, This represents the attention score of the k-th feature at a certain dimension h. The final feature is weighted by the corresponding adaptive dimension attention weights. Then, the weighted feature is passed through a convolutional layer to smooth the features across different dimensions.

9. The 360° image compression device with adaptive latitude-sensing transform coding according to claim 8, characterized in that: In the multi-scale gated convolutional neural network module, the convolutional kernel is used to explore multi-scale representations: X3=DConv 3×3 (X l ) (11) X5 = DConv 5×5 (X l ) (12) X7=DConv 7×7 (X l ) (13) Then, the feature representations extracted from multiple scales are concatenated and channel fusion is performed through convolution; finally, the GELU function is applied as a gating mechanism to the fused features, and element-wise multiplied with the original features; this exploration and integration process is represented as follows: X′=Conv 1x1 ([X3,X5,X7]) (14) X″=GELU(X′)·Conv 1x1 (X l ) (15) In this context, [·] indicates splicing by channel.

10. The 360° image compression device with adaptive latitude-sensing transform coding according to claim 9, characterized in that: In the transform modulation module, m is used as an external condition for spatial feature transformation to generate affine parameters α and β for each element of the potential representation y. This process is represented as follows: in, These are characteristics before modulation.

Images