A multi-scale based end-to-end 360-degree image compression method

Abstract

The present application relates to the technical field of image coding, in particular to an end-to-end 360-degree image compression method based on multi-scale, comprising constructing and training an image compression model, the image compression model comprising three layers of paths, each layer of path comprising an encoding module, an adaptive scale block encoding module and a decoding module; and realizing end-to-end 360-degree image compression through the trained image compression model; the present application can improve compression efficiency and reduce bit rate redundancy caused by overstretching.

Description

Technical Field

[0001] This invention relates to the field of image coding technology, and more specifically to an end-to-end 360-degree image compression method based on multiple scales. Background Technology

[0002] 360° images, also known as panoramic images, are a new type of media that can record a scene from all angles, allowing users to explore the scene from any perspective. Currently, with the rapid development of virtual reality (VR) devices and the widespread use of mobile phones capable of capturing panoramic images, the market size for panoramic images has expanded significantly. Compared to 2D images, panoramic images can encompass more content, thus requiring more storage space and transmission bandwidth. This presents a significant challenge to the ever-increasing storage and transmission demands of panoramic images. To adapt to the future development of VR devices, a novel image compression method suitable for panoramic images must be proposed.

[0003] Currently, 360° images need to be converted into planar images with a specific projection using projection methods, such as Equidistant Rectangular Projection (ERP), and then saved using existing codecs. However, there are some problems with using the ERP method to project a sphere onto a rectangle to obtain ERP images. First, objects are distorted, making the distortion on the ERP image inconsistent with what people see on VR or 360° devices. Second, ERP images also suffer from unbalanced sampling; currently, regions of different latitudes are represented by the same number of pixels, i.e., the width of the ERP image, so objects in high-latitude regions are oversampled. Different techniques have been investigated to overcome these problems, such as introducing region downsampling and reprojection methods to reduce bits in oversampled regions. However, these methods either decompose the ERP image into tiles, downsample them, and then merge the tiles into a new image; or map the sphere onto a polyhedron and merge all surfaces into a new planar image. In both cases, the new image destroys the local smoothness of the natural image and creates artificially sharp edges between different tiles or surfaces.

[0004] With the rapid development of end-to-end image generation and entropy coding, there is a growing desire to move beyond pixel-based encoding in traditional image compression methods and utilize feature encoding based on deep neural networks. Deep neural networks have demonstrated more reliable global smoothness and higher compression ratios in image coding than traditional methods. Some researchers utilize autoencoder-like architectures to learn nonlinear transformations and combine them with quantization to optimize the entire process end-to-end, achieving higher compression efficiency than traditional hand-crafted methods. Others have shown that combining autoregression and hierarchical priors can better utilize the probabilistic structure of latent representations. While these end-to-end models perform well for 2D images, they perform poorly for 360° images because they cannot adequately address the overstretching problem in ERP images. Recent research has explored deep neural network models for 360° image compression. Some researchers have introduced an adaptive 360° image patch representation, optimizing the size of each patch to address the overstretching issue. Still others have proposed the first end-to-end optimized 360° image compression model, which uses an adaptive latitude map to mask features, thereby reducing rate loss. While their work achieved state-of-the-art performance in end-to-end 360° image compression, they did not take into account that different regions of a 360° image should be processed separately.

[0005] In general, current 360° image compression has two problems: (1) There is a lack of effective optimization methods for intermediate features of 360° images. (2) There is no very effective deep neural network model for the overstretching characteristics of 360° images at both extremes. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides an end-to-end 360-degree image compression method based on multiple scales. The method is characterized by constructing and training an image compression model, which includes three layers, each comprising an encoding module, an adaptive scale block encoding module, and a decoding module. End-to-end 360-degree image compression is achieved through the trained image compression model, comprising the following steps:

[0007] S1. Perform ERP projection on the acquired 360-degree image to form a 2D projection image, and divide the 2D projection image into 3 groups of images;

[0008] S2. Input the i=1,2,3 group of images into the encoding module in the i-th layer path to obtain a set of encoded features;

[0009] S3. Input the i-th group of encoded features into the adaptive scale block coding module in the i-th layer path to obtain a set of optimal block pattern diagrams. Each set of optimal block pattern diagrams includes multiple blocks. In the first set of optimal block pattern diagrams, all blocks have a size of 1×8, in the second set of optimal block pattern diagrams, all blocks have a size of 2×4, and in the third set of optimal block pattern diagrams, all blocks have a size of 4×4.

[0010] S4. Pass the i-th group of images and the i-th group of encoded features through an average pooling layer to obtain a set of similar merged features. Quantize and upsample each set of similar merged features to obtain a set of quantized features.

[0011] S5. Input the i-th group of quantized features into the decoding module in the i-th layer path, and merge the outputs of the decoding modules of the three layers path to obtain the 360-degree image compression result.

[0012] The beneficial effects of this invention are:

[0013] Projecting 360-degree images onto an ERP system results in overstretching, increasing storage requirements without a corresponding improvement in visual quality. To improve compression efficiency, reducing the bitrate redundancy caused by this overstretching is crucial. Therefore, this invention proposes a multi-scale, end-to-end 360-degree image compression method.

[0014] First, to accommodate different overstretch rates in different regions, this invention introduces multi-scale rectangular convolutional kernels to capture rectangular receptive fields and reduce overfitting based on the overstretch rate. Second, an adaptive scale block method is proposed, which simultaneously considers the overstretch rate and texture complexity of the 360° image, effectively reducing bitrate redundancy in overstretched and less important regions. Simultaneously, a novel end-to-end compression framework specifically for 360° images is proposed. Experimental results show that, compared to state-of-the-art 360° image compression models, our model outperforms all other image compression methods and reduces the bitrate by 15.76%. Attached Figure Description

[0015] Figure 1 This is a flowchart of the method of the present invention;

[0016] Figure 2 This is a schematic diagram of excessive stretching of the ERP image in this invention;

[0017] Figure 3 This is a schematic diagram of the image compression model structure of the present invention;

[0018] Figure 4 This is a detailed structural diagram of each module of the model of the present invention;

[0019] Figure 5 Rate-distortion curves for various kernel sizes in Group 1 of this invention;

[0020] Figure 6 This is a specific implementation of the adaptive scale block method of the present invention;

[0021] Figure 7 The figure shows the simulation results of this invention;

[0022] Figure 8 This is a schematic diagram of model training in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Currently, although some scholars have proposed the first end-to-end optimized 360° image compression model, their models are still designed based on the characteristics of traditional planar images. Their specific schemes require mask-based adaptive reduction of intermediate features to lower the bitrate of these features, which can be understood as feature reduction. However, in the projected image generated from a 360° image, the degree of overstretching varies between different regions. These methods do not consider the independence between different regions but process the entire image uniformly. This cannot efficiently balance high-latitude and low-latitude regions. Furthermore, feature reduction creates gaps in feature information. Although this method uses latitude-adaptive reduction, the oversampling of the 360° image is due to projection stretching, meaning a small amount of information is stretched to fill in the gaps. Therefore, we believe that simple feature reduction is not the optimal solution, and the equatorial region of the 360° image is not stretched and should not be reduced.

[0025] Furthermore, current networks for 360° image compression all employ network structures designed for planar images. Without proposing a model equipped with a specialized deep neural network, it is difficult to fundamentally solve the oversampling problem of 360° images.

[0026] Based on the above, this invention provides a multi-scale end-to-end 360-degree image compression method, including constructing and training an image compression model. The image compression model comprises three layers, each including an encoding module, an adaptive scale block encoding module, and a decoding module. End-to-end 360-degree image compression is achieved through the trained image compression model. Figure 1 As shown, it includes the following steps:

[0027] S1. Perform ERP projection on the acquired 306-degree image to form a 2D projection image, and divide the 2D projection image into 3 groups of images.

[0028] Specifically, 360° images are typically represented as a sphere indexed by longitude and latitude. For ease of compression, the sphere must first be projected onto a 2D image. For example... Figure 2 As shown, regions A and B on the sphere have different sizes, but after projection mapping, they appear as regions of equal size in the ERP image. This results in horizontal pixel overstretching between the sphere and the ERP image content, a problem known as horizontal overstretching. Specifically, the closer a region is to the pole, the smaller its area on the sphere, and the more pronounced the stretching in the corresponding ERP image. Pixel stretching varies by latitude. To quantify the degree of this distortion, we introduce an overstretching factor.

[0029] Dividing the image into too many regions makes the model overly complex, while dividing it into too few regions reduces its efficiency. This invention divides the ERP image into 16 regions based on its stretching distribution and height. Encoding each region separately would complicate the process; therefore, we calculate the overstretch factor value for each region and merge regions with similar average overstretch ratios to form three sets of images. Based on this, we divide our model into three layers according to the divided regions. For each layer, the model uses convolutional kernels and pooling blocks of different scales for processing.

[0030] Table 1 Classification Criteria

[0031]

[0032] Specifically, as shown in Table 1, the first column represents the row number of the region image in its corresponding 2D projection image, the second column represents the overstretch factor value of the region image, the third column is the average overstretch ratio of the region image, and the fourth column is the group to which the region image belongs. Step S1 divides the 2D projection image into 3 groups of images according to the overstretch factor, including:

[0033] The 2D projected image is divided into 16 regions based on height; the overstretch factor value is calculated for each region using its center pixel using the following formula:

[0034]

[0035]

[0036] Where Lat(i) represents the overstretch factor value of the i-th region image, Indicates the scale parameter weights. represents the latitude value of the i-th region image, h represents the height of the 2D projected image, and N represents the number of region images; the average overstretch ratio of each region image is calculated using the overstretch factor value.

[0037] The first image, the second image, the fifteenth image, and the sixteenth image constitute the first group of images;

[0038] The third, fourth, thirteenth, and fourteenth region images constitute the second group of images;

[0039] The images of the 5th, 6th, 7th, 8th, 9th, 10th, 11th, and 12th regions constitute the third group of images.

[0040] S2. Input the i=1,2,3 group of images into the encoding module in the i-th layer path to obtain a set of encoded features.

[0041] S3. Input the i-th group of coding features into the adaptive scale block coding module in the i-th layer path to obtain a set of optimal block pattern diagrams. Each set of optimal block pattern diagrams includes multiple blocks. The size of all blocks in the first group of coding features is 1×8, the size of all blocks in the second group of coding features is 2×4, and the size of all blocks in the third group of coding features is 4×4.

[0042] S4. The i-th group of encoded features and the i-th group of best block pattern graphs are passed through an average pooling layer to obtain a set of similar merged features. Each set of similar merged features is quantized and upsampled to obtain a set of quantized features.

[0043] S5. Input the i-th group of quantized features into the decoding module in the i-th layer path, and merge the outputs of the decoding modules of the three layers path to obtain the 360-degree image compression result.

[0044] In one embodiment, such as Figure 3 As shown, the image compression model consists of three layers, each of which includes an encoding module, an adaptive scale block encoding module, and a decoding module.

[0045] Preferably, the encoding module adopts the design of the current mainstream end-to-end image compression model, using convolutional layers for feature extraction, and residual blocks (RBs) and attention modules to enhance the network performance. Because ERP images (i.e., 2D projected images) suffer from overstretching, square convolutional kernels are difficult to effectively extract features from overstretched regions; simultaneously, square convolutional kernels are prone to overfitting. Therefore, this invention proposes using rectangular convolutional kernels as an effective alternative.

[0046] like Figure 4As shown, the encoding modules of the three-layer path have the same structure, each including 4 rectangular convolutional layers, 3 residual blocks and 2 attention modules; wherein, a residual block is connected between every two adjacent rectangular convolutional layers, and an attention module is connected after the second residual block and the fourth rectangular convolutional layer respectively.

[0047] Specifically, most end-to-end networks use 5×5 convolutional kernels to balance computational complexity and receptive field size; however, this invention proposes using convolutional kernels with a larger horizontal dimension relative to their height. Since the average overstretch ratio varies significantly among the three sets of images, different rectangular convolutional kernels need to be designed for each set.

[0048] In this embodiment, five rectangular convolution kernels were designed specifically for the first group of images, balancing the trade-off between overstretching and time efficiency. To evaluate distortion, this invention uses WS-PSNR as the metric. Figure 5 Rate-distortion curves for five different rectangular convolutional kernels are presented. Observation reveals that the 5×7 rectangular convolutional kernel performs the worst in the first layer path compared to other rectangular kernels. In contrast, the 5×9, 5×11, and 5×13 rectangular convolutional kernels exhibit similar performance. However, larger rectangular convolutional kernels result in longer training and inference times. In particular, the 5×11 and 5×13 rectangular convolutional kernels require more processing time but offer performance comparable to the 5×9 rectangular convolutional kernel. Considering the time consumption, we chose the 5×9 rectangular convolutional kernel for the first group of images, or the first layer path. The same testing method was applied to the other groups. Finally, the image compression model uses rectangular convolutional kernels of three scales to match the three groups of images. The first group of images is the most severely overstretched, so a wider 5×9 rectangular convolution kernel is used in the first layer path for processing the first group of images. For the second group of images, which is moderately overstretched, a 5×7 rectangular convolution kernel is used in the second layer path for processing the second group of images. Since the average overstretch rate of the third group of images is close to 1, a 5×5 rectangular convolution kernel is used in the third layer path for processing the third group of images.

[0049] Preferably, such as Figure 3 As shown, the adaptive scale block coding modules of the three-layer path have the same structure, each including two adaptive scale block units; each adaptive scale block unit includes a first convolutional layer, a residual block, a rectangular convolutional layer, a second convolutional layer, a BN layer and a Gumbel softmax layer cascaded in sequence.

[0050] Specifically, due to the overstretching problem, overstretched pixels are often very similar, so merging is used to reduce bitrate costs. As shown in Table 1, the average overstretch ratio of the first group of images is approximately 1:7.2, the average overstretch ratio of the second group of images is approximately 1:2, and the average overstretch ratio of the third group of images is approximately 1:1.3. To match these average overstretch ratios, this invention designs blocks of three sizes. As shown in Figure 6(a), for the first layer with an average overstretch ratio of 1:7.2, this invention designs a 1×8 horizontal block to approximate this ratio; for the second layer, a 2×4 block is used; and for the third layer, a 4×4 block is used to match the ratio. The purpose of this is to adaptively compress the bitrate of unimportant regions. The block size determines the pooling size of each group of encoded features.

[0051] Not only overstretching, but also texture complexity is related to block design. To accommodate different texture complexities, this invention divides blocks of each size into different scales, such as... Figure 6 As shown in (a). Specifically, in Figure 6(a), block 1 in layer 1 is suitable for features with complex textures to preserve detailed information, block 2 or block 3 is more suitable for simpler or overstretched textures, and the remaining blocks are designed for medium texture complexity. After a set of encoded features of each image is processed by the corresponding adaptive scale block encoding module, the neural network selects the optimal pooling mode for each encoded feature in each set of encoded features, that is, selects the optimal pooling size and pooling scale for each encoded feature to form a block of optimal size mode, thus obtaining a set of optimal block mode maps.

[0052] exist Figure 6 (b) uses one of the coded features from group 3 as an example. First, an optimal-size block is generated from the coded feature using an adaptive scaling block coding module. Then, the coded feature and its optimal-size blocks are passed through an average pooling layer to merge similar values. Quantization is then performed to obtain quantized values, which are upsampled to generate the final quantized feature, and then transmitted to the decoding module. We observe that similar values ​​in the upper right corner of the coded feature are merged into one value, while different values ​​in the lower right corner are retained, thus achieving adaptive compression rate.

[0053] Preferably, the decoding modules of the three-layer path have the same structure, each including 4 transposed rectangular convolutional layers, 3 residual blocks, and 2 attention modules; wherein, a residual block is connected between every two adjacent transposed rectangular convolutional layers, and an attention module is connected before the second residual block and the first transposed rectangular convolutional layer respectively; in the first layer path, all transposed rectangular convolutional layers use a rectangular convolutional kernel size of 5×9; in the second layer path, all transposed rectangular convolutional layers use a rectangular convolutional kernel size of 5×7; in the third layer path, all transposed rectangular convolutional layers use a rectangular convolutional kernel size of 5×5.

[0054] Preferably, rate-distortion is a widely used metric for evaluating the coding performance of a model. During the training process of the image compression model, training aims to minimize the rate-distortion loss; the rate-distortion loss is calculated using the following formula:

[0055]

[0056]

[0057]

[0058] Where Loss represents rate distortion loss, R i This represents the quantization features used to encode the corresponding i-th group of images. The number of digits, D i X represents the distortion value of the i-th image group, λ represents the hyperparameter, pooling(·) represents the average pooling operation, and P(·) represents the Gaussian mixture distribution model; i Represents the i-th group of images, This represents the output obtained by the image compression model for the i-th group of images. This indicates the operation for calculating the mean square error. ω i This represents the pixel group weight of the i-th image group. This pixel group weight is obtained by summing the position weights of all pixels in that image group. The formula for calculating the position weight of each pixel is as follows:

[0059]

[0060] ω(k,l) represents the position weight of pixel (k,l) in the 2D projected image, and h represents the height of the 2D projected image.

[0061] Specifically, this invention uses a training dataset designed specifically for end-to-end 360° image compression during the training phase. The training dataset consists of 18,160 high-quality ERP images, each with a resolution of 512×1024 pixels. The images in the training dataset include panoramic natural images and panoramic indoor images, featuring rich textures and natural structures that closely reflect real-world photographic environments.

[0062] Simultaneously, this invention selects 200 high-quality ERP images from the training dataset to form a test dataset. To evaluate the objective quality of 360° images in the observation space, this invention uses weighted, spherically uniform quality-perceived methods, including WS-PSNR and WS-SSIM. These methods aim to provide accurate assessments of image quality.

[0063] This invention trains the image compression model in two stages, with an additional entropy estimation network added during the model training process, such as... Figure 8 As shown. In the first stage, the model was trained 24 times with a learning rate ξ=0.001; in the second stage, the learning rate was reduced to ξ=0.0001, and the model was trained 12 more times until it stabilized. The encoder, decoder, and entropy estimation network were all trained together using the same learning rate. During training, the batch size was set to 4. The Adam optimizer, which is based on PyTorch and supports CUDA, was used. WS-MSE was used as the distortion loss for WS-PSNR. To fine-tune the model in the second stage, WS-MSE and WS-SSIM were used as distortion losses, and the model was trained 12 more times.

[0064] In one embodiment, to evaluate the effectiveness of the proposed method, the method is compared with state-of-the-art learning-based models, including Li360, Balle18, Minnen18, and traditional compression standards such as VVC intra, BPG, and JPEG.

[0065] like Figure 7 As shown, the method of this invention significantly outperforms all other learning-based methods in terms of WS PSNR and WS-SSIM. Notably, Li360 represents the latest advancements in 360° image compression. Our method achieves a 0.5 dB higher WSPSNR than Li360 at 0.16 bpp, and we achieve a 15.76% average bitrate reduction compared to Li360 on the same dataset. Furthermore, VVC Intra is a well-known video compression standard that incorporates various handcrafted modules; the results show that our model outperforms VVC Intra in terms of performance.

[0066] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "setting," "connection," "fixing," "rotation," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Unless otherwise explicitly limited, those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0067] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-scale, end-to-end 360-degree image compression method, characterized in that, An image compression model is constructed and trained. This model comprises three layers, each including an encoding module, an adaptive scale block encoding module, and a decoding module. The encoding modules in all three layers have identical structures, each consisting of four rectangular convolutional layers, three residual blocks, and two attention modules. A residual block is connected between every two adjacent rectangular convolutional layers, and an attention module is connected after the second and fourth rectangular convolutional layers, respectively. End-to-end 360-degree image compression is achieved using the trained image compression model, including the following steps: S1. Perform ERP projection on the acquired 360-degree image to form a 2D projection image, and divide the 2D projection image into 3 groups of images according to the overstretch factor; Formula for calculating the overstretch factor: Where Lat(i) represents the overstretch factor value of the i-th region image, Indicates the scale parameter weights. Represents the latitude value of the i-th region image, and N represents the number of regions to be divided; S2. Input the i=1,2,3 group of images into the encoding module in the i-th layer path to obtain a set of encoded features; S3. Input the i-th group of encoded features into the adaptive scale block coding module in the i-th layer path to obtain a set of optimal block pattern diagrams. Each set of optimal block pattern diagrams includes multiple blocks. In the first set of optimal block pattern diagrams, all blocks have a size of 1×8, in the second set of optimal block pattern diagrams, all blocks have a size of 2×4, and in the third set of optimal block pattern diagrams, all blocks have a size of 4×4. S4. Pass the i-th group of encoded features and the i-th group of best block pattern graphs through an average pooling layer to obtain a set of similar merged features. Quantize and upsample each set of similar merged features to obtain a set of quantized features. S5. Input the i-th group of quantized features into the decoding module in the i-th layer path, and merge the outputs of the decoding modules of the three layers path to obtain the 360-degree image compression result.

2. The end-to-end 360-degree image compression method based on multiple scales according to claim 1, characterized in that, Step S1 divides the 2D projected image into 3 groups of images based on the overstretch factor, including: The 2D projected image is divided into 16 regions based on its height. The first image, the second image, the fifteenth image, and the sixteenth image constitute the first group of images; The third, fourth, thirteenth, and fourteenth region images constitute the second group of images; The images of the 5th, 6th, 7th, 8th, 9th, 10th, 11th, and 12th regions constitute the third group of images.

3. The end-to-end 360-degree image compression method based on multi-scale as described in claim 1, characterized in that, The three-layer path adaptive scaling block coding modules have the same structure, each including two adaptive scaling block units; each adaptive scaling block unit includes a first convolutional layer, a residual block, a rectangular convolutional layer, a second convolutional layer, a BN layer, and a Gumbel softmax layer cascaded in sequence.

4. A multi-scale, end-to-end 360-degree image compression method according to any one of claims 1 or 3, characterized in that, In the first layer, all rectangular convolutional layers use a rectangular kernel size of 5×9; in the second layer, all rectangular convolutional layers use a rectangular kernel size of 5×7; and in the third layer, all rectangular convolutional layers use a rectangular kernel size of 5×5.

5. The end-to-end 360-degree image compression method based on multi-scale according to claim 1, characterized in that, The decoding modules of the three-layer path have the same structure, each including 4 transposed rectangular convolutional layers, 3 residual blocks, and 2 attention modules. Each pair of adjacent transposed rectangular convolutional layers is connected by a residual block, and an attention module is connected before the second residual block and the first transposed rectangular convolutional layer, respectively. In the first layer, all transposed rectangular convolutional layers use a rectangular convolutional kernel size of 5×9. In the second layer, all transposed rectangular convolutional layers use a rectangular convolutional kernel size of 5×7. In the third layer, all transposed rectangular convolutional layers use a rectangular convolutional kernel size of 5×5.

6. The end-to-end 360-degree image compression method based on multi-scale according to claim 1, characterized in that, During the training of the image compression model, the goal is to minimize the rate-distortion loss; the rate-distortion loss is calculated using the following formula: Where Loss represents rate distortion loss, R i This represents the quantization features used to encode the corresponding i-th group of images. The number of digits, D i Let λ represent the distortion value of the i-th image group, λ represent the hyperparameter, pooling(·) represent the average pooling operation, P(·) represent the Gaussian mixture distribution model, and ω represent the distortion value of the i-th image group. i This represents the pixel group weight of the i-th image group. Represents the i-th group of images, This represents the output obtained by the image compression model for the i-th group of images. This indicates the operation for calculating the mean square error.

Images