Spherical image compression method and device based on spherical deep neural network
Through the spherical image compression method based on spherical deep neural network, the problem of low spherical image compression efficiency is solved, efficient spherical image compression and bit rate optimization are achieved, and image quality evaluation indexes are improved.
Patent Information
- Application Number
- CN202411057952.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-02
AI Technical Summary
The existing spherical image compression method is inefficient, resulting in inefficient rate distortion optimization process, and the inefficient sampling and structural damage of spherical images cannot be effectively processed.
Using spherical image compression method based on spherical deep neural network, uniform sampling is performed through resampling of spherical measurements, spherical convolution kernel and spherical transformer model of twelve neighborhoods are designed, and the bitrate model is combined with spherical entropy model is used to realize the image Highly efficient compression.
This improves the efficiency of spherical image compression, reduces the bit rate, and improves image quality evaluation indicators such as SPSNR and WSPSNR, saving bit rate compared to traditional methods.
Smart Images

Figure CN119071494A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image compression, and in particular to a spherical image compression method based on a spherical deep neural network, and a spherical image compression device based on a spherical deep neural network. Background Art
[0002] Spherical images are generated by stitching multiple views, also known as panoramic images or 360-degree images. Spherical images provide a 360-degree view around a specific scene, and viewers can observe it from any angle. Virtual reality (VR) technology provides consumers with a unique immersive experience through head-mounted display devices. With the vigorous development of virtual reality technology, spherical images, as the main medium for carrying VR content, have received increasing attention. However, compared with flat images, obtaining ultra-high-definition panoramic images requires higher resolution (e.g., 4K, 8K or even higher resolution), which further exacerbates the challenges of storage and transmission. Therefore, it has become urgent to propose a new spherical image compression method.
[0003] For spherical image compression, the spherical image is usually projected into a 2D image before encoding, and then the projected 2D image is directly compressed using existing compression standards. Different projection methods have been studied, such as equirectangular projection (ERP), cubic projection, adjusted cubic projection, and segmented sphere projection. However, since the spherical image is on a sphere, the projection from the sphere to a 2D plane inevitably leads to non-uniform sampling or structural damage of the sphere. Therefore, no matter what projection method is used, the projection will introduce non-uniform sampling on the sphere, which will lead to correlation destruction and make the rate-distortion optimization process in compression inefficient.
[0004] In recent years, neural image compression has made significant progress, mainly due to improvements in the transformation, quantization, and entropy modeling stages of the compression process. The transformations are mainly based on convolutional neural networks and transformers. Due to the uniform sampling of image pixels, these nonlinear transformations can better model planar images to generate compact latent features, thereby improving coding efficiency. Therefore, current neural image compression methods are not suitable for directly compressing projected spherical images. To address non-uniform sampling, Li et al. proposed a 360-degree image compression method that applied latitude-adaptive end-to-end optimization to the entire ERP image. Bidgoli et al. proposed a convolutional neural network compression framework for panoramic images based on HEALPix sampling. However, learning-based spherical image compression methods are still in their infancy and need further exploration.
[0005] In addition to the design of the transformation module, the entropy model is also a key step in the learning-based image compression process. Balle et al. proposed an entropy model based on a hyper-prior. Minnen et al. further improved the entropy model by introducing a context model that predicts the current element based on the previously decoded latent element. Planar images are usually modeled by an autoregressive context using a Z-shaped encoding order in a two-dimensional Euclidean space via masked convolution. If, similar to planar images, the encoding order of the spherical image represented by SMSIR relies on the index order of the ordered root tree to establish the masked spherical convolution context, the encoding order cannot match the local support region of the spherical convolution filter. This mismatch makes it relatively difficult to establish such a context. Summary of the invention
[0006] In order to overcome the defects of the prior art, the technical problem to be solved by the present invention is to provide a spherical image compression method based on a spherical deep neural network, which can solve the problem of low efficiency of traditional spherical compression methods.
[0007] The technical solution of the present invention is: a spherical image compression method based on a spherical deep neural network, comprising the following steps:
[0008] (1) Spherical image resampling SMSIR based on spherical measurement uniformly samples the spherical image;
[0009] (2) Design the basic components of spherical image learning, including the following steps:
[0010] (2.1) Perform a block operation on the spherical image, where a block represents an area of the image. The size of the spherical block is 4 n , where n is an integer less than the partition level;
[0011] (2.2) Design a spherical convolution kernel with twelve neighborhoods to expand the neighborhood. In this spherical convolution kernel, the middle triangle area represents the center of the convolution kernel, and the other triangles represent the neighborhood. Each small triangle corresponds to a learnable weight. The arrangement of the convolution kernel weight index at different positions follows the SMSIR resampling method. The convolution operation on the sphere is defined as follows:
[0012] (1)
[0013] Among them, X c,m represents the feature at the mth neighbor (m=1…k), whose center point is c, X c,m ∈N(x c ), N(x c ) represents the neighborhood of c, W m represents the learnable weight of the convolution kernel, <,> represents the inner product operation, y crepresents the output feature, corresponding to the center of the input feature;
[0014] (2.3) Slide in index order with a step size of 4 n , where n represents the number of downsampling operations, and each downsampling reduces the resolution by half relative to the original resolution. In the image decoding process, in the classic deep learning framework embedded with transposed convolution, the transposed convolution parameters are modified to achieve image resolution upsampling through a learning-based method.
[0015] (2.4) The leaf nodes of any level subtree are arranged in ascending order according to the index value to form a sequence, and the spherical subtree is used as a window to limit the calculation of the transformer within the tree structure window, thereby constructing a spherical transformer model;
[0016] (3) By integrating spherical convolution and self-attention, the spherical transformation is modeled using the bit rate of the spherical entropy model.
[0017] The present invention uniformly samples spherical images based on spherical image resampling SMSIR of spherical measurement, designs basic components of spherical image learning, and models the spherical transformation by integrating spherical convolution and self-attention, and using the bit rate of the spherical entropy model, thereby solving the problem of low efficiency of traditional spherical compression methods.
[0018] A spherical image compression device based on a spherical deep neural network is also provided, the device comprising:
[0019] An image resampling module, which is configured to resample spherical images based on spherical measurement and uniformly sample spherical images using SMSIR;
[0020] The design module, which is configured to design the basic components of spherical image learning, performs:
[0021] (2.1) Perform a block operation on the spherical image, where a block represents an area of the image. The size of the spherical block is 4 n , where n is an integer less than the partition level;
[0022] (2.2) Design a spherical convolution kernel with twelve neighborhoods to expand the neighborhood. In this spherical convolution kernel, the middle triangle area represents the center of the convolution kernel, and the other triangles represent the neighborhood. Each small triangle corresponds to a learnable weight. The arrangement of the convolution kernel weight index at different positions follows the SMSIR resampling method. The convolution operation on the sphere is defined as follows:
[0023]
[0024] Among them, X c,m represents the feature at the mth neighbor (m=1…k), whose center point is c, Xc,m ∈N(x c ), N(x c ) represents the neighborhood of c, W m represents the learnable weight of the convolution kernel, <,> represents the inner product operation, y c represents the output feature, corresponding to the center of the input feature;
[0025] (2.3) Slide in index order with a step size of 4 n , where n represents the number of downsampling operations, and each downsampling reduces the resolution by half relative to the original resolution; in the image decoding process, in a typical deep learning framework that embeds transposed convolution, the transposed convolution parameters are modified to achieve image resolution upsampling through a learning-based method;
[0026] (2.4) The leaf nodes of any level subtree are arranged in ascending order according to the index value to form a sequence, and the spherical subtree is used as a window to limit the calculation of the transformer within the tree structure window, thereby constructing a spherical transformer model;
[0027] The transformation modeling module, whose configuration comes from the spherical transformation by integrating spherical convolution and self-attention, is modeled using the code rate of the spherical entropy model. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 A flowchart of step (1) of the spherical image compression method based on a spherical deep neural network according to the present invention is shown.
[0029] Figure 2 Convolution kernels at different positions on the spherical surface according to the present invention are shown.
[0030] Figure 3 An implementation of spherical convolution according to the present invention is shown.
[0031] Figure 4 The overall flow chart of the spherical transformer according to the present invention is shown.
[0032] Figure 5 The modules of the spherical Mixformer according to the present invention are shown.
[0033] Figure 6 Mask context modeling for planar and spherical images is shown.
[0034] Figure 7 A flowchart of step (3) of the spherical image compression method based on a spherical deep neural network according to the present invention is shown.
[0035] Figure 8 A flow chart of a spherical image compression method based on a spherical deep neural network according to the present invention is shown. DETAILED DESCRIPTION
[0036] like Figure 8 As shown, a spherical image compression method based on a spherical deep neural network comprises the following steps:
[0037] (1) Spherical image resampling based on spherical measure SMSIR uniformly samples the spherical image;
[0038] (2) Design the basic components of spherical image learning, including the following steps:
[0039] (2.1) Perform a block operation on the spherical image, where a block represents an area of the image. The size of the spherical block is 4 n , where n is an integer less than the partition level;
[0040] (2.2) Design a spherical convolution kernel with twelve neighborhoods to expand the neighborhood. In this spherical convolution kernel, the middle triangle area represents the center of the convolution kernel, and the other triangles represent the neighborhood. Each small triangle corresponds to a learnable weight. The arrangement of the convolution kernel weight index at different positions follows the SMSIR resampling method. The convolution operation on the sphere is defined as follows:
[0041]
[0042] Among them, X c,m represents the feature at the mth neighbor (m=1…k), whose center point is c, X c,m ∈N(x c ), N(x c ) represents the neighborhood of c, W m represents the learnable weight of the convolution kernel, <,> represents the inner product operation, y c represents the output feature, corresponding to the center of the input feature;
[0043] (2.3) Slide in index order with a step size of 4 n , where n represents the number of downsampling operations, and each downsampling reduces the resolution by half relative to the original resolution. In the image decoding process, in the classic deep learning framework embedded with transposed convolution, the transposed convolution parameters are modified to achieve image resolution upsampling through a learning-based method.
[0044] (2.4) The leaf nodes of any level subtree are arranged in ascending order according to the index value to form a sequence, and the spherical subtree is used as a window to limit the calculation of the transformer within the tree structure window, thereby constructing a spherical transformer model;
[0045] (3) By integrating spherical convolution and self-attention, the spherical transformation is modeled using the bit rate of the spherical entropy model.
[0046] The present invention uniformly samples spherical images based on spherical image resampling of spherical measurement SMSIR, designs basic components for spherical image learning, and models the spherical transformation by integrating spherical convolution and self-attention, using the bit rate of the spherical entropy model, thereby solving the problem of low efficiency of traditional spherical compression methods.
[0047] Preferably, in step (1), the spherical image is divided into eight spherical triangles, and then the midpoints of the three sides of each spherical triangle are calculated; these midpoints are connected by a large arc to generate four smaller spherical triangles; the indexes of the spherical triangle pixels are sorted based on the center, top / bottom, left and right positions; this quadtree subdivision process is recursively applied to generate smaller spherical triangles until the maximum division level is reached; the hierarchical index structure of the spherical triangle pixels generated on the spherical surface in a recursive manner constitutes an index structure based on an ordered root tree, and after SMSIR resampling, the spherical surface is divided into approximately uniform spherical triangle pixels.
[0048] Preferably, in step (2.2), the number of pixels of the spherical image after SMSIR resampling is expressed as 8×4 n , where n represents the division level; assuming that the shape of the input feature map F1 on the sphere is represented as
[0049] [C in ,1,8×4 n ], where C in represents the number of channels in F1; find twelve neighbors for each pixel and get the feature map F2, whose shape is [C in ,N,8×4 n ], where N represents the convolution kernel size; F2 is deformed to a shape of [C in ,1,N×8×4 n ] feature map F3, using the shape of [C out ,C in ,1,N] is convolved with the feature map F3 to obtain the spherical output feature map F o , whose shape is [C out ,1,8×4 n ], where C out Indicates the number of output channels.
[0050] Preferably, in step (2.4), the components of the spherical transformer include:
[0051] Block embedding: Spherical triangular pixels are represented as a one-dimensional sequence according to their index order, and each element in the sequence is embedded, and the final input sequence is embedded into the vector representation S l-1 =RN×d Where N is the sequence length and d is the model dimension, as the input of the transformer
[0052] S l-1 =[E(x1); E(x2); ...; E(x n )] (2)
[0053] in Represents embedding, which is a linear mapping function;
[0054] Window-based multi-head self-attention: Given an embedded sequence, the sequence is divided into subtree windows with a window size of 4. n , where n is an integer, the spherical block is divided into four windows, each with a window size of 4, for the local window feature S l-1 ∈R N×d , the query matrix Q, key matrix K and value matrix V are calculated as follows:
[0055] Q=S l-1 W Q ,K=S l-1 W K ,V=S l-1 W V (3)
[0056] W Q ,W K ,W V is the projection matrix shared between different windows. For the calculation of multi-head attention, the shapes of Q, K, and V are first deformed into , where h is the head in the multi-head attention mechanism
[0057] The self-attention calculation process in the local window is as follows:
[0058]
[0059]
[0060] Among them, B is a learnable positional encoding, W-MSA refers to window multi-head self-attention, and LN represents layer normalization;
[0061] Feedforward network layer: The feedforward neural network mainly uses multi-layer perceptron MLP, which consists of linear layers and activation layers. It performs nonlinear transformation and feature extraction on the attention output features, thereby enhancing the representation ability of the model.
[0062]
[0063] Preferably, in step (3), the spherical transformation integrating spherical convolution and self-attention is:
[0064] Assume that the size of the input feature F is L×C, apply a linear mapping function to F to obtain a feature of size L×C, where L represents the length of the feature and C represents the number of feature channels; divide the output feature evenly into two feature maps along the channel dimension, F sattn and F scnn , their dimensions are L×C / 2; for F sattn Apply layer normalization LN and spherical window multi-head self-attention SW-MSA to get the feature Using the convolution-friendly properties of batch normalization (BN), F scnn Apply BN and spherical convolution to get features Will and The concatenated features of are used as the input of the linear function to promote feature fusion. This process produces an F with a shape of L×C. sca features, thereby promoting the connection between windows to expand the receptive field; applying another LN and multi-layer
[0065] After the perceptron MLP, the features are passed out; the whole process is shown as follows:
[0066] F scnn ,F sattn =Channel_split(Linear(F)) (7)
[0067]
[0068]
[0069]
[0070] F out =MLP(LN(F sca )) (11).
[0071] Preferably, in step (3), modeling using the bit rate of the spherical entropy model includes:
[0072] (I) Bitrate modeling using super priors, super priors Using nonparametric fully factorized density models
[0073] Modeling:
[0074]
[0075] Among them, ψ i Represents each univariate distribution Parameters, Each element in is modeled as a Gaussian distribution with mean μ iand standard deviation σ i Super Prior predict:
[0076]
[0077] and The code rate is calculated as follows:
[0078]
[0079]
[0080] (II) Combine the super prior and spherical transformer context model for bit rate modeling, the formula is as follows:
[0081]
[0082] Where M represents a spherical mask with the lower triangular part set to 1 and all other values set to negative infinity, ⊙ represents the Hadamard product,
[0083] Using the spherical context via an autoregressive approach, the conditional Gaussian distribution for modeling entropy is derived as follows:
[0084]
[0085] in, Indicates the feature map Previous causal elements, sorted by ordered root tree index, μ i and σ i From these causal elements and the super prior Medium estimate.
[0086] Preferably, in step (II), when it is a spherical four-pixel block, assuming that the current element to be predicted is 3, the attention of the pixel with spherical index 3 only involves pixels with indexes 0, 1 and 2.
[0087] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes the steps of the above-mentioned embodiment method, and the storage medium can be: ROM / RAM, disk, CD, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes a spherical image compression device based on a spherical deep neural network, which is usually represented in the form of functional modules corresponding to the steps of the method. The device includes:
[0088] An image resampling module, which is configured to resample spherical images based on spherical measurement and uniformly sample spherical images using SMSIR;
[0089] The design module, which is configured to design the basic components of spherical image learning, performs:
[0090] (2.1) Perform a block operation on the spherical image, where a block represents an area of the image. The size of the spherical block is 4 n , where n is an integer less than the partition level;
[0091] (2.2) Design a spherical convolution kernel with twelve neighborhoods to expand the neighborhood. In this spherical convolution kernel, the middle triangle area represents the center of the convolution kernel, and the other triangles represent the neighborhood. Each small triangle corresponds to a learnable weight. The arrangement of the convolution kernel weight index at different positions follows the SMSIR resampling method. The convolution operation on the sphere is defined as follows:
[0092]
[0093] Among them, X c,m represents the feature at the mth neighbor (m=1…k), whose center point is c, X c,m ∈N(x c ), N(x c ) represents the neighborhood of c, W m represents the learnable weight of the convolution kernel, <,> represents the inner product operation, y c represents the output feature, corresponding to the center of the input feature;
[0094] (2.3) Slide in index order with a step size of 4 n , where n represents the number of downsampling operations, and each downsampling reduces the resolution by half relative to the original resolution. In the image decoding process, in the classic deep learning framework embedded with transposed convolution, the transposed convolution parameters are modified to achieve image resolution upsampling through a learning-based method.
[0095] (2.4) The leaf nodes of any level subtree are arranged in ascending order according to the index value to form a sequence, and the spherical subtree is used as a window to limit the calculation of the transformer within the tree structure window, thereby constructing a spherical transformer model;
[0096] Transformation modeling module, which is configured to perform spherical transformation by integrating spherical convolution and self-attention,
[0097] The bit rate of the spherical entropy model is used for modeling.
[0098] Preferably, in the image resampling module, the spherical image is divided into eight spherical triangles, and then the midpoints of the three sides of each spherical triangle are calculated; these midpoints are connected by large arcs to generate four smaller spherical triangles; the indexes of the spherical triangle pixels are sorted based on the center, top / bottom, left and right positions; this quadtree subdivision process is recursively applied to generate smaller spherical triangles until the maximum division level is reached; the hierarchical index structure of the spherical triangle pixels generated on the spherical surface in a recursive manner constitutes an index structure based on an ordered root tree, and after SMSIR resampling, the sphere is divided into roughly uniform spherical triangle pixels.
[0099] Preferably, in the design module, the number of pixels of the spherical image after SMSIR resampling is expressed as 8×4 n , where n represents the division level; assuming that the shape of the input feature map F1 on the sphere is represented as
[0100] [C in ,1,8×4 n ], where C in represents the number of channels in F1; find twelve neighbors for each pixel and get the feature map F2, whose shape is [C in ,N,8×4 n ], where N represents the convolution kernel size; F2 is deformed to a shape of [C in ,1,N×8×4 n ] feature map F3, using the shape of [C out ,C in ,1,N] is convolved with the feature map F3 to obtain the spherical output feature map F o , whose shape is [C out ,1,8×4 n ], where C out Indicates the number of output channels;
[0101] The components of the spherical transformer include:
[0102] Block embedding: Spherical triangular pixels are represented as a one-dimensional sequence according to their index order, and each element in the sequence is embedded, and the final input sequence is embedded into the vector representation S l-1 =R N×d Where N is the sequence length and d is the model dimension, as the input of the transformer
[0103] S l-1 =[E(x1); E(x2); ...; E(x n )] (2)
[0104] in Represents embedding, which is a linear mapping function;
[0105] Window-based multi-head self-attention: Given an embedded sequence, the sequence is divided into subtree windows with a window size of 4. n , where n is an integer, the spherical block is divided into four windows, each with a window size of 4, for the local window feature S l-1 ∈R N×d , the query matrix Q, key matrix K and value matrix V are calculated as follows:
[0106] Q=S l-1 W Q ,K=S l-1 W K ,V=S l-1 W V (3)
[0107] W Q ,W K ,W V is the projection matrix shared between different windows. For the calculation of multi-head attention, the shapes of Q, K, and V are first deformed into Where h is the number of heads in the multi-head attention mechanism, and the self-attention calculation process within the local window is as follows:
[0108]
[0109]
[0110] Among them, B is a learnable positional encoding, W-MSA refers to window multi-head self-attention, and LN represents layer normalization;
[0111] Feedforward network layer: The feedforward neural network mainly uses multi-layer perceptron MLP, which consists of linear layers and activation layers. It performs nonlinear transformation and feature extraction on the attention output features, thereby enhancing the representation ability of the model.
[0112]
[0113] The method of the present invention is described in more detail below.
[0114] Step 1: Spherical image resampling based on spherical measurement
[0115] A spherical image is an image signal defined on a spherical surface. Due to its geometric properties, it becomes difficult to process spherical images. In order to facilitate the processing of this structure, spherical images are usually projected as equirectangular projection (ERP) images. ERP images are obtained by unfolding the spherical image along longitude and latitude. In an ERP image, each latitude circle has the same number of sampled pixels. Therefore, as the ERP image moves from the equator to the poles, the sampling points become more and more dense, the distortion from the equator to the poles increases significantly, and there is redundant information near the poles. It is inappropriate to directly use a learning-based planar image compression method to process ERP images. Therefore, SMSIR is used to uniformly sample the spherical image. For the convenience of visualization and subsequent description, the resampling process of the sphere is represented as a plane, as shown in Figure 1 As shown. Specifically, this method divides the spherical image into eight spherical triangles, and then calculates the midpoints of the three sides of each spherical triangle. These midpoints are connected by great circle arcs to generate four smaller spherical triangles. The indexes of the spherical triangle pixels are sorted based on the center, top / bottom, left, and right positions. This quadtree subdivision process is recursively applied to generate smaller spherical triangles until the maximum division level is reached. The hierarchical index structure of the spherical triangle pixels generated recursively on the sphere constitutes an index structure based on an ordered root tree. After SMSIR resampling, the sphere is divided into roughly uniform spherical triangle pixels.
[0116] Step 2: Design of basic components for spherical image learning
[0117] 1. Obtaining spherical blocks
[0118] Spherical images have extremely high resolution, which makes it impractical to use the entire spherical image as input to the neural network because it requires huge hardware resources. Therefore, it is necessary to perform block operations on the spherical image, where a block represents a region of the image. However, randomly cropping fixed-size blocks (e.g., 256×256) from SMSIR images destroys its hierarchical structure. This destruction hinders subsequent spherical convolutions and hierarchical up / down sampling operations. In order to maintain the hierarchical structure during learning, the size of the spherical block must be 4 n , where n is an integer less than the partition level. Figure 1 As shown in the second level subdivision, assuming that the block size defined on the sphere is 2×2, then this spherical block corresponds to a block of four pixels of different colors on the sphere.
[0119] 2. Spherical convolution
[0120] Spherical convolution is the core operation in spherical learning, and the key lies in the design of the convolution kernel. Figure 1As shown in FIG. 1 , the spherical image representation of the present invention is based on a spherical quadtree triangle recursive subdivision structure. If a three-neighborhood convolution kernel based on a quadtree is used, the receptive field of the convolution kernel will be too small. Therefore, a twelve-neighborhood spherical convolution kernel is designed to perform neighborhood expansion, such as Figure 2 As shown. The size of this spherical convolution kernel is similar to the 3×3 convolution kernel commonly used in planar images. In the proposed spherical convolution kernel, the middle triangle area represents the center of the convolution kernel, while the other triangles represent the neighborhood. Each small triangle corresponds to a learnable weight. The arrangement of the convolution kernel weight index at different positions follows the SMSIR resampling method. The convolution operation on the sphere is defined as follows:
[0121]
[0122] Among them, X c,m represents the feature at the mth neighbor (m=1…k), whose center point is c, i.e. X c,m ∈N(x c ), N(x c ) represents the neighborhood of c, W m represents the learnable weight of the convolution kernel, and <,> represents the inner product operation. c Represents the output feature, corresponding to the center of the input feature.
[0123] Due to the non-square shape of the spherical convolution kernel and the non-Euclidean spatial structure of the SMSIR representation, it is not feasible to directly apply the existing deep learning tools for planar image convolution operations to spherical images. In order to use the traditional 2D convolution method for spherical convolution based on SMSIR, the implementation process of spherical convolution is redefined, such as Figure 3 shown.
[0124] The number of pixels of the spherical image after SMSIR resampling is represented as 8×4 n , where n represents the division level. Assume that the shape of the input feature map F1 on the sphere is represented as [C in ,1,8×4 n ], where C in represents the number of channels in F1. Then, twelve neighbors are found for each pixel to obtain the feature map F2, whose shape is [C in ,N,8×4 n ], where N represents the convolution kernel size. In order to use the traditional 2D convolution, F2 is deformed to a shape of [C in ,1,N×8×4 n ] feature map F3. Now, we can use the shape of [C out ,C in ,1,N] with learnable convolution weights and feature map F3 to perform traditional 2D convolution operation. The resulting spherical output feature map F o, whose shape is [C out ,1,8×4 n ], where C out Indicates the number of output channels.
[0125] 3. Sampling up and down on the sphere
[0126] Downsampling can reduce the resolution of an image and reduce computational complexity. In learning-based planar image compression, downsampling is usually achieved through convolution with a stride greater than 2. Based on the ordered root tree index structure represented by SMSIR, slide in index order with a step size of 4 n , where n represents the number of downsampling operations. Each time the downsampling is performed, the resolution is halved relative to the original resolution. For example, Figure 1 In the second level simplified planar triangle image, taking the first large triangle as an example, a sliding convolution with a stride of 4 is performed. The elements involved in the convolution are 0, 4, 8, and 12. The arrangement of the spherical triangle pixels after downsampling corresponds to the arrangement of the first level of division. In other words, after downsampling, the structure of the current division level is transferred to the previous division level, which effectively maintains the integrity of the spherical image structure.
[0127] In the image decoding process, it is necessary to increase the resolution of the image by upsampling for reconstruction. Upsampling methods usually include sub-pixel convolution and transposed convolution. Sub-pixel convolution first increases the number of channels through convolution, and then increases the image resolution by rearranging the channels into space. However, this method introduces additional computational complexity while increasing the number of channels. Therefore, the transposed convolution method is used to increase the image resolution. Using the spherical triangular pixel arrangement structure and the implementation process of spherical convolution, in the classic deep learning framework embedded with transposed convolution, the transposed convolution parameters can be simply modified to achieve image resolution upsampling through a learning-based method.
[0128] 4. Spherical transformer
[0129] Transformers have shown remarkable success in various visual tasks, leading to the emergence of transformer-based image compression methods. However, when dealing with spherical images, applying transformers faces some specific challenges: first, how to effectively embed spherical images; second, how to implement spherical transformers and reduce complexity.
[0130] Fortunately, a spherical representation method based on SMSIR is adopted. The quadtree structure of SMSIR is described by an ordered root tree index with continuous index values. Taking advantage of the continuity of this index value, the leaf nodes of any level subtree are arranged in ascending order of index values to form a sequence, and the transformer performs well in sequence embedding modeling. In order to alleviate the complexity of the transformer and maintain the hierarchy of the spherical structure, the spherical subtree is used as a window to limit the calculation of the transformer within the tree structure window. Based on these insights, the following is proposed: Figure 4 The spherical transformer model shown in Figure 1. The components of the spherical transformer are designed as follows:
[0131] 1) Block embedding: Spherical triangle pixels are represented as a one-dimensional sequence according to their index order. Each element in the sequence is embedded. The final input sequence is embedded into the vector representation S l-1 =R N×d , where N is the sequence length and d is the model dimension, as the input of the transformer.
[0132] S l-1 =[E(x1); E(x2); ...; E(x n )] (2)
[0133] in Represents embedding, usually a linear mapping function.
[0134] 2) Window-based multi-head self-attention: Given an embedded sequence, in order to reduce computational complexity and preserve the hierarchical spherical structure, the sequence is divided into subtree windows. The window size is set to 4 n , where n is an integer. Figure 4 As shown, the spherical block is divided into four windows, each with a window size of 4. For the local window feature S l-1 ∈R N×d , the query matrix Q, key matrix K and value matrix V are calculated as follows:
[0135] Q=S l-1 W Q ,K=S l-1 W K ,V=S l-1 W V (3)
[0136] W Q ,W K ,W V is the projection matrix shared between different windows. For the calculation of multi-head attention, firstly transform the shapes of Q, K, V into Where h is the number of heads in the multi-head attention mechanism. The self-attention calculation process within the local window is as follows:
[0137]
[0138]
[0139] Among them, B is a learnable positional encoding, W-MSA refers to windowed multi-head self-attention, and LN represents layer normalization.
[0140] 3) Feedforward network layer: The feedforward neural network mainly uses a multilayer perceptron (MLP), which consists of a linear layer and an activation layer. Its main function is to perform nonlinear transformation and feature extraction on the attention output features, thereby enhancing the representation ability of the model.
[0141]
[0142] Step 3: Spherical Image Compression with Spherical MixFormer and Spherical Transformer Context
[0143] 1. Spherical Mixformer
[0144] Based on the basic components of spherical learning proposed in step 2, a spherical transformer is used to extract non-local features and perform downsampling / upsampling in spherical image compression. However, in the spherical transformer, the attention of each window ignores the connection between different windows, resulting in a limited receptive field. Convolution can capture local contextual features of an image. Therefore, a simple method is proposed to use spherical convolution and spherical self-attention in parallel to increase the connection between different windows and expand the receptive field. The local information extracted by spherical convolution can also serve as a useful supplement to attention-based modeling of non-local information. Therefore, a spherical transformer module integrating spherical convolution and self-attention, called spherical MixFormer, is proposed, as shown in Figure 5 shown.
[0145] Assuming that the size of the input feature F of the spherical MixFormer is L×C, we first apply a linear mapping function to F to obtain a feature of size L×C, where L represents the length of the feature and C represents the number of feature channels. Then, the output feature is evenly divided into two feature maps along the channel dimension, F sattn and F scnn , their dimensions are both L×C / 2. Next, sattn Apply layer normalization (LN) and spherical window multi-head self-attention (SW-MSA) to obtain the features At the same time, the convolution-friendly properties of batch normalization (BN) are used to scnnApply BN and spherical convolution to get features Afterwards, and The concatenated features of are used as the input of the linear function to promote feature fusion. This process produces an F with a shape of L×C. sca The features are then transferred to the spherical MixFormer module, thereby promoting the connection between windows to expand the receptive field. Finally, after applying another LN and a multi-layer perceptron (MLP), the features are passed on. The entire process of the spherical MixFormer module can be expressed as follows:
[0146] F scnn ,F sattn =Channel_split(Linear(F)) (7)
[0147]
[0148]
[0149]
[0150] F out =MLP(LN(F sca )) (11)
[0151] The proposed spherical MIXFormer module, as a powerful nonlinear transformation module to enhance feature extraction, is integrated into the main analysis transform, main synthesis transform, super analysis transform and super synthesis transform.
[0152] 2. Bit rate modeling of spherical entropy model
[0153] In the image compression model, the bit rate is the cross entropy between the true distribution and the estimated distribution of the latent variable. The more accurate the estimation of the latent variable distribution is, the lower the bit rate is. By establishing a suitable context entropy estimation model, the estimation accuracy of the latent variable distribution can be improved. Therefore, the bit rate modeling of the spherical entropy model is as follows.
[0154] 1) Use super-prior bitrate modeling
[0155] Super Prior The model is modeled using a nonparametric fully decomposed density model, which is described as follows:
[0156]
[0157] Among them, ψ i Represents each univariate distribution Parameters. Each element in is modeled as a Gaussian distribution with mean μ i and standard deviation σ iSuper Prior predict:
[0158]
[0159] and The code rate is calculated as follows:
[0160]
[0161]
[0162] 2) Combined hyper-prior and spherical transformer context model for rate modeling
[0163] In the context of classical learning-based planar image compression, Figure 6 (a) is an example. The dark gray element represents the element to be encoded or decoded. The element before the dotted box element is the decoded element, and the element after it is the undecoded element. Context modeling can be performed by establishing a two-dimensional 3×3 mask convolution with a dark gray element as the center. For spherical images, in order to establish a context model similar to the mask convolution in planar images, Figure 2 The convolution kernel in constructs a context model based on the structure of ordered tree root index, such as Figure 6 (b). The encoding order based on the ordered tree root index cannot match the local support area of the fixed masked spherical convolution kernel at different encoding positions. Therefore, it becomes difficult to establish the masked spherical convolution context at different positions. The flexibility of the spherical transformer proposed in step 2 enables the use of context that conforms to the encoding order. In the self-attention mechanism of the spherical transformer, a mask is introduced in equation (4) to model the spherical transformer context model, which is as follows:
[0164]
[0165] Where M represents a spherical mask, the value of the lower triangular part is set to 1, and all other values are set to negative infinity, and ⊙ represents the Hadamard product. Taking a spherical four-pixel block as an example, assuming that the current element to be predicted is 3, such as Figure 6 As shown in (c), the corresponding spherical Mask self-attention map is as follows Figure 6 (d). Figure 6 (d) It can be seen that the attention of the pixel with spherical index 3 can only involve pixels with indices 0, 1, and 2.
[0166] By using the spherical context through autoregressive methods, we can derive the conditional Gaussian distribution for modeling entropy as follows:
[0167]
[0168] in, Indicates the feature map Previous causal elements, sorted by ordered root tree index. μ i and σ i From these causal elements and the super prior Estimated in.
[0169] Through the above three steps, the overall framework of spherical compression based on learning is established. Figure 7 shown.
[0170] 1. Experimental data: Considering the lack of high-resolution public spherical image datasets for spherical image compression, this paper uses 2000 high-resolution spherical images in the Refer360 dataset as a compression dataset. In order to further expand the dataset, 4786 high-quality images with a resolution of 4096×2048 are collected from the Internet. By combining these two datasets, the compression model is optimized.
[0171] 2. Metrics: Spherical image quality assessment uses SPSNR and WSPSNR as target quality assessment. Two types of measurement methods include rate distortion (RD) and bit rate saving (BD-BR).
[0172] 3. Experimental settings: Adam optimizer is selected to optimize the objective function. In all experiments, the learning rate is set to 0.0001, the batch size is set to 8, and a total of 400 epochs are trained. After 300 epochs, the learning rate will be 0.00001.
[0173] 4. Comparison with different compression methods: This method is compared with several traditional codecs and learning-based methods. Traditional codecs include JPEG, JPEG2000 and BPG. Among the learning-based methods, it is compared with Balle18, Minnen18 planar image compression methods and OSLO, a spherical image compression method based on hyper-prior.
[0174] In Table 1, the compression method is compared with the planar compression method, and the BD-BR and BD-distortion metrics are calculated using RATE-SPSNR and RATE-WSPSNR. The number of resampled pixels is 8×4 9(2048×1024). The BPG method is used as the baseline method. For RATE-SPSNR, this method improves the S-PSNR by 0.2dB over the BPG method and achieves an average bitrate reduction of 13.85%. As for RATE-WSPSNR, this method improves the WS-PSNR of BPG by 0.95dB and achieves an average bitrate reduction of 17.49%, achieving the best results compared with other methods. In Table 2, the spherical learning compression methods under different sampling numbers are compared. When the number of divisions n is equal to 9, this method resamples fewer pixels, resulting in a lower RATE-SPSNR metric than the OSLO method. When n is equal to 10, both RATE-SPSNR and RATE-WSPSNR metrics are significantly improved.
[0175] Table 1 Performance comparison with planar compression methods on BD-SPSNR, BD-WSPSNR and BD-BR indicators (bold indicates the best effect)
[0176]
[0177] Table 2 Performance comparison with spherical compression method on BD-SPSNR, BD-WSPSNR and BD-BR indicators (bold indicates the best effect)
[0178]
[0179]
[0180] 5. Ablation experiment: When the pixel resampling number is 8×4 9 Ablation studies are performed on the proposed spherical MixFormer module and the spherical transformer context model. BD-SPSNR, BD-WSPSNR, and BD-RATE are calculated using the Balle18 model trained on ERP images as a baseline. In the SmixFormer module, spherical convolution is combined with the self-attention of the spherical transformer. The performance of different modules of the proposed method is shown in Table 3.
[0181] As can be seen from Table 3, the advantages of spherical transformer and spherical convolution complement each other. When the spherical transformer context model is added to the spherical MixFormer, the performance is significantly improved, which proves the effectiveness of the proposed spherical transformer context model. Overall, the proposed method saves an average of 16.65% bit rate in RATE-SPSNR and 24.42% bit rate in RATE-WSPSNR compared to the Balle18 method.
[0182] Table 3 Performance comparison of ablation methods on BD-SPSNR, BD-WSPSNR and BD-BR indicators (bold indicates the best effect)
[0183]
[0184] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the protection scope of the technical solution of the present invention.
Claims
1. A spherical image compression method based on a spherical deep neural network, characterized in that: The method comprises the following steps: (1) Spherical image resampling SMSIR based on spherical measurement uniformly samples the spherical image; (2) Design the basic components of spherical image learning, including the following steps: (2.1) Perform a block operation on the spherical image, where a block represents an area of the image. The size of the spherical block is 4 n , where n is an integer less than the partition level; (2.2) Design a spherical convolution kernel with twelve neighborhoods to expand the neighborhood. In this spherical convolution kernel, the middle triangle area represents the center of the convolution kernel, and the other triangles represent the neighborhood. Each small triangle corresponds to a learnable weight. The arrangement of the convolution kernel weight index at different positions follows the SMSIR resampling method. The convolution operation on the sphere is defined as follows: Among them, X c,m represents the feature at the mth neighbor (m=1…k), whose center point is c, X c,m ∈N(x c ), N(x c ) represents the neighborhood of c, W m represents the learnable weight of the convolution kernel, <,> represents the inner product operation, y c represents the output feature, corresponding to the center of the input feature; (2.3) Slide in index order with a step size of 4 n , where n represents the number of downsampling operations, and each downsampling reduces the resolution by half relative to the original resolution; in the image decoding process, in a typical deep learning framework that embeds transposed convolution, the transposed convolution parameters are modified to achieve image resolution upsampling through a learning-based method; (2.4) The leaf nodes of any level subtree are arranged in ascending order according to the index value to form a sequence, and the spherical subtree is used as a window to limit the calculation of the transformer within the tree structure window, thereby constructing a spherical transformer model; (3) By integrating spherical convolution and self-attention, the spherical transformation is modeled using the bit rate of the spherical entropy model.
2. The spherical image compression method based on a spherical deep neural network according to claim 1, characterized in that: In the step (1), the spherical image is divided into eight spherical triangles, and then the midpoints of the three sides of each spherical triangle are calculated; these midpoints are connected by a large arc to generate four smaller spherical triangles; the indexes of the spherical triangle pixels are sorted based on the center, top / bottom, left and right positions; this quadtree subdivision process is recursively applied to generate smaller spherical triangles until the maximum division level is reached; the hierarchical index structure of the spherical triangle pixels generated on the spherical surface in a recursive manner constitutes an index structure based on an ordered root tree, and after SMSIR resampling, the spherical surface is divided into approximately uniform spherical triangle pixels.
3. The spherical image compression method based on spherical deep neural network according to claim 2 is characterized in that: In the step (2.2), the number of pixels of the spherical image after SMSIR resampling is expressed as 8×4 n , where n represents the division level; assuming that the shape of the input feature map F1 on the sphere is represented by [C in ,1,8×4 n ], where C in represents the number of channels in F1; find twelve neighbors for each pixel and get the feature map F2, whose shape is [C in ,N,8×4 n ], where N represents the convolution kernel size; F2 is deformed to a shape of [C in ,1,N×8×4 n ] feature map F3, using the shape of [C out ,C in ,1,N] is convolved with the feature map F3 to obtain the spherical output feature map F o , whose shape is [C out ,1,8×4 n ], where C out Indicates the number of output channels.
4. The spherical image compression method based on a spherical deep neural network according to claim 3, characterized in that: In step (2.4), the components of the spherical transformer include: Block embedding: Spherical triangle pixels are represented as a one-dimensional sequence according to their index order, and each element in the sequence is embedded, and the final input sequence is embedded into the vector representation S l-1 =R N×d Where N is the sequence length and d is the model dimension, as the input of the transformer S l-1 =[E(x1);E(x2);...;E(x n )] (2) in Represents embedding, which is a linear mapping function; Window-based multi-head self-attention: Given an embedded sequence, the sequence is divided into subtree windows with a window size of 4. n , where n is an integer, the spherical block is divided into four windows, each with a window size of 4, for the local window feature S l-1 ∈R N ×d , the query matrix Q, key matrix K and value matrix V are calculated as follows: Q=S l-1 W Q ,K=S l-1 W K ,V=S l-1 W V (3) W Q ,W K ,W V is the projection matrix shared between different windows. For the calculation of multi-head attention, the shapes of Q, K, and V are first deformed into Where h is the number of heads in the multi-head attention mechanism, and the self-attention calculation process within the local window is as follows: Among them, B is a learnable positional encoding, W-MSA refers to window multi-head self-attention, and LN represents layer normalization; Feedforward network layer: The feedforward neural network mainly uses multi-layer perceptron MLP, which consists of linear layers and activation layers. It performs nonlinear transformation and feature extraction on the attention output features, thereby enhancing the representation ability of the model.
5. The spherical image compression method based on spherical deep neural network according to claim 4 is characterized in that: In step (3), the spherical transformation integrating spherical convolution and self-attention is: Assume that the size of the input feature F is L×C, apply a linear mapping function to F to obtain a feature of size L×C, where L represents the length of the feature and C represents the number of feature channels; divide the output feature evenly into two feature maps along the channel dimension, F sattn and F scnn , their dimensions are both L×C / 2; for F sattn Apply layer normalization LN and spherical window multi-head self-attention SW-MSA to get the feature Using the convolution-friendly properties of batch normalization (BN), F scnn Apply BN and spherical convolution to get features Will and The concatenated features of are used as the input of the linear function to promote feature fusion. This process produces an F with a shape of L×C. sca Features are obtained by facilitating the connection between windows to expand the receptive field; after applying another LN and a multi-layer perceptron MLP, the features are passed on; The whole process is shown as follows: F scnn ,F sattn =Channel_split(Linear(F)) (7) F out =MLP(LN(F sca )) (11)。 6. The spherical image compression method based on spherical deep neural network according to claim 5, characterized in that: In the step (3), modeling using the bit rate of the spherical entropy model includes: (I) Bitrate modeling using super priors, super priors Modeling using a nonparametric fully factorized density model: Among them, ψ i Denotes each univariate distribution p zi |ψ i Parameters, Each element in is modeled as a Gaussian distribution with mean μ i and standard deviation σ i Super Prior predict: and The code rate is calculated as follows: (II) Combine the super prior and spherical transformer context model for bit rate modeling, the formula is as follows: Where M represents a spherical mask with the lower triangular part set to 1 and all other values set to negative infinity, ⊙ represents the Hadamard product, Using the spherical context via an autoregressive approach, the conditional Gaussian distribution for modeling entropy is derived as follows: in, Indicates the feature map Previous causal elements, sorted by ordered root tree index, μ i and σ i From these causal elements and the super prior Medium estimate.
7. The spherical image compression method based on spherical deep neural network according to claim 6, characterized in that: In the step (II), when it is a spherical four-pixel block, assuming that the current element to be predicted is 3, the attention of the pixel with spherical index 3 only involves pixels with indexes 0, 1 and 2.
8. A spherical image compression device based on a spherical deep neural network, characterized in that: The device includes: Image resampling module, whose configuration comes from spherical image resampling based on spherical measurement, using SMSIR to uniformly sample the spherical image; The design module, which is configured to design the basic components of spherical image learning, performs: (2.1) Perform a block operation on the spherical image, where a block represents an area of the image. The size of the spherical block is 4 n , where n is an integer less than the partition level; (2.2) Design a spherical convolution kernel with twelve neighborhoods to expand the neighborhood. In this spherical convolution kernel, the middle triangle area represents the center of the convolution kernel, and the other triangles represent the neighborhood. Each small triangle corresponds to a learnable weight. The arrangement of the convolution kernel weight index at different positions follows the SMSIR resampling method. The convolution operation on the sphere is defined as follows: Among them, X c,m represents the feature at the mth neighbor (m=1…k), whose center point is c, X c,m ∈N(x c ), N(x c ) represents the neighborhood of c, W m represents the learnable weight of the convolution kernel, <,> represents the inner product operation, y c represents the output feature, corresponding to the center of the input feature; (2.3) Slide in index order with a step size of 4 n , where n represents the number of downsampling operations, and each downsampling reduces the resolution by half relative to the original resolution; in the image decoding process, in a typical deep learning framework that embeds transposed convolution, the transposed convolution parameters are modified to achieve image resolution upsampling through a learning-based method; (2.4) The leaf nodes of any level subtree are arranged in ascending order according to the index value to form a sequence, and the spherical subtree is used as a window to limit the calculation of the transformer within the tree structure window, thereby constructing a spherical transformer model; The transformation modeling module is configured to model the code rate using the spherical entropy model by integrating spherical convolution and self-attention.
9. The spherical image compression device based on spherical deep neural network according to claim 8, characterized in that: In the image resampling module, the spherical image is divided into eight spherical triangles, and then the midpoints of the three sides of each spherical triangle are calculated; these midpoints are connected by a large arc to generate four smaller spherical triangles; the indexes of the spherical triangle pixels are sorted based on the center, top / bottom, left and right positions; this quadtree subdivision process is recursively applied to generate smaller spherical triangles until the maximum division level is reached; the hierarchical index structure of the spherical triangle pixels generated on the spherical surface in a recursive manner constitutes an index structure based on an ordered root tree, and after SMSIR resampling, the spherical surface is divided into approximately uniform spherical triangle pixels.
10. The spherical image compression device based on spherical deep neural network according to claim 9, characterized in that: In the design module, the number of pixels of the spherical image after SMSIR resampling is expressed as 8×4 n , where n represents the division level; assuming that the shape of the input feature map F1 on the sphere is represented by [C in ,1,8×4 n ], where C in represents the number of channels in F1; find twelve neighbors for each pixel and get the feature map F2, whose shape is [C in ,N,8×4 n ], where N represents the convolution kernel size; F2 is deformed to a shape of [C in ,1,N×8×4 n ] feature map F3, using the shape of [C out ,C in ,1,N] is convolved with the feature map F3 to obtain the spherical output feature map F o , whose shape is [C out ,1,8×4 n ], where C out Indicates the number of output channels; The components of the spherical transformer include: Block embedding: Spherical triangle pixels are represented as a one-dimensional sequence according to their index order, and each element in the sequence is embedded, and the final input sequence is embedded into the vector representation S l-1 =R N×d Where N is the sequence length and d is the model dimension, as the input of the transformer S l-1 =[E(x1);E(x2);...;E(x n )] (2) in Represents embedding, which is a linear mapping function; Window-based multi-head self-attention: Given an embedded sequence, the sequence is divided into subtree windows with a window size of 4. n , where n is an integer, the spherical block is divided into four windows, each with a window size of 4, for the local window feature S l-1 ∈R N ×d , the query matrix Q, key matrix K and value matrix V are calculated as follows: Q=S l-1 W Q ,K=S l-1 W K ,V=S l-1 W V (3) W Q ,W K ,W V is the projection matrix shared between different windows. For the calculation of multi-head attention, the shapes of Q, K, and V are first deformed into Where h is the number of heads in the multi-head attention mechanism, and the self-attention calculation process within the local window is as follows: Among them, B is a learnable positional encoding, W-MSA refers to window multi-head self-attention, and LN represents layer normalization; Feedforward network layer: The feedforward neural network mainly uses multi-layer perceptron MLP, which consists of linear layers and activation layers. It performs nonlinear transformation and feature extraction on the attention output features, thereby enhancing the representation ability of the model.
Citation Information
Patent Citations
Spherical image indexing method and device
CN112364190A
Spherical image compression method based on spherical wavelet transform
CN114449248A
Coding Method and Device Depth Video by Spherical Surface Modeling
KR1020180079074A
Calculation method of three-dimensional model's spherical expression based on multi-stage deformation reconstruction
US20220222891A1
Cited By
Point cloud data compression method and device based on spherical quadtree
CN120563639A
A point cloud data compression method and device based on spherical quadtree
CN120563639B
