Spherical image compression method and device based on spherical depth neural network

Abstract

A spherical image compression method and device based on a spherical depth neural network are disclosed.The present application is based on the resampling of a spherical image based on a spherical measure,uniformly samples the spherical image using SMSIR,designs a basic component for learning a spherical image,through the integration of spherical convolution and self-attention spherical transformation,models the code rate using a spherical entropy model,so as to solve the problem of low efficiency of traditional spherical compression methods.

Description

Technical Field

[0001] This invention relates to the technical field of image compression, and more particularly to a spherical image compression method based on a spherical deep neural network, and a spherical image compression device based on a spherical deep neural network. Background Technology

[0002] Spherical images are generated by stitching together multiple views and are also known as panoramic or 360-degree images. Spherical images provide a 360-degree view around a specific scene, allowing viewers to observe it from any angle. Virtual reality (VR) technology offers consumers a unique immersive experience through head-mounted displays. With the rapid development of VR technology, spherical images, as a primary medium for carrying VR content, have received increasing attention. However, compared to planar images, obtaining ultra-high-definition panoramic images requires higher resolutions (e.g., 4K, 8K, or even higher), further exacerbating storage and transmission challenges. Therefore, proposing a new spherical image compression method has become an urgent need.

[0003] For spherical image compression, the spherical image is typically projected onto a 2D image before encoding, and then the projected 2D image is directly compressed using existing compression standards. Different projection methods have been studied, such as equal rectangular projection (ERP), cube projection, adjusted cube projection, and piecewise spherical projection. However, since the spherical image is on a sphere, the projection from the sphere to the 2D plane inevitably leads to non-uniform sampling or structural damage to the sphere. Therefore, regardless of the projection method used, projection introduces non-uniform sampling on the sphere, which leads to correlation destruction and makes the rate-distortion optimization process in compression inefficient.

[0004] In recent years, neural image compression has made significant progress, mainly due to improvements in the transform, quantization, and entropy modeling stages of the compression process. Transformations are primarily based on convolutional neural networks and transformers. Due to the uniform sampling of image pixels, these nonlinear transforms can better model planar images to generate compact latent features, thereby improving coding efficiency. Therefore, current neural image compression methods are not suitable for directly compressing projected spherical images. To address non-uniform sampling, Li et al. proposed a 360-degree image compression method that applies latitude-adaptive end-to-end optimization to the entire ERP image. Bidgoli et al. proposed a convolutional neural network compression framework based on HEALPix sampling for panoramic images. However, learning-based spherical image compression methods are still in their early stages and require further exploration.

[0005] Besides the design of the transform module, the entropy model is also a crucial step in the learning-based image compression process. Balle et al. proposed an entropy model based on a super-prior. Minnen et al. further improved the entropy model by introducing a context model, which predicts the current element based on previously decoded latent elements. Planar images are typically modeled using a zigzag coding order in two-dimensional Euclidean space through masked convolution to establish an autoregressive context model. If, similar to planar images, the coding order of spherical images represented by SMSIR relies on the index order of an ordered root tree to establish the masked spherical convolution context, then the coding order cannot match the local support regions of the spherical convolution filter. This mismatch makes establishing such a context relatively difficult. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, the technical problem to be solved by the present invention is to provide a spherical image compression method based on a spherical deep neural network, which can solve the problem of low efficiency of traditional spherical compression methods.

[0007] The technical solution of this invention is: a spherical image compression method based on a spherical deep neural network, comprising the following steps:

[0008] (1) Spherical image resampling based on spherical measure (SMSIR) performs uniform sampling of spherical images;

[0009] (2) Design the basic components for spherical image learning, including the following steps:

[0010] (2.1) Perform a block extraction operation on the spherical image, where each block represents a region of the image, and the size of the spherical block is 4. n , where n is an integer less than the classification level;

[0011] (2.2) Design a 12-neighborhood spherical convolution kernel for neighborhood expansion. In this spherical convolution kernel, the central triangular region represents the center of the convolution kernel, while the other triangles represent the neighborhood. Each small triangle corresponds to a learnable weight. The arrangement of the convolution kernel weight indices at different positions follows the SMSIR resampling method. The convolution operation on the sphere is defined as follows:

[0012]

[0013] Among them, X c,m Let X represent the feature of the m-th neighbor (m = 1…k), with center point c. c,m ∈N(x c ), N(x c ) represents the neighborhood of c, W m The y represents the learnable weights of the convolution kernel, <,> represents the inner product operation, and y c The output feature corresponds to the center of the input feature;

[0014] (2.3) Slide according to the index order, with a step size of 4. n , where n represents the number of downsampling operations. Each downsampling operation halves the resolution relative to the original resolution. During image decoding, in the classic deep learning framework that embeds transposed convolution, the transposed convolution parameters are modified to achieve image resolution upsampling through a learning-based method.

[0015] (2.4) The leaf nodes of any level of subtree are arranged in ascending order of index value to form a sequence. The spherical subtree is used as a window to limit the calculation of the transformer within the tree structure window, thereby constructing the spherical transformer model.

[0016] (3) By integrating spherical convolution and self-attention spherical transformation, the code rate of the spherical entropy model is used for modeling.

[0017] This invention uses spherical image resampling (SMSIR) based on spherical measure to uniformly sample spherical images, designs basic components for spherical image learning, and integrates spherical transformation with spherical convolution and self-attention. It also uses the bit rate of the spherical entropy model for modeling, thus solving the problem of low efficiency in traditional spherical compression methods.

[0018] A spherical image compression device based on a spherical deep neural network is also provided, the device comprising:

[0019] The image resampling module is configured to resample spherical images based on spherical measures, using SMSIR to uniformly sample the spherical image;

[0020] The design module, configured to design the basic components for spherical image learning, executes:

[0021] (2.1) Perform a block extraction operation on the spherical image, where each block represents a region of the image, and the size of the spherical block is 4. n , where n is an integer less than the classification level;

[0022] (2.2) Design a 12-neighborhood spherical convolution kernel for neighborhood expansion. In this spherical convolution kernel, the central triangular region represents the center of the convolution kernel, while the other triangles represent the neighborhood. Each small triangle corresponds to a learnable weight. The arrangement of the convolution kernel weight indices at different positions follows the SMSIR resampling method. The convolution operation on the sphere is defined as follows:

[0023]

[0024] Among them, X c,m Let X represent the feature of the m-th neighbor (m = 1…k), with center point c. c,m ∈N(x c), N(x c ) represents the neighborhood of c, W m The y represents the learnable weights of the convolution kernel, <,> represents the inner product operation, and y c The output feature corresponds to the center of the input feature;

[0025] (2.3) Slide according to the index order, with a step size of 4. n , where n represents the number of downsampling operations. Each downsampling operation halves the resolution relative to the original resolution. In the image decoding process, in a typical deep learning framework that embeds transposed convolutions, the transposed convolution parameters are modified to achieve image resolution upsampling through a learning-based method.

[0026] (2.4) The leaf nodes of any level of subtree are arranged in ascending order of index value to form a sequence. The spherical subtree is used as a window to limit the calculation of the transformer within the tree structure window, thereby constructing the spherical transformer model.

[0027] The transformation modeling module, whose configuration comes from spherical transformations through the integration of spherical convolution and self-attention.

[0028] Alternatively, we can use the bitrate of the spherical entropy model for modeling. Attached Figure Description

[0029] Figure 1 A flowchart of step (1) of the spherical image compression method based on a spherical deep neural network according to the present invention is shown.

[0030] Figure 2 The convolution kernels at different positions on the sphere according to the present invention are shown.

[0031] Figure 3 An implementation of spherical convolution according to the present invention is shown.

[0032] Figure 4 An overall flowchart of the spherical transformer according to the present invention is shown.

[0033] Figure 5 A module of the spherical Mixformer according to the present invention is shown.

[0034] Figure 6 Mask context modeling for planar and spherical images is shown.

[0035] Figure 7 A flowchart of step (3) of the spherical image compression method based on a spherical deep neural network according to the present invention is shown.

[0036] Figure 8 A flowchart of a spherical image compression method based on a spherical deep neural network according to the present invention is shown. Detailed Implementation

[0037] like Figure 8 As shown, a spherical image compression method based on a spherical deep neural network includes the following steps:

[0038] (1) Spherical image resampling based on spherical measure SMSIR performs uniform sampling on spherical images;

[0039] (2) Design the basic components for spherical image learning, including the following steps:

[0040] (2.1) Perform a block extraction operation on the spherical image, where each block represents a region of the image, and the size of the spherical block is 4. n , where n is an integer less than the classification level;

[0041] (2.2) Design a 12-neighborhood spherical convolution kernel for neighborhood expansion. In this spherical convolution kernel, the central triangular region represents the center of the convolution kernel, while the other triangles represent the neighborhood. Each small triangle corresponds to a learnable weight. The arrangement of the convolution kernel weight indices at different positions follows the SMSIR resampling method. The convolution operation on the sphere is defined as follows:

[0042]

[0043] Among them, X c,m Let X represent the feature of the m-th neighbor (m = 1…k), with center point c. c,m ∈N(x c ), N(x c ) represents the neighborhood of c, W m The y represents the learnable weights of the convolution kernel, <,> represents the inner product operation, and y c The output feature corresponds to the center of the input feature;

[0044] (2.3) Slide according to the index order, with a step size of 4. n , where n represents the number of downsampling operations. Each downsampling operation halves the resolution relative to the original resolution. During image decoding, in the classic deep learning framework that embeds transposed convolution, the transposed convolution parameters are modified to achieve image resolution upsampling through a learning-based method.

[0045] (2.4) The leaf nodes of any level of subtree are arranged in ascending order of index value to form a sequence. The spherical subtree is used as a window to limit the calculation of the transformer within the tree structure window, thereby constructing the spherical transformer model.

[0046] (3) By integrating spherical convolution and self-attention spherical transformation, the code rate of the spherical entropy model is used for modeling.

[0047] This invention uses spherical image resampling based on the spherical metric SMSIR to uniformly sample spherical images, designs basic components for spherical image learning, and integrates spherical transformation with spherical convolution and self-attention. It also uses the bit rate of the spherical entropy model for modeling, thus solving the problem of low efficiency in traditional spherical compression methods.

[0048] Preferably, in step (1), the spherical image is divided into eight spherical triangles, and then the midpoints of the three sides of each spherical triangle are calculated; these midpoints are connected by great circle arcs to generate four smaller spherical triangles; the indexes of the spherical triangle pixels are sorted based on the center, top / bottom, left and right positions; this quadtree subdivision process is recursively applied to generate smaller spherical triangles until the maximum subdivision level is reached; the hierarchical index structure of the spherical triangle pixels generated on the sphere in the recursive manner constitutes an index structure based on an ordered root tree, and after SMSIR resampling, the sphere is divided into approximately uniform spherical triangle pixels.

[0049] Preferably, in step (2.2), the number of pixels in the spherical image after SMSIR resampling is represented as 8×4. n Where n represents the division level; assuming the shape of the input feature map F1 on the sphere is represented as...

[0050] [C in ,1,8×4 n ], where C in This represents the number of channels in F1; for each pixel, twelve neighbors are found to obtain feature map F2, whose shape is [C]. in N,8×4 n ], where N represents the kernel size; transforming F2 yields a shape of [C in ,1,N×8×4 n Feature map F3, using shape [C] out C in The learnable convolutional weights [1, N] are convolved with the feature map F3 in a 2D manner to obtain a spherical output feature map F. o Its shape is [C out ,1,8×4 n ], where C out Indicates the number of output channels.

[0051] Preferably, in step (2.4), the components of the spherical transformer include:

[0052] Block embedding: Spherical triangular pixels are represented as a one-dimensional sequence according to their index order, and each element in the sequence... The input sequence is ultimately embedded into the vector representation S. l-1 =RN×d In the diagram, N is the sequence length and d is the model dimension, which serve as the input to the transformer.

[0053] S l-1 = [E(x1); E(x2); ...; E(x n (2)

[0054] in Embedding is represented by a linear mapping function;

[0055] Window-based multi-head self-attention: Given an embedded sequence, divide the sequence into subtree windows, with a window size of 4. n Where n is an integer, the spherical patch is divided into four windows, each window being 4 in size, and for the local window feature S l-1 ∈R N×d The query matrix Q, key matrix K, and value matrix V are calculated as follows:

[0056] Q = S l-1 W Q K = S l-1 W K V = S l-1 W V (3)

[0057] W Q W K W V It is a projection matrix shared between different windows. For multi-head attention calculation, the shapes of Q, K, and V are first deformed into... Where h is the head in the multi-head attention mechanism.

[0058] The self-attention calculation process within a local window is as follows:

[0059]

[0060] Where B is a learnable positional encoding, W-MSA refers to window multi-head self-attention, and LN represents layer normalization;

[0061] Feedforward Network Layers: Feedforward neural networks primarily utilize multilayer perceptrons (MLPs), consisting of linear layers and activation layers. They perform nonlinear transformations and feature extraction on the attention output features, thereby enhancing the model's representational capabilities.

[0062] Preferably, in step (3), the spherical transformation integrating spherical convolution and self-attention is:

[0063] Assuming the input feature F has a size of L×C, applying a linear mapping function to F yields a feature map of size L×C, where L represents the feature length and C represents the number of feature channels. The output feature is then uniformly divided into two feature maps along the channel dimension, F. sattn and F scnn Their dimensions are all L×C / 2; for F sattn Apply layer normalized LN and spherical window multi-head self-attention SW-MSA to obtain features. Leveraging the convolution-friendly properties of batch normalization (BN), F scnn By applying Batch Normalization (BN) and spherical convolution, features are obtained. Will and The concatenation features are used as input to a linear function to facilitate feature fusion, and this process produces an F function of shape L×C. sca Features that facilitate connections between windows to expand the receptive field; in the application of another LN and multilayer

[0064] After the perceptron MLP is applied, the features are passed out; the entire process is represented as follows:

[0065] F scnn, F sattn =Channel_split(Linear(F)) (7)

[0066]

[0067] F out =MLP(LN(F sca (11). Preferably, in step (3), modeling using the bitrate of the spherical entropy model includes:

[0068] (I) Using super-prior bitrate modeling, super-prior Density modulus using nonparametric complete decomposition

[0069] Modeling the type:

[0070]

[0071] Where, ψ i Represents the distribution of each univariate. The parameters, Each element in Modeled as a Gaussian distribution with mean μ i and standard deviation σ i All are derived from prior knowledge predict:

[0072]

[0073] and The bitrate is calculated as follows:

[0074]

[0075] (II) Bitrate modeling is performed using a combination of super-prior and spherical transformer context model, as shown in the following formula:

[0076]

[0077] Where M represents a spherical mask, with the lower triangular portion set to 1 and all other values ​​set to negative infinity, and ⊙ represents the Hadamard product.

[0078] By utilizing the spherical context through an autoregressive method, the conditional Gaussian distribution used to model entropy is derived as follows:

[0079]

[0080] in, Representation of feature map The previous causal elements, sorted by ordered root tree index, μ i and σ i From these causal elements and prior knowledge The estimate is as follows.

[0081] Preferably, in step (II), when it is a spherical four-pixel block, assuming that the element to be predicted is 3, the attention of the pixel with spherical index 3 only involves the pixels with indices 0, 1 and 2.

[0082] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. When executed, the program includes the steps of the methods of the above embodiments. The storage medium can be ROM / RAM, magnetic disk, optical disk, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes a spherical image compression device based on a spherical deep neural network. This device is typically represented in the form of functional modules corresponding to the steps of the method. The device includes:

[0083] The image resampling module is configured to resample spherical images based on spherical measures, using SMSIR to uniformly sample the spherical image;

[0084] The design module, configured to design the basic components for spherical image learning, executes:

[0085] (2.1) Perform a block extraction operation on the spherical image, where each block represents a region of the image, and the size of the spherical block is 4.n , where n is an integer less than the classification level;

[0086] (2.2) Design a 12-neighborhood spherical convolution kernel for neighborhood expansion. In this spherical convolution kernel, the central triangular region represents the center of the convolution kernel, while the other triangles represent the neighborhood. Each small triangle corresponds to a learnable weight. The arrangement of the convolution kernel weight indices at different positions follows the SMSIR resampling method. The convolution operation on the sphere is defined as follows:

[0087]

[0088] Among them, X c,m Let X represent the feature of the m-th neighbor (m = 1…k), with center point c. c,m ∈N(x c ), N(x c ) represents the neighborhood of c, W m The y represents the learnable weights of the convolution kernel, <,> represents the inner product operation, and y c The output feature corresponds to the center of the input feature;

[0089] (2.3) Slide according to the index order, with a step size of 4. n , where n represents the number of downsampling operations. Each downsampling operation halves the resolution relative to the original resolution. During image decoding, in the classic deep learning framework that embeds transposed convolution, the transposed convolution parameters are modified to achieve image resolution upsampling through a learning-based method.

[0090] (2.4) The leaf nodes of any level of subtree are arranged in ascending order of index value to form a sequence. The spherical subtree is used as a window to limit the calculation of the transformer within the tree structure window, thereby constructing the spherical transformer model.

[0091] The transformation modeling module is configured to integrate spherical transformations with spherical convolution and self-attention.

[0092] Modeling is performed using the bit rate of the spherical entropy model.

[0093] Preferably, in the image resampling module, the spherical image is divided into eight spherical triangles, and then the midpoints of the three sides of each spherical triangle are calculated; these midpoints are connected by great circle arcs to generate four smaller spherical triangles; the indexes of the spherical triangle pixels are sorted based on the center, top / bottom, left, and right positions; this quadtree subdivision process is recursively applied to generate smaller spherical triangles until the maximum subdivision level is reached; the hierarchical index structure of the spherical triangle pixels generated on the sphere in the recursive manner constitutes an index structure based on an ordered root tree, and after SMSIR resampling, the sphere is divided into approximately uniform spherical triangle pixels.

[0094] Preferably, in the design module, the number of pixels in the spherical image after SMSIR resampling is represented as 8×4. n Where n represents the division level; assuming the shape of the input feature map F1 on the sphere is represented as...

[0095] [C in ,1,8×4 n ], where C in This represents the number of channels in F1; for each pixel, twelve neighbors are found to obtain feature map F2, whose shape is [C]. in N,8×4 n ], where N represents the kernel size; transforming F2 yields a shape of [C in ,1,N×8×4 n Feature map F3, using shape [C] out C in The learnable convolutional weights [1, N] are convolved with the feature map F3 in a 2D manner to obtain a spherical output feature map F. o Its shape is [C out ,1,8×4 n ], where C out Indicates the number of output channels;

[0096] The components of a spherical transformer include:

[0097] Block embedding: Spherical triangular pixels are represented as a one-dimensional sequence according to their index order, and each element in the sequence... The input sequence is ultimately embedded into the vector representation S. l-1 =R N×d In the diagram, N is the sequence length and d is the model dimension, which serve as the input to the transformer.

[0098] S l-1 = [E(x1); E(x2); ...; E(x n (2)

[0099] in Embedding is represented by a linear mapping function;

[0100] Window-based multi-head self-attention: Given an embedded sequence, divide the sequence into subtree windows, with a window size of 4. n Where n is an integer, the spherical patch is divided into four windows, each window being 4 in size, and for the local window feature S l-1 ∈R N×d The query matrix Q, key matrix K, and value matrix V are calculated as follows:

[0101] Q = Sl-1 W Q K = S l-1 W K V = S l-1 W V (3)

[0102] W Q W K W V It is a projection matrix shared between different windows. For multi-head attention calculation, the shapes of Q, K, and V are first deformed into... Where h is the number of heads in the multi-head attention mechanism, the self-attention calculation process within the local window is as follows:

[0103]

[0104] Where B is a learnable positional encoding, W-MSA refers to window multi-head self-attention, and LN represents layer normalization;

[0105] Feedforward Network Layers: Feedforward neural networks primarily utilize multilayer perceptrons (MLPs), consisting of linear layers and activation layers. They perform nonlinear transformations and feature extraction on the attention output features, thereby enhancing the model's representational capabilities.

[0106]

[0107] The method of the present invention will be described in more detail below.

[0108] Step 1: Spherical Image Resampling Based on Spherical Measure

[0109] A spherical image is an image signal defined on the surface of a sphere. Processing spherical images is difficult due to their geometric properties. To facilitate handling this structure, spherical images are typically projected as equal rectangular projection (ERP) images. ERP images are obtained by unfolding the spherical image along meridians and parallels. In an ERP image, each latitude circle has the same number of sampled pixels. Therefore, as the ERP image moves from the equator to the poles, the sampling points become increasingly dense, distortion increases significantly from the equator to the poles, and redundant information exists near the poles. Directly using learning-based planar image compression methods to process ERP images is unsuitable. Therefore, SMSIR is used to uniformly sample the spherical image. For ease of visualization and subsequent description, the spherical resampling process is represented as a plane, as shown below. Figure 1As shown. Specifically, this method divides the spherical image into eight spherical triangles, then calculates the midpoints of the three sides of each spherical triangle. These midpoints are connected by great circle arcs to generate four smaller spherical triangles. The indices of the spherical triangle pixels are sorted based on their center, top / bottom, left, and right positions. This quadtree subdivision process is recursively applied to generate smaller spherical triangles until the maximum subdivision level is reached. The hierarchical index structure of the spherical triangle pixels generated recursively on the sphere constitutes an index structure based on an ordered root tree. After SMSIR resampling, the sphere is divided into approximately uniform spherical triangle pixels.

[0110] Step 2: Design of basic components for spherical image learning

[0111] 1. Obtaining spherical blocks

[0112] Spherical images have extremely high resolution, making it impractical to use an entire spherical image as input to a neural network due to the enormous hardware resources required. Therefore, it is necessary to perform a patching operation on the spherical image, where each patch represents a region of the image. However, randomly cropping fixed-size patches (e.g., 256×256) from an SMSIR image destroys its hierarchical structure. This destruction hinders subsequent spherical convolutions and hierarchical upsampling / downsampling operations. To preserve the hierarchical structure during learning, the size of the spherical patch must be 4. n , where n is an integer less than the classification level. For example... Figure 1 As shown in the second level of subdivision, assuming that the block size defined on the sphere is 2×2, then this spherical block corresponds to a block of four different colored pixels on the sphere.

[0113] 2. Spherical Convolution

[0114] Spherical convolution is a core operation in spherical learning, and its key lies in the design of the convolution kernel. For example... Figure 1 As shown, the spherical image representation of this invention is based on a spherical quadtree triangular recursive subdivision structure. If a three-neighborhood convolutional kernel based on a quadtree is used, the receptive field of the kernel will be too small. Therefore, a twelve-neighborhood spherical convolutional kernel is designed to expand the neighborhood, as shown... Figure 2 As shown. The size of this spherical convolution kernel is similar to that of a 3×3 convolution kernel commonly used in planar images. In the proposed spherical convolution kernel, the central triangular region represents the center of the kernel, while the other triangles represent the neighborhood. Each small triangle corresponds to a learnable weight. The arrangement of the kernel weight indices at different positions follows the SMSIR resampling method. The convolution operation on the sphere is defined as follows:

[0115]

[0116] Among them, X c,mLet X represent the feature of the m-th neighbor (m = 1…k), with its center point c, i.e., X. c,m ∈N(x c ), N(x c ) represents the neighborhood of c, W m y represents the learnable weights of the convolution kernel, and <,> represents the inner product operation. c This represents the output feature, which corresponds to the center of the input feature.

[0117] Due to the non-square shape of the spherical convolution kernel and the non-Euclidean spatial structure of the SMSIR representation, directly applying existing deep learning tools for planar image convolution operations to spherical images is not feasible. To perform spherical convolution based on SMSIR using traditional 2D convolution methods, the implementation process of spherical convolution is redefined, as follows: Figure 3 As shown.

[0118] The number of pixels in the spherical image after SMSIR resampling is represented as 8×4. n , where n represents the division level. Assume the shape of the input feature map F1 on the sphere is represented as [C in ,1,8×4 n ], where C in This represents the number of channels in F1. Then, for each pixel, twelve neighbors are found to obtain feature map F2, which has the shape [C]. in N,8×4 n ], where N represents the kernel size. To utilize traditional 2D convolution, F2 is deformed to obtain a shape of [C in ,1,N×8×4 n Feature map F3. Now, we can use a shape of [C] out C in Learnable convolution weights [1, N] are used to perform a traditional 2D convolution operation on the feature map F3. The resulting spherical output feature map F o Its shape is [C out ,1,8×4 n ], where C out Indicates the number of output channels.

[0119] 3. Sampling on the spherical surface

[0120] Downsampling can reduce image resolution and computational complexity. In learning-based planar image compression, downsampling is typically achieved through convolutions with a stride greater than 2. Based on the ordered root tree index structure represented by SMSIR, the convolutions slide in index order with a stride of 4. n , where n represents the number of downsampling operations. Each downsampling operation halves the resolution relative to the original resolution. For example, in Figure 1In the second-level simplified planar triangle image, taking the first large triangle as an example, a sliding convolution with a stride of 4 is performed. The elements involved in the convolution are 0, 4, 8, and 12. After downsampling, the arrangement of pixels in the spherical triangle corresponds to the arrangement of the first-level partition. In other words, after downsampling, the structure of the current partition level is transferred to the previous partition level, thus effectively preserving the structural integrity of the spherical image.

[0121] In image decoding, upsampling is needed to increase image resolution for reconstruction. Upsampling methods typically include subpixel convolution and transposed convolution. Subpixel convolution first increases the number of channels through convolution, then increases image resolution by rearranging the channels spatially. However, this method introduces additional computational complexity while increasing the number of channels. Therefore, transposed convolution is used to increase image resolution. Using a spherical triangle pixel arrangement structure and the implementation process of spherical convolution, image resolution upsampling can be achieved by simply modifying the transposed convolution parameters within a classic deep learning framework that embeds transposed convolution, through a learning-based approach.

[0122] 4. Spherical transformer

[0123] Transformers have demonstrated remarkable success in various vision tasks, leading to the emergence of transformer-based image compression methods. However, applying transformers when processing spherical images presents several specific challenges: first, how to effectively model the embedding of spherical images; and second, how to implement a spherical transformer while reducing its complexity.

[0124] Fortunately, a spherical representation method based on SMSIR is adopted, where the quadtree structure of SMSIR is described by an ordered root tree index with continuous index values. Leveraging this continuity of index values, leaf nodes of subtrees at any level are arranged in ascending order of index values ​​to form a sequence, and the transformer performs well in sequence embedding modeling. To alleviate the complexity of the transformer and maintain the hierarchical nature of the spherical structure, spherical subtrees are used as windows to constrain the computation of the transformer within the tree structure window. Based on these insights, a method is proposed as follows: Figure 4 The spherical transformer model is shown below. The components of the spherical transformer are designed as follows:

[0125] 1) Block embedding: Spherical triangular pixels are represented as a one-dimensional sequence according to their index order, and each element in the sequence... Embedded. The final input sequence is embedded into the vector representation S. l-1 =R N×d In the diagram, N is the sequence length and d is the model dimension, which serve as the input to the transformer.

[0126] S l-1 = [E(x1); E(x2); ...; E(x n (2)

[0127] in An embedding is typically represented by a linear mapping function.

[0128] 2) Window-based multi-head self-attention: Given an embedded sequence, to reduce computational complexity and preserve the hierarchical spherical structure, the sequence is divided into subtree windows. The window size is set to 4. n , where n is an integer. For example Figure 4 As shown, the spherical patch is divided into four windows, each with a size of 4. For the local window feature S... l-1 ∈R N×d The query matrix Q, key matrix K, and value matrix V are calculated as follows:

[0129] Q = S l-1 W Q K = S l-1 W K V = S l-1 W V (3)

[0130] W Q W K W V It is a projection matrix shared across different windows. For multi-head attention calculation, the shapes of Q, K, and V are first transformed into... Where h is the number of heads in the multi-head attention mechanism. The self-attention calculation process within the local window is as follows:

[0131]

[0132] Where B is a learnable positional code, W-MSA refers to window multi-head self-attention, and LN represents layer normalization.

[0133] 3) Feedforward Network Layer: Feedforward neural networks mainly use multilayer perceptrons (MLPs), which consist of linear layers and activation layers. Their main function is to perform nonlinear transformations and feature extraction on the attention output features, thereby enhancing the model's representational power.

[0134]

[0135] Step 3: Spherical image compression within the context of spherical MixFormer and spherical Transformer

[0136] 1. Spherical Mixformer

[0137] Based on the basic components of spherical learning proposed in step two, a spherical transformer is used to extract non-local features and downsampling / upsampling is performed during spherical image compression. However, in the spherical transformer, the attention of each window ignores the connections between different windows, resulting in a limited receptive field. Convolution can capture the local contextual features of an image. Therefore, a simple method is proposed, namely, using spherical convolution and spherical self-attention in parallel to increase the connections between different windows and expand the receptive field. The local information extracted by spherical convolution can also serve as a useful supplement to attention-based modeling of non-local information. Therefore, a spherical transformer module integrating spherical convolution and self-attention, called Spherical MixFormer, is proposed, such as... Figure 5 As shown.

[0138] Assuming the input feature F of the spherical MixFormer has a size of L×C, a linear mapping function is first applied to F to obtain a feature of size L×C, where L represents the feature length and C represents the number of feature channels. Then, the output feature is uniformly divided into two feature maps along the channel dimension, F. sattn and F scnn Their dimensions are all L×C / 2. Next, for F... sattn Layer normalization (LN) and spherical window multi-head self-attention (SW-MSA) are applied to obtain features. At the same time, leveraging the convolution-friendly properties of batch normalization (BN), F scnn By applying Batch Normalization (BN) and spherical convolution, features are obtained. Then, and The concatenation of features is used as input to a linear function to facilitate feature fusion, and this process produces an F function of shape L×C. sca Features are then used to facilitate connections between windows, thereby expanding the receptive field. Finally, after applying another LN and a multilayer perceptron (MLP), the features are passed out. The entire process of the spherical MixFormer module can be represented as follows:

[0139] F scnn, F sattn =Channel_split(Linear(F)) (7)

[0140]

[0141] F out =MLP(LN(F sca (11)

[0142] The proposed spherical MIXFormer module, as a powerful nonlinear transformation module to enhance feature extraction, is integrated into the principal analysis transformation, principal composition transformation, super analysis transformation, and super composition transformation.

[0143] 2. Bitrate modeling using the spherical entropy model

[0144] In image compression models, bitrate is the cross-entropy between the true and estimated distributions of latent variables. The more accurate the estimation of the latent variable distribution, the lower the bitrate. The estimation accuracy of the latent variable distribution can be improved by establishing a suitable contextual entropy estimation model. Therefore, the bitrate modeling for the spherical entropy model is as follows.

[0145] 1) Using super-prior bitrate modeling

[0146] Super-prior Modeling is performed using a nonparametric, fully decomposed density model, as described below:

[0147]

[0148] Where, ψ i Represents the distribution of each univariate. The parameters. Each element in Modeled as a Gaussian distribution with mean μ i and standard deviation σ i All are derived from prior knowledge predict:

[0149]

[0150] and The bitrate is calculated as follows:

[0151]

[0152] 2) Bitrate modeling using a combination of super-prior and spherical transformer context model

[0153] In the context of classic learning-based planar image compression, with Figure 6 (a) For example, the dark gray element represents the element to be encoded or decoded. Elements before the element in the dashed box are decoded elements, and elements after it are undecoded elements. Context modeling can be performed by establishing a 2D 3×3 masked convolution centered on a dark gray element. For spherical images, in order to establish a context model similar to masked convolution in planar images, based on... Figure 2 The convolutional kernels in the model construct a context model based on an ordered tree root index structure, such as... Figure 6As shown in (b), the encoding order based on the ordered tree root index cannot match the local support regions of the fixed masked spherical convolution kernel at different encoding positions. Therefore, it becomes difficult to establish the masked spherical convolution context at different positions. The flexibility of the spherical transformer proposed in step two enables the use of contexts that conform to the encoding order. In the self-attention mechanism of the spherical transformer, a mask is introduced in equation (4) to model the spherical transformer context model, and its formula is as follows:

[0154]

[0155] Here, M represents a spherical mask, with the lower triangular portion set to 1 and all other values ​​set to negative infinity. ⊙ represents the Hadamard product. Taking a four-pixel spherical block as an example, assuming the element to be predicted is 3, ... Figure 6 As shown in (c), its corresponding spherical mask self-attention map is as follows: Figure 6 As shown in (d). From Figure 6 (d) It can be seen that the attention of the pixel with spherical index 3 can only involve pixels with indices 0, 1 and 2.

[0156] By utilizing the spherical context through autoregressive methods, the conditional Gaussian distribution for modeling entropy can be derived as follows:

[0157]

[0158] in, Representation of feature map The previous causal elements are sorted by their ordered root tree indices. μ i and σ i From these causal elements and prior knowledge The estimate is based on the data provided.

[0159] Through the three steps described above, a learning-based overall framework for spherical compression is established. Figure 7 As shown.

[0160] 1. Experimental Data: Considering the current lack of high-resolution publicly available spherical image datasets for spherical image compression, this invention utilizes 2000 high-resolution spherical images from the Refer360 dataset as the compression dataset. To further expand the dataset, 4786 high-quality images with a resolution of 4096×2048 were collected from the Internet. By combining these two datasets, the compression model was optimized.

[0161] 2. Metrics: Spherical image quality assessment uses SPSNR and WSPSNR as target quality assessments. The two types of measurement methods include rate distortion (RD) and bit rate saving (BD-BR).

[0162] 3. Experimental Setup: The Adam optimizer was selected to optimize the objective function. In all experiments, the learning rate was set to 0.0001, the batch size was set to 8, and the training was conducted for a total of 400 epochs. After 300 epochs, the learning rate was reduced to 0.00001.

[0163] 4. Comparison with different compression methods: This method is compared with several traditional codecs and learning-based methods. Traditional codecs include JPEG, JPEG2000, and BPG. Among the learning-based methods, it is compared with Balle18, Minnen18 planar image compression methods, and OSLO, a spherical image compression method based on super-prior.

[0164] Table 1 compares this compression method with planar compression methods, and calculates BD-BR and BD-distortion metrics using RATE-SPSNR and RATE-WSPSNR. The number of resampled pixels is 8×4. 9 (2048×1024). The BPG method was used as the baseline method. For RATE-SPSNR, this method improved the S-PSNR by 0.2 dB compared to the BPG method and achieved a 13.85% reduction in average bit rate. As for RATE-WSPSNR, this method improved the WS-PSNR of BPG by 0.95 dB and achieved a 17.49% reduction in average bit rate, achieving the best results compared to other methods. Table 2 compares the spherical learning compression methods under different sampling numbers. When the number of partitions n equals 9, the number of resampled pixels in this method is less, resulting in a lower RATE-SPSNR metric than the OSLO method. When n equals 10, both RATE-SPSNR and RATE-WSPSNR metrics are significantly improved.

[0165] Table 1 compares the performance of the planar compression method with the BD-SPSNR, BD-WSPSNR, and BD-BR metrics (bold text indicates best performance).

[0166]

[0167] Table 2 compares the performance of the spherical compression method with the BD-SPSNR, BD-WSPSNR, and BD-BR metrics (bold text indicates best performance).

[0168]

[0169]

[0170] 5. Ablation Experiment: When the pixel resampling number is 8×4 9In this study, ablation analysis was conducted on the proposed spherical MixFormer module and spherical transformer context model. BD-SPSNR, BD-WSPSNR, and BD-RATE were calculated using a Balle18 model trained on ERP images as a baseline. In the MixFormer module, spherical convolution is combined with the self-attention of the spherical Transformer. The performance of different modules of the proposed method is shown in Table 3.

[0171] As shown in Table 3, the advantages of spherical transformers and spherical convolutions are complementary. Adding a spherical transformer context model to the spherical MixFormer significantly improves performance, demonstrating the effectiveness of the proposed spherical Transformer context model. Overall, compared to the Balle18 method, the proposed method saves an average of 16.65% of the bit rate in RATE-SPSNR and 24.42% in RATE-WSPSNR.

[0172] Table 3 compares the performance of ablation methods on BD-SPSNR, BD-WSPSNR, and BD-BR indices (bold text indicates best performance).

[0173]

[0174] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A spherical image compression method based on a spherical deep neural network, characterized in that: The method includes the following steps: (1) Spherical image resampling based on spherical measure (SMSIR) performs uniform sampling of spherical images; (2) Design the basic components for spherical image learning, including the following steps: (2.1) Perform a block extraction operation on the spherical image, where each block represents a region of the image, and the size of the spherical block is 4. n , where n is an integer less than the classification level; (2.2) Design a 12-neighborhood spherical convolution kernel for neighborhood expansion. In this spherical convolution kernel, the central triangular region represents the center of the convolution kernel, while the other triangles represent the neighborhood. Each small triangle corresponds to a learnable weight. The arrangement of the convolution kernel weight indices at different positions follows the SMSIR resampling method. The convolution operation on the sphere is defined as follows: Among them, X c,m Let X represent the feature of the m-th neighbor (m = 1…k), with center point c. c,m ∈N(x c ), N(x c ) represents the neighborhood of c, W m The y represents the learnable weights of the convolution kernel, <,> represents the inner product operation, and y c The output feature corresponds to the center of the input feature; (2.3) Slide according to the index order, with a step size of 4. n , where n represents the number of downsampling operations. Each downsampling operation halves the resolution relative to the original resolution. In the image decoding process, in a typical deep learning framework that embeds transposed convolutions, the transposed convolution parameters are modified to achieve image resolution upsampling through a learning-based method. (2.4) The leaf nodes of any level of subtree are arranged in ascending order of index value to form a sequence. The spherical subtree is used as a window to limit the calculation of the transformer within the tree structure window, thereby constructing the spherical transformer model. (3) By integrating spherical convolution and self-attention spherical transformation, the code rate of the spherical entropy model is used for modeling.

2. The spherical image compression method based on a spherical deep neural network according to claim 1, characterized in that: In step (1), the spherical image is divided into eight spherical triangles, and then the midpoints of the three sides of each spherical triangle are calculated. These midpoints are connected by great circle arcs to generate four smaller spherical triangles. The indexes of the spherical triangle pixels are sorted based on the center, top / bottom, left, and right positions. This quadtree subdivision process is recursively applied to generate smaller spherical triangles until the maximum subdivision level is reached. The hierarchical index structure of the spherical triangle pixels generated on the sphere in the recursive manner constitutes an index structure based on an ordered root tree. After SMSIR resampling, the sphere is divided into approximately uniform spherical triangle pixels.

3. The spherical image compression method based on a spherical deep neural network according to claim 2, characterized in that: In step (2.2), the number of pixels in the spherical image after SMSIR resampling is represented as 8×4. n Where n represents the division level; assuming the shape of the input feature map F1 on the sphere is represented as [C in ,1,8×4 n ], where C in This represents the number of channels in F1; for each pixel, twelve neighbors are found to obtain feature map F2, whose shape is [C]. in N,8×4 n ], where N represents the kernel size; transforming F2 yields a shape of [C in ,1,N×8×4 n Feature map F3, using shape [C] out C in The learnable convolutional weights [1, N] are convolved with the feature map F3 in a 2D manner to obtain a spherical output feature map F. o Its shape is [C out ,1,8×4 n ], where C out Indicates the number of output channels.

4. The spherical image compression method based on a spherical deep neural network according to claim 3, characterized in that: In step (2.4), the components of the spherical transformer include: Block embedding: Spherical triangular pixels are represented as a one-dimensional sequence according to their index order, and each element in the sequence... The input sequence is ultimately embedded into the vector representation S. l-1 =R N×d In the diagram, N is the sequence length and d is the model dimension, which serve as the input to the transformer. S l-1 =[E(x1);E(x2);...;E(x n )] (2) in Embedding is represented by a linear mapping function; Window-based multi-head self-attention: Given an embedded sequence, divide the sequence into subtree windows, with a window size of 4. n Where n is an integer, the spherical patch is divided into four windows, each window being 4 in size, and for the local window feature S l-1 ∈R N ×d The query matrix Q, key matrix K, and value matrix V are calculated as follows: Q=S l-1 W Q ,K=S l-1 W K ,V=S l-1 W V (3) W Q W K W V It is a projection matrix shared between different windows. For multi-head attention calculation, the shapes of Q, K, and V are first deformed into... Where h is the number of heads in the multi-head attention mechanism, the self-attention calculation process within the local window is as follows: Where B is a learnable positional encoding, W-MSA refers to window multi-head self-attention, and LN represents layer normalization; Feedforward Network Layers: Feedforward neural networks primarily utilize multilayer perceptrons (MLPs), consisting of linear layers and activation layers. They perform nonlinear transformations and feature extraction on the attention output features, thereby enhancing the model's representational capabilities.

5. The spherical image compression method based on a spherical deep neural network according to claim 4, characterized in that: In step (3), the spherical transformation integrating spherical convolution and self-attention is as follows: Assuming the input feature F has a size of L×C, applying a linear mapping function to F yields a feature map of size L×C, where L represents the feature length and C represents the number of feature channels. The output feature is then uniformly divided into two feature maps along the channel dimension, F. sattn and F scnn Their dimensions are all L×C / 2; for F sattn Apply layer normalized LN and spherical window multi-head self-attention SW-MSA to obtain features. Leveraging the convolution-friendly properties of batch normalization (BN), F scnn By applying Batch Normalization (BN) and spherical convolution, features are obtained. Will and The concatenation features are used as input to a linear function to facilitate feature fusion, and this process produces an F function of shape L×C. sca Features are used to facilitate connections between windows to expand the receptive field; after applying another LN and a multilayer perceptron (MLP), the features are passed out. The entire process is shown below: F scnn, F sattn =Channel_split(Linear(F)) (7) F out =MLP(LN(F sca )) (11)。 6. The spherical image compression method based on a spherical deep neural network according to claim 5, characterized in that: In step (3), the bitrate modeling using the spherical entropy model includes: (I) Using super-prior bitrate modeling, super-prior Modeling is performed using a density model with complete nonparametric decomposition: Where, ψ i Let p represent each univariate distribution zi |ψ i The parameters, Each element in Modeled as a Gaussian distribution with mean μ i and standard deviation σ i All are derived from prior knowledge predict: and The bitrate is calculated as follows: (II) Bitrate modeling is performed using a combination of super-prior and spherical transformer context model, as shown in the following formula: Where M represents a spherical mask, with the lower triangular portion set to 1 and all other values ​​set to negative infinity, and ⊙ represents the Hadamard product. By utilizing the spherical context through an autoregressive method, the conditional Gaussian distribution used to model entropy is derived as follows: in, Representation of feature map The previous causal elements, sorted by ordered root tree index, μ i and σ i From these causal elements and prior knowledge The estimate is as follows.

7. The spherical image compression method based on a spherical deep neural network according to claim 6, characterized in that: In step (II), when it is a spherical four-pixel block, assuming that the element to be predicted is 3, the attention of the pixel with spherical index 3 only involves the pixels with indices 0, 1 and 2.

8. A spherical image compression device based on a spherical deep neural network, characterized in that: The device includes: The image resampling module is configured from spherical image resampling based on spherical measure, and uses SMSIR to uniformly sample the spherical image; The design module, configured to design the basic components for spherical image learning, executes: (2.1) Perform a block extraction operation on the spherical image, where each block represents a region of the image, and the size of the spherical block is 4. n , where n is an integer less than the classification level; (2.2) Design a 12-neighborhood spherical convolution kernel for neighborhood expansion. In this spherical convolution kernel, the central triangular region represents the center of the convolution kernel, while the other triangles represent the neighborhood. Each small triangle corresponds to a learnable weight. The arrangement of the convolution kernel weight indices at different positions follows the SMSIR resampling method. The convolution operation on the sphere is defined as follows: Among them, X c,m Let X represent the feature of the m-th neighbor (m = 1…k), with center point c. c,m ∈N(x c ), N(x c ) represents the neighborhood of c, W m The y represents the learnable weights of the convolution kernel, <,> represents the inner product operation, and y c The output feature corresponds to the center of the input feature; (2.3) Slide according to the index order, with a step size of 4. n , where n represents the number of downsampling operations. Each downsampling operation halves the resolution relative to the original resolution. In the image decoding process, in a typical deep learning framework that embeds transposed convolutions, the transposed convolution parameters are modified to achieve image resolution upsampling through a learning-based method. (2.4) The leaf nodes of any level of subtree are arranged in ascending order of index value to form a sequence. The spherical subtree is used as a window to limit the calculation of the transformer within the tree structure window, thereby constructing the spherical transformer model. The transformation modeling module is configured to model using the bitrate of a spherical entropy model by integrating spherical convolution and self-attention spherical transformation.

9. The spherical image compression device based on a spherical deep neural network according to claim 8, characterized in that: In the image resampling module, the spherical image is divided into eight spherical triangles, and then the midpoints of the three sides of each spherical triangle are calculated. These midpoints are connected by great circle arcs to generate four smaller spherical triangles. The indexes of the spherical triangle pixels are sorted based on the center, top / bottom, left, and right positions. This quadtree subdivision process is recursively applied to generate smaller spherical triangles until the maximum subdivision level is reached. The hierarchical index structure of spherical triangle pixels generated on the sphere by recursion constitutes an index structure based on an ordered root tree. After SMSIR resampling, the sphere is divided into approximately uniform spherical triangle pixels.

10. The spherical image compression device based on a spherical deep neural network according to claim 9, characterized in that: In the design module, the number of pixels in the spherical image after SMSIR resampling is represented as 8×4. n Where n represents the division level; assuming the shape of the input feature map F1 on the sphere is represented as [C in ,1,8×4 n ], where C in This represents the number of channels in F1; for each pixel, twelve neighbors are found to obtain feature map F2, whose shape is [C]. in N,8×4 n ], where N represents the kernel size; transforming F2 yields a shape of [C in ,1,N×8×4 n Feature map F3, using shape [C] out C in The learnable convolutional weights [1, N] are convolved with the feature map F3 in a 2D manner to obtain a spherical output feature map F. o Its shape is [C out ,1,8×4 n ], where C out Indicates the number of output channels; The components of a spherical transformer include: Block embedding: Spherical triangular pixels are represented as a one-dimensional sequence according to their index order, and each element in the sequence... The input sequence is ultimately embedded into the vector representation S. l-1 =R N×d In the diagram, N is the sequence length and d is the model dimension, which serve as the input to the transformer. S l-1 =[E(x1);E(x2);...;E(x n )] (2) in Embedding is represented by a linear mapping function; Window-based multi-head self-attention: Given an embedded sequence, divide the sequence into subtree windows, with a window size of 4. n Where n is an integer, the spherical patch is divided into four windows, each window being 4 in size, and for the local window feature S l-1 ∈R N ×d The query matrix Q, key matrix K, and value matrix V are calculated as follows: Q=S l-1 W Q ,K=S l-1 W K ,V=S l-1 W V (3) W Q W K W V It is a projection matrix shared between different windows. For multi-head attention calculation, the shapes of Q, K, and V are first deformed into... Where h is the number of heads in the multi-head attention mechanism, the self-attention calculation process within the local window is as follows: Where B is a learnable positional encoding, W-MSA refers to window multi-head self-attention, and LN represents layer normalization; Feedforward Network Layers: Feedforward neural networks primarily utilize multilayer perceptrons (MLPs), consisting of linear layers and activation layers. They perform nonlinear transformations and feature extraction on the attention output features, thereby enhancing the model's representational capabilities.

Images