An image compression method and system for feature preprocessing based on feature stacking attention

Abstract

The present invention relates to an image compression method and system for feature preprocessing based on feature stacking attention, including: Feature preprocessing: segmenting the original input image; extracting features from the original input image and sub-images; performing stacking to obtain new features; performing refinement to obtain the final feature to be encoded; Encoding, including: first obtaining latent features through a basic encoder, sending them into a hyperprior encoder to obtain hyperprior latent features, and quantizing the latent features and hyperprior latent features respectively; Decoding, including: successively passing through entropy encoding and entropy decoding to obtain new latent features and new hyperprior latent features, obtaining the hyperprior of the hyperprior latent features through a hyperprior decoder; generating decoded features through a basic decoder; Reconstruction: the decoded features are operated through the inverse process of feature preprocessing to obtain reconstructed features, realizing image compression. The present invention effectively enhances the non-linear expression ability of the network and further eliminates redundant information.

Description

Technical Field

[0001] The present invention relates to an image coding technology, in particular to an image compression method and system for feature preprocessing based on feature stacking attention, belonging to the technical field of image processing. Background Art

[0002] With the continuous growth of people's demand for visual information transmission, as one of the most important types of multimedia data, images are vivid and expressive, which well suits the characteristics of human visual perception and play an indispensable role in people's daily lives. To solve the problems of transmission and storage caused by the growth of massive image data, image data compression has always been a concern in both the academic and industrial fields and is one of the core research contents in the field of image processing. In recent years, learning-based lossy image compression technology has achieved remarkable results and outperformed traditional methods such as JEPG and BPG in terms of improving rate-distortion performance. These traditional image coding methods consist of transformation, quantization, and entropy coding. The main techniques include discrete cosine transform (DCT) and discrete wavelet transform (DWT), context-based variable length coding (CAVLC), and context-based binary arithmetic coding (CABAC). These traditional modules are based on manual design and are difficult to optimize jointly. However, the deep learning-based image compression algorithm (LIC) can achieve end-to-end training to optimize the entire network. The latest LIC algorithm has surpassed the VVC method in terms of PSNR and MS-SSIM metrics. VVC is one of the best hand-designed image and video coding standards currently. All these fully demonstrate the broad prospects of learning-based image compression methods in future applications.

[0003] Currently, the LIC method based on the variational autoencoder (VAE) structure has been widely applied and usually focuses on two aspects of exploration: how to learn a more accurate probability model and how to learn a more compact latent representation. Early works adopted relatively simple entropy models with limited performance. Currently, the main technologies use Context-adaptive techniques to improve the entropy model. The Google team integrated an autoregressive model and a hyperprior model (see Minnen D, Ballé J, Toderici G D. Joint autoregressive and hierarchical priors for learned image compression[J]. Advances in neural information processing systems, 2018, 31.) to obtain more accurate Gaussian distribution parameters for estimating the latent feature distribution, but this serial approach greatly limits the encoding efficiency. SenseTime proposed a checkerboard context model to improve the entropy model in a parallel manner (see He D, Zheng Y, Sun B, et al. Checkerboard context model for efficient learned image compression[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021:14771-14780.), achieving a good balance between encoding performance and efficiency. Inspired by parallel probability estimation research, Nanjing University proposed a multi-stage context model (see Lu M, Ma Z. High-efficiency lossy image coding through adaptive neighborhood information aggregation[J]. arXiv preprint arXiv:2204.11448, 2022.), achieving a better balance between performance and complexity. For feature extraction, CNN has also been widely used in image coding due to its excellent feature extraction ability. New York University first proposed an image compression framework that can be trained end-to-end, using CNN to achieve non-linear transformation (see Ballé J, Laparra V, Simoncelli E P. End-to-end optimized image compression[J]. arXiv preprint arXiv:1611.01704, 2016.).Subsequently, they proposed an encoding network based on variational autoencoders, which captures the spatial redundancy of latent features through a hyperprior (see Ballé J, Minnen D, Singh S, et al. Variational image compression with a scale hyperprior[J]. arXiv preprint arXiv:1802.01436, 2018.). Waseda University used a discrete Gaussian mixture model to improve entropy coding and introduced an attention mechanism into the network structure (see Cheng Z, Sun H, Takeuchi M, et al. Learned image compression with discretized gaussian mixture likelihoods and attention modules[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020:7939-7948.). In recent years, with the popularity of self-attention, VIT has also received extensive attention. The Qualcomm team constructed a non-linear transformation using a Swim transformer (see Zhu Y, Yang Y, Cohen T. Transformer-based transform coding[C] / / International Conference on Learning Representations. 2022.). Nanjing University used a neural transformation unit (NTU) to extract features based on the VAE architecture (see Lu M, Guo P, Shi H, et al. Transformer-based image compression[J]. arXiv preprint arXiv:2111.06707, 2021.), and then they further adopted an integrated convolutional and self-attention unit to achieve context-adaptive transformation.

[0004] Previous work has tried to introduce deeper networks, attention modules, or invertible structures into the VAE-based basic codec. Although the introduction of these structures can improve performance to a certain extent, the feature extraction ability is still very limited, and there is still redundant information in the features. Summary of the Invention

[0005] Aiming at the deficiencies of the prior art, the present invention proposes an image compression method and system based on feature stacking attention for feature preprocessing;

[0006] Different from the existing methods that directly encode and decode the input image, the present invention proposes a method for feature preprocessing of the input image, using a feature stacking attention module to learn more effective features for encoding and decoding. Specifically, the present invention proposes a feature preprocessing module (FPM) to segment the input image instead of directly encoding the input image. The obtained sub-images have intra-frame redundancy and inter-frame redundancy. The feature extraction module is used to extract preliminary features from these sub-images, and then the constructed feature stacking attention module (FSAM) is used to further extract the preliminary features. First, the features of the sub-images and the features of the original image are stacked, and then an attention refinement module (ARB) based on a cascaded three-dimensional convolutional residual block is used to capture their correlations in the channel, intra-sub-image frame, and inter-sub-image frame dimensions. Through this method, the non-linear expression ability of the network can be effectively enhanced and redundant information can be further eliminated.

[0007] The present invention also proposes an image compression system based on feature stacking attention for feature preprocessing.

[0008] Term Explanation:

[0009] 1. LIC (Learned image compression): A deep learning-based image coding algorithm.

[0010] 2. VAE (Variational Autoencoders): A variational autoencoder is a generative model used to learn the latent representation of input data and generate new data samples.

[0011] 3. FPM (Feature Pre-processing Module): A module for feature preprocessing of the input image.

[0012] 4. FSAM (Feature Stack Attention Module): A module that further extracts features using feature stacking and attention modules.

[0013] 5. ARB (Attention Refinement Block): A feature refinement module composed of an attention mechanism and a residual network.

[0014] 6. ICSA (Integrated Convolution and Self-Attention): An integrated convolution and self-attention unit, composed of a convolutional layer and multiple residual block neighborhood attention blocks, dynamically representing the feature information of the input.

[0015] 7. MCM (Multi-stage Context Model): Multi-stage Context Model, which gradually uses available neighborhood information in the given spatial channel order to perform probability estimation in parallel.

[0016] 8. Q (Quantization): Quantization step, the process of mapping the continuous values of a signal to discrete values, which is an important step in the image coding process.

[0017] 9. AE / AD (Arithmetic Encoding and Arithmetic Decoding): Arithmetic Encoding / Arithmetic Decoding, which writes the quantized features into the bitstream at the encoding end and decodes them at the decoding end through entropy coding.

[0018] The technical solution of the present invention is as follows:

[0019] An image compression method based on feature stacking attention for feature preprocessing, including:

[0020] Feature preprocessing, including: segmenting the original input image to obtain a number of sub-images; using the method of convolutional stacking to extract features from the original input image and the sub-images respectively to obtain the original input image features and sub-image features; stacking the original input image features and sub-image features to obtain new features; refining the new features to obtain the final features to be encoded;

[0021] Encoding, including: the final features to be encoded first pass through a basic encoder to obtain latent features, the latent features are further sent to a hyperprior encoder to obtain hyperprior latent features, and the latent features and hyperprior latent features are quantized respectively;

[0022] Decoding, including: for the quantized latent features and hyperprior latent features, successively passing through entropy encoding and entropy decoding to obtain new latent features and new hyperprior latent features, the new hyperprior latent features pass through a hyperprior decoder to obtain the hyperprior of the hyperprior latent features; the new latent features pass through a basic decoder to generate decoded features;

[0023] Reconstruction: The decoded features are operated through the inverse process of feature preprocessing to obtain reconstructed features, realizing image compression.

[0024] Preferably according to the present invention, using the method of convolutional stacking to extract features from the original image and the sub-images respectively to obtain the input image features and sub-image features, including: respectively passing the sub-image S t through two convolutional layers to obtain the corresponding sub-image features, denoted as f St; The convolutional kernels of these two convolutional layers are both 3×3 in size and the stride is 1; t = 1, 2, 3, 4. In addition, a convolution with a convolutional kernel of 5×5 and a stride of 1 and a convolution with a convolutional kernel of 3×3 and a stride of 2 are successively used to extract the features of the original input image x, obtaining the original input image features, denoted as f x .

[0025] According to a preferred embodiment of the present invention, the original input image features and the sub-image features are stacked to obtain new features, including:

[0026] Adding a time dimension, performing a feature stacking operation on the sub-image features; adding the branch of f x to construct a new feature Specifically, as shown in formula (I):

[0027]

[0028] In formula (I), represents the feature stacking operation.

[0029] According to a preferred embodiment of the present invention, the new features are refined to obtain the final features to be encoded, including:

[0030] The feature f′ passes through a three-dimensional convolution with a convolutional kernel of 5×5×5 and a three-dimensional convolution with a convolutional kernel of 3×3×3 to extract features;

[0031] Using the PReLU activation function and the skip connection structure, the extracted feature f″ is sent to the feature attention block (AB).

[0032] In the feature attention, first, the feature f″ is sent to three cascaded three-dimensional convolutional residual blocks (3D Res). Each three-dimensional convolutional residual block includes two cascaded residual blocks, using PReLU as the activation function and using the skip connection to obtain the feature f 3dr ;

[0033] Secondly, the feature f 3dr passes through a channel attention block (CA) with global residuals, and the output feature is added to the feature f″ to obtain the feature f ca ;

[0034] Then, the feature f ca passes through a spatial attention block (SA) to obtain the output feature f sa ;

[0035] Finally, the feature f sa passes through a feature attention block (Attention) to generate the feature f att, The feature attention block includes three branches, uses cascaded 3D convolutional residual blocks for feature extraction, and then changes the dimension of the features output by the feature attention block to W / 2×H / 2×5C to obtain the final encoded feature f after refinement processing.

[0036] Further preferably, the 3D convolutional residual block includes three different network structures. The first way is that the 3D convolutional residual block includes one residual block; the second way is that the 3D convolutional residual block includes two residual blocks formed by skip connections; the third way is that the 3D convolutional residual block includes three residual blocks connected in series and formed by skip connections; a module composed of two 3D convolutions using PReLU as the activation function and skip connections is called a residual block (RB).

[0037] According to the preference of the present invention, the final encoded feature first passes through the basic encoder to obtain the latent feature, and the latent feature is further sent to the hyperprior encoder to obtain the hyperprior latent feature, and the latent feature and the hyperprior latent feature are respectively quantized; the specific implementation formulas are shown in formulas (II), (III), (IV), (V), (VI), (VII):

[0038] y = g a (f; θ me ) (II)

[0039]

[0040]

[0041] z = h a (y; θ he ) (V)

[0042]

[0043]

[0044] In formulas (II), (III), and (IV), the encoded feature f passes through the basic encoder g a , to generate the latent feature y; is the result after quantization of y, Q represents the quantization operation, passes through the basic decoder g s to obtain the decoded feature θ me and θ md respectively represent the network parameters of the basic encoder and the basic decoder;

[0045] In formulas (V), (VI), and (VII), the latent feature y passes through the hyperprior encoder h a, generate the hyperprior latent feature z; is the result after quantization of z; the output of the hyperprior decoder h s is used to solve the probability distribution under conditions θ he and θ hd respectively represent the network parameters of the hyperprior encoder and the hyperprior decoder.

[0046] Further preferably, the base encoder g a includes four integrated convolutional and self-attention mechanism modules, namely ICSA, and each ICSA includes a convolutional layer (Conv) and a residual neighborhood attention block (RNAB); the base decoder g s includes four reversible integrated convolutional and self-attention mechanism modules, and the data transmission direction is opposite to that of the ICSA of the base encoder g a , and the ICSA of the base decoder g s includes a transposed convolutional layer (Tconv) and a residual neighborhood attention block (RNAB); the hyperprior encoder h a includes two ICSAs, and the hyperprior decoder h s includes two reversible ICSAs.

[0047] According to the preference of the present invention, the latent feature y and the hyperprior latent feature z are respectively quantized and entropy encoded to obtain the quantized latent feature and the quantized hyperprior latent feature The hyperprior Ψ of the latent feature is obtained through the hyperprior decoder to better predict the probability distribution of the latent feature, and the quantized latent feature generates the decoded feature through the base decoder The specific implementation is shown in Equations (II) to (VII).

[0048] According to the preference of the present invention, the decoded feature undergoes an inverse process operation of feature preprocessing to obtain the reconstructed feature, realizing image compression, including:

[0049] Perform dimensional transformation to generate Send it into the feature attention block (Attention) to generate Send it into the spatial attention block (SA) to generate Send it into the channel attention block (CA) to generate Send it into three cascaded three-dimensional convolutional residual blocks (3D Res) and the skip connection from to generate the feature Feed in a 3D convolution with a convolution kernel size of 3×3×3 and a 3D convolution with a convolution kernel size of 5×5×5 to extract features, and use the PReLU activation function and skip connection structure to obtain the reconstructed features

[0050] Perform feature rearrangement to obtain the reconstructed sub-image features and and Pass through two transposed convolution layers respectively to obtain the corresponding reconstructed sub-images and Perform reversible operations according to the image partitioning method and Obtain the reconstructed input image through the reconstruction module

[0051] Preferably according to the present invention, the loss function is expressed as Equation (VIII):

[0052]

[0053] In Equation (VIII), the bit rate R is composed of the entropy of the feature y and the entropy of the hyperprior latent feature z , and the distortion represents the reconstruction error between the input image x and the reconstructed image , and λ represents the trade-off factor for controlling rate distortion

[0054] A computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of an image compression method based on feature stacking attention for feature preprocessing

[0055] A computer-readable storage medium stores a computer program thereon. When the computer program is executed by a processor, it implements the steps of an image compression method based on feature stacking attention for feature preprocessing

[0056] An image compression system based on feature stacking attention for feature preprocessing includes:

[0057] A feature preprocessing module (FPM), configured to: segment the original input image to obtain a plurality of sub-images; respectively extract features from the original input image and the sub-images by using the method of convolution stacking to obtain the original input image features and the sub-image features; stack the original input image features and the sub-image features to obtain new features; refine the new features to obtain the final features to be encoded

[0058] The encoding module is configured to: first, the final feature to be encoded passes through a basic encoder to obtain a latent feature, and the latent feature is further fed into a hyperprior encoder to obtain a hyperprior latent feature, and the latent feature and the hyperprior latent feature are respectively quantized;

[0059] The decoding module is configured to: for the quantized latent feature and hyperprior latent feature, they sequentially pass through entropy encoding and entropy decoding to obtain a new latent feature and a new hyperprior latent feature. The new hyperprior latent feature passes through a hyperprior decoder to obtain the hyperprior of the hyperprior latent feature; the new latent feature passes through a basic decoder to generate a decoded feature;

[0060] The reconstruction module is configured to: the decoded feature undergoes an inverse process operation of feature preprocessing to obtain a reconstructed feature, thereby realizing image compression.

[0061] Further preferably, the feature preprocessing module includes an image segmentation module, a feature extraction module, and a Feature Stacking Attention Module (FSAM); the Feature Stacking Attention Module includes a feature stacking module and an attention refinement module;

[0062] The image segmentation module is configured to: segment the original input image to obtain a plurality of sub-images;

[0063] The feature extraction module is configured to: respectively extract features from the original input image and the sub-images by using a method of stacking convolutions to obtain the original input image features and the sub-image features;

[0064] The feature stacking module is configured to: stack the original input image features and the sub-image features to obtain a new feature;

[0065] The attention refinement module is configured to: refine the new feature to obtain the final feature to be encoded.

[0066] The beneficial effects of the present invention are as follows:

[0067] 1. Different from existing methods that directly encode and decode the input image, the present invention proposes an image compression method for preprocessing the features of the input image, and extracts more effective features through a feature refinement network with an attention mechanism to fully exploit the spatial correlation within the image.

[0068] 2. Compared with other methods, under the condition of the same objective quality of the decoded image, the encoding bitrate of the present invention is lower, indicating that the present invention improves the encoding efficiency of the image. Compared with the latest encoding standard H.266 / VVC, the BD-rate of the method proposed by the present invention is reduced by 4.8% and 6.4% respectively on the Kodak and CLIC datasets. Description of the Drawings

[0069] Figure 1 This is the overall flowchart of the image compression method based on feature stacking attention for feature preprocessing in the present invention;

[0070] Figure 2 This is a schematic diagram of image segmentation and reconstruction in the present invention;

[0071] Figure 3 This is a schematic diagram of refinement proposed in the present invention;

[0072] Figure 4 This is a schematic diagram of the reversible process of refinement proposed in the present invention;

[0073] Figure 5(a) is a schematic structural diagram of the three-dimensional convolutional residual block in the first cascading method proposed in the present invention;

[0074] Figure 5(b) is a schematic structural diagram of the three-dimensional convolutional residual block in the first cascading method proposed in the present invention;

[0075] Figure 5(c) is a schematic structural diagram of the three-dimensional convolutional residual block in the first cascading method proposed in the present invention;

[0076] Figure 6(a) is a schematic diagram of the rate-distortion performance comparison between the present invention and different coding schemes on the Kodak dataset;

[0077] Figure 6(b) is a schematic diagram of the rate-distortion performance comparison between the present invention and different coding schemes on the CLIC2020 dataset. Specific embodiments

[0078] The present invention will be further limited below in conjunction with the accompanying drawings of the specification and embodiments, but not limited thereto.

[0079] Embodiment 1

[0080] An image compression method based on feature stacking attention for feature preprocessing, as Figure 1 shown, includes:

[0081] Feature preprocessing, including: segmenting the original input image to obtain a plurality of sub-images; extracting features from the original input image and the sub-images respectively by using the method of convolutional stacking to obtain the original input image features and the sub-image features; stacking the original input image features and the sub-image features to obtain new features; refining the new features to obtain the final features to be encoded;

[0082] Encoding, including: the final features to be encoded first pass through the basic encoder to obtain latent features, the latent features are further sent to the hyperprior encoder to obtain hyperprior latent features, and the latent features and the hyperprior latent features are quantized respectively;

[0083] Decoding includes: successively performing entropy encoding and entropy decoding on the quantized latent features and the hyperprior latent features to obtain new latent features and new hyperprior latent features; the new hyperprior latent features pass through a hyperprior decoder to obtain the hyperprior of the hyperprior latent features; the new latent features pass through a base decoder to generate decoded features;

[0084] Reconstruction: The decoded features are operated through the inverse process of feature preprocessing to obtain reconstructed features, realizing image compression.

[0085] Embodiment 2

[0086] A method for image compression based on feature stacking attention according to Embodiment 1, which is characterized in that:

[0087] The original image is segmented to obtain a number of sub-images, as Figure 2 shown, including: for the input image x with a size of H×W×3, where H represents the height of the image and W represents the width of the image; it is divided into m×n patches, each patch contains 2×2 pixels, and use to represent the patch in the i-th row and j-th column, where i = 1,..., m, j = 1,..., n; the pixels at the corresponding positions in each patch are extracted and rearranged according to the indices i and j to obtain four sub-images Through this segmentation method, the sub-images S 1 , S 2 , S 3 and S 4 are highly correlated with each other and can be regarded as four frames with temporal correlation in a video. In addition, these sub-images completely retain the pixel information of the original input.

[0088] The convolution stacking method is used to extract features from the original image and the sub-images respectively to obtain the input image features and the sub-image features, including: respectively passing the sub-image S t through two convolutional layers to obtain the corresponding sub-image features, denoted as The kernel sizes of these two convolutional layers are both 3×3 and the stride is 1; t = 1, 2, 3, 4. In addition, a convolution with a kernel size of 5×5 and a stride of 1 and a convolution with a kernel size of 3×3 and a stride of 2 are successively used to extract the features of the original input image x to obtain the original input image features, denoted as f x .

[0089] The original input image features and the sub-image features are stacked to obtain new features, including:

[0090] Through the image segmentation module and feature extraction in FPM, five three-dimensional features f x , and It can be observed that the sub-images S 1 , S 2 , S 3 and S 4 have extremely high similarity with each other, and there is not only intra-frame redundancy but also inter-frame redundancy. In order to explore the intra-frame and inter-frame correlations, a time dimension is added, and a feature stacking operation is performed on the sub-image features; in addition, the features of the original image contribute to better extraction of spatial features. Therefore, a branch of f x is added to construct a new feature Specifically, as shown in Equation (I):

[0091]

[0092] In Equation (I), represents the feature stacking operation.

[0093] The new feature is refined to obtain the final feature to be encoded, as shown in Figure 3 and includes:[[]]

[0094] In order to better extract features effectively, the present invention constructs an attention refinement module of an attention mechanism and uses three-dimensional convolution for feature extraction.

[0095] The feature f′ is passed through a three-dimensional convolution with a convolution kernel size of 5×5×5 and a three-dimensional convolution with a convolution kernel size of 3×3×3 to extract features;

[0096] Using the PReLU activation function and the skip connection structure, the extracted feature f″ is fed into the feature attention block (AB).

[0097] In the feature attention, first, the feature f″ is fed into three cascaded three-dimensional convolutional residual blocks (3D Res). Each three-dimensional convolutional residual block includes two cascaded residual blocks, uses PReLU as the activation function and uses the skip connection to obtain the feature f 3dr ;

[0098] Secondly, the feature f 3drAfter passing through a channel attention block (CA) with global residuals (see Zhang Y, Li K, Li K, et al. Image super-resolution using very deep residual channel attention networks[C] / / Proceedings of the European conference on computer vision (ECCV). 2018: 286-301.), the output feature is added to the feature f″ to obtain the feature f ca ;

[0099] Then, the feature f ca passes through a spatial attention block (SA) (see Woo S, Park J, Lee J Y, et al. Cbam: Convolutional block attention module[C] / / Proceedings of the European conference on computer vision (ECCV). 2018: 3-19.) to obtain the output feature f sa ;

[0100] Finally, the feature f sa passes through a feature attention block (Attention) to generate the feature f att , and the feature attention block includes three branches. The three-dimensional convolutional residual block in cascade is used for feature extraction, and then the dimension of the feature output by the feature attention block is changed to W / 2×H / 2×5C to obtain the final encoded feature f after refinement processing. Among them, CA and SA pay more attention to the feature information in the channel and space, and Attention uses cascaded 3D Res to pay attention to more challenging feature information.

[0101] The three-dimensional convolutional residual block includes three different network structures. As shown in Figure 5(a), the first way is that the three-dimensional convolutional residual block includes a residual block; as shown in Figure 5(b), the second way is that the three-dimensional convolutional residual block includes two residual blocks formed by skip connections; as shown in Figure 5(c), the third way is that the three-dimensional convolutional residual block includes three residual blocks connected in series and formed by skip connections; a module composed of two three-dimensional convolutions using PReLU as the activation function and skip connections is called a residual block (RB).

[0102] The final feature to be encoded first passes through a basic encoder to obtain a latent feature, and the latent feature is further fed into a hyperprior encoder to obtain a hyperprior latent feature. Quantization is performed on the latent feature and the hyperprior latent feature respectively. The specific implementation formulas are shown in Formulas (II), (III), (IV), (V), (VI), and (VII):

[0103] y = g a (f; θ me ) (II)

[0104]

[0105]

[0106] z = h a (y; θ he ) (V)

[0107]

[0108]

[0109] In Formulas (II), (III), and (IV), the feature f to be encoded passes through the basic encoder g a , generating the latent feature y; is the result after quantization of y. Q represents the quantization operation, which is implemented by adding uniformly distributed noise during the training process and by using a rounding function during the testing process; After passing through the basic decoder g s , the decoded feature θ me and θ md respectively represent the network parameters of the basic encoder and the basic decoder;

[0110] In Formulas (V), (VI), and (VII), the latent feature y passes through the hyperprior encoder h a , generating the hyperprior latent feature z; is the result after quantization of z; the output of the hyperprior decoder h s is used to solve the probability distribution under the condition of ; θ he and θ hd respectively represent the network parameters of the hyperprior encoder and the hyperprior decoder.

[0111] The present invention adopts Tiny-LIC as the basic structure, and the basic encoder g aIt includes four integrated convolution and self-attention mechanism modules, namely ICSA. Each ICSA includes a convolutional layer (Conv) and a residual neighborhood attention block (RNAB); Consistent with the encoder, the basic decoder g s includes four reversible integrated convolution and self-attention mechanism modules, and the data transmission direction is opposite to that of the ICSA of the basic encoder g a The ICSA of the basic decoder g s includes a transposed convolutional layer (Tconv) and a residual neighborhood attention block (RNAB); The hyperprior encoder h a includes two ICSAs, and the hyperprior decoder h s includes two reversible ICSAs. The latent feature y obtained by the basic encoder g a is encoded and decoded using a multi-stage context model. While ensuring the encoding performance, the encoding efficiency is improved through parallel processing. The present invention improves the network structure of the codec based on Tiny-LIC. For the ICSA module and the corresponding reversible module in the basic codec and the hyperprior codec, on the basis of the original one Conv and RNAB, a cascaded residual module (CRM) stacked by two residual blocks is added, and PReLU is used as the activation function.

[0112] For the latent feature y and the hyperprior latent feature z, quantization and entropy coding are respectively performed to obtain the quantized latent feature and the quantized hyperprior latent feature The hyperprior Ψ of the latent feature is obtained through the hyperprior decoder to better predict the probability distribution of the latent feature. The quantized latent feature generates the decoded feature through the basic decoder The specific implementation is shown in equations (II) to (VII).

[0113] The decoded feature undergoes an inverse process operation of feature preprocessing to obtain the reconstructed feature, realizing image compression, including:

[0114] The decoded feature is fed into the reversible modules of the feature stacking attention module (FSAM) and the feature preprocessing module (FPM). The former includes the reversible modules of the feature rearrangement module and the attention refinement block (ARB).

[0115] As Figure 4 shown, the network structure of the reversible module of the attention refinement block (ARB) is the same as that of ARB, except that the convolutional layer is replaced by a transposed convolutional layer. Perform dimensional transformation to generate and feed it into the feature attention block (Attention) to generate Feed into the spatial attention block (SA) for generation Feed into the channel attention block (CA) for generation Feed into three cascaded three-dimensional convolutional residual blocks (3D Res) and the skip connection from for feature generation Feed into a three-dimensional convolution with a convolution kernel size of 3×3×3 and a three-dimensional convolution with a convolution kernel size of 5×5×5 to extract features, and use the PReLU activation function and the skip connection structure to obtain the reconstructed features

[0116] Feed into the feature rearrangement module for feature rearrangement to obtain the reconstructed sub-image features and Further feed into the reversible module of FPM and Respectively pass through two transposed convolutional layers (both with a convolution kernel size of 3×3 and a stride of 1) to obtain the corresponding reconstructed sub-images and Perform reversible operations according to the division method of the image segmentation module in FPM and Pass through the reconstruction module to obtain the reconstructed input image

[0117] The image coding problem based on deep learning can be regarded as a rate-distortion optimization problem based on Lagrange multipliers. The loss function for the training of the entire model consists of the rate R and the distortion D, and the two restrict and contradict each other. Therefore, the rate-distortion optimization technology is used to minimize the loss function, and the loss function is expressed as Equation (VIII):

[0118]

[0119] In Equation (VIII), the rate R consists of the entropy of the feature y and the entropy of the hyperprior latent feature z. The distortion represents the reconstruction error between the input image x and the reconstructed image , and λ represents the trade-off factor for controlling rate-distortion.

[0120] The effects of the present invention can be further illustrated by experiments.

[0121] The experimental tests compared the rate-distortion (RD) performance of the method of the present invention with different coding methods, including traditional coding standards and the latest LIC method. Figures 6(a) and 6(b) respectively compare the RD performance under the Kodak dataset and the CLIC dataset. It can be seen that, compared with other methods, under the condition of the same objective quality of the decoded image, the coding bit rate of the present invention is lower, indicating that the present invention improves the coding efficiency of images. Compared with the latest coding standard H.266 / VVC, the BD-rate of the method proposed by the present invention is reduced by 4.8% and 6.4% respectively on the Kodak and CLIC datasets.

[0122] Embodiment 3

[0123] A computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it implements the steps of the image compression method based on feature stacking attention in Embodiment 1 or 2.

[0124] Embodiment 4

[0125] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the steps of the image compression method based on feature stacking attention in Embodiment 1 or 2.

[0126] Embodiment 5

[0127] An image compression system based on feature stacking attention includes:

[0128] A feature preprocessing module (FPM), configured to: divide the original input image to obtain a plurality of sub-images; extract features from the original input image and the sub-images respectively by using the method of convolutional stacking to obtain the original input image features and the sub-image features; stack the original input image features and the sub-image features to obtain new features; refine the new features to obtain the final features to be encoded;

[0129] The feature preprocessing module includes an image segmentation module, a feature extraction module, and a feature stacking attention module (FSAM); the feature stacking attention module includes a feature stacking module and an attention refinement module;

[0130] The image segmentation module is configured to: divide the original input image to obtain a plurality of sub-images;

[0131] The feature extraction module is configured to: extract features from the original input image and the sub-images respectively by using the method of convolutional stacking to obtain the original input image features and the sub-image features;

[0132] A feature stacking module, configured to: stack the original input image features and the sub-image features to obtain new features;

[0133] An attention refinement module, configured to: refine the new features to obtain the final features to be encoded.

[0134] Input the input image x into the feature preprocessing module (FPM). Four sub-images S 1 , S 2 , S 3 and S 4 are obtained through the image segmentation module, and the corresponding sub-image features are respectively obtained by using the feature extraction module and In the feature stacking attention module (FSAM), stack the sub-image features and the input image feature f x to obtain new features f′, and use the attention refinement module (ARB) to refine f′ to obtain the final features to be encoded f;

[0135] An encoding module, configured to: first pass the final features to be encoded through a basic encoder to obtain latent features, and further send the latent features into a hyperprior encoder to obtain hyperprior latent features, and respectively quantize the latent features and the hyperprior latent features;

[0136] Send the features to be encoded f into the basic encoder (Codec). During the encoding process, first pass through the basic encoder g a to obtain latent features y, g a is composed of four integrated convolutional and self-attention mechanism modules (ICSA), where each ICSA includes a convolutional layer, a cascaded residual block, and a residual neighborhood attention block (RNAB). y is further sent into the hyperprior encoder h a to obtain hyperprior latent features z, h a is composed of two ICSAs. Quantize the features y and z respectively, and a multi-stage context model is used to perform entropy coding on y.

[0137] A decoding module, configured to: for the quantized latent features and hyperprior latent features, sequentially perform entropy coding and entropy decoding to obtain new latent features and new hyperprior latent features, and the new hyperprior latent features pass through a hyperprior decoder to obtain the hyperprior of the hyperprior latent features; the new latent features pass through a basic decoder to generate decoded features;

[0138] After quantization, the features y and z are subjected to entropy coding and entropy decoding to obtain and Pass through the hyperprior decoder h s to obtain the hyperprior ψ of the feature y to assist the MCM in performing entropy coding and entropy decoding. After the basic decoder g s Generate the decoded features To match the encoder, g s and h s also include four ICSAs and two ICSAs respectively. Operating through the inverse process of FSAM, the network structure of ARB is the same as that of the encoding end to obtain the reconstructed features

[0139] The reconstruction module is configured to: the decoded features are operated through the inverse process of feature preprocessing to obtain the reconstructed features, realizing image compression.

[0140] Perform feature rearrangement to obtain the reconstructed sub-image features and Feed them into the inverse process of FPM, perform reversible operations according to the division method of the segmentation module, and the reconstructed sub-images and Pass through the reconstruction module to obtain the reconstructed input image Different from the encoder, in the network structure of the decoding process, the convolutional layer is replaced by the transposed convolutional layer.

Claims

1. An image compression method, characterized in that, comprising: Feature preprocessing, including: segmenting the original input image to obtain a number of sub-images S t ; using the method of convolutional stacking to extract features from the original input image and sub-images respectively to obtain the original input image feature f x and sub-image features Stack the original input image feature and sub-image features to obtain a new feature f'; refine the new feature to obtain the final feature to be encoded; Encoding, including: the final feature to be encoded first passes through a basic encoder to obtain a latent feature, the latent feature is further fed into a hyperprior encoder to obtain a hyperprior latent feature, and the latent feature and the hyperprior latent feature are respectively quantized; Decoding, including: for the quantized latent feature and hyperprior latent feature, successively passing through entropy encoding and entropy decoding to obtain a new latent feature and a new hyperprior latent feature, the new hyperprior latent feature passes through a hyperprior decoder to obtain the hyperprior of the hyperprior latent feature; the new latent feature passes through a basic decoder to generate a decoded feature; Reconstruction: the decoded feature undergoes an inverse process operation of feature preprocessing to obtain a reconstructed feature, realizing image compression; Refining the new feature to obtain the final feature to be encoded, including: The feature f′ passes through a 5×5×5 three-dimensional convolution, a PReLU activation function, a 3×3×3 three-dimensional convolution, and a skip connection structure to extract the obtained feature f″; The feature f″ is fed into three cascaded three-dimensional convolutional residual blocks to obtain the feature f 3dr ; Secondly, feature f 3dr Passes through a channel attention block with global residuals, and the output feature is added to feature f″ to obtain feature f ca ; Then, feature f ca passes through a spatial attention block to obtain the output feature f sa ; Finally, feature f sa is generated through a feature attention block att . The feature attention block includes three branches, uses cascaded 3D convolutional residual blocks for feature extraction, and then changes the dimension of the feature output by the feature attention block to W / 2×H / 2×5C to obtain the final encoded feature f after refinement processing; The three-dimensional convolutional residual block includes three different network structures. The first way is: the three-dimensional convolutional residual block includes a residual block; the second way is: the three-dimensional convolutional residual block includes two residual blocks formed by using skip connections; the third way is: the three-dimensional convolutional residual block includes three residual blocks connected in series and formed by using skip connections; a module composed of two three-dimensional convolutions using PReLU as the activation function and a skip connection is called a residual block.

2. An image compression method according to claim 1, characterized in that, The method of convolutional stacking is used to extract features from the original image and the sub-images respectively, obtaining the input image features and the sub-image features, including: respectively subjecting the sub-image S t to two convolutional layers to obtain the corresponding sub-image features The convolutional kernels of these two convolutional layers are both 3×3 in size and the stride is 1; t = 1, 2, 3, 4; then, in sequence, a convolution with a convolutional kernel of 5×5 in size and a stride of 1 and a convolution with a convolutional kernel of 3×3 in size and a stride of 2 are used to extract the features of the original input image x, obtaining the original input image features fx.

3. An image compression method according to claim 1, characterized in that, Stacking the original input image feature and the sub-image feature to obtain a new feature, including: Add a time dimension and perform a feature stacking operation on the sub-image features; add a branch of fx to construct a new feature Specifically, as shown in formula (I): In formula (I), represents a feature stacking operation.

4. An image compression method according to claim 1, characterized in that, The final feature to be encoded first passes through a basic encoder to obtain a latent feature, the latent feature is further fed into a hyperprior encoder to obtain a hyperprior latent feature, and the latent feature and the hyperprior latent feature are respectively quantized; the specific implementation formulas are shown in formulas (II), (III), (IV), (V), (VI), (VII): y = ga(f; θ me )(II) z = ha(y; θ he )(V) In Formula (II), Formula (III), and Formula (IV), the feature f to be encoded passes through the basic encoder ga to generate the latent feature y; is the result after quantization of y, where Q represents the quantization operation, and the decoded feature is obtained through the basic decoder gs θ me and θ md respectively represent the network parameters of the basic encoder and the basic decoder; In Formula (V), Formula (VI), and Formula (VII), the latent feature y passes through the hyperprior encoder ha to generate the hyperprior latent feature z; is the result after quantization of z; the output of the hyperprior decoder hs is used to solve the probability distribution under conditions θ he and θ hd represent the network parameters of the hyperprior encoder and the hyperprior decoder, respectively.

5. An image compression method according to claim 4, characterized in that, The basic encoder ga includes four integrated convolutional and self-attention mechanism modules, namely ICSA, and each ICSA includes a convolutional layer and a residual neighborhood attention block; the basic decoder gs includes four reversible integrated convolutional and self-attention mechanism modules, and the data transmission direction is opposite to that of the ICSA of the basic encoder g a ; the ICSA of the basic decoder g s includes a transposed convolutional layer and a residual neighborhood attention block; the hyperprior encoder h a includes two ICSA, and the hyperprior decoder h s includes two reversible ICSA.

6. An image compression method according to claim 1, characterized in that, Quantize and entropy code the latent feature y and the hyper-prior latent feature z respectively to obtain the quantized latent feature and the quantized hyper-prior latent feature The hyper-prior Ψ of the latent feature is obtained through the hyper-prior decoder to better predict the probability distribution of the latent feature, and the quantized latent feature generates the decoded feature through the base decoder The specific implementation is shown in Equations (II) to (VII).

7. An image compression method according to any one of claims 1-6, characterized in that, The decoded feature undergoes an inverse process operation of feature preprocessing to obtain a reconstructed feature, realizing image compression, including: Perform dimensional transformation to generate Feed into the feature attention block to generate Feed into the spatial attention block to generate Feed into the channel attention block to generate Feed into three cascaded 3D convolutional residual blocks and the skip connection from to generate features Feed into a 3D convolution with a kernel size of 3×3×3 and a 3D convolution with a kernel size of 5×5×5 to extract features, and use the PReLU activation function and the skip connection structure to obtain the reconstructed features Perform feature rearrangement to obtain the reconstructed sub-image features and and Respectively pass through two transposed convolutional layers to obtain the corresponding reconstructed sub-images and Perform reversible operations according to the image partitioning method and Obtain the reconstructed input image through the reconstruction module 8. An image compression method according to claim 7, characterized in that, The loss function is expressed as formula (VIII): In formula (VIII), the code rate R consists of the entropy of feature y and the entropy of the super prior latent feature z . The distortion represents the reconstruction error between the input image x and the reconstructed image . λ represents the trade-off factor for controlling the rate distortion 9. A computer device, including a memory and a processor, the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of an image compression method according to any one of claims 1-8.

10. A computer-readable storage medium, on which a computer program is stored, characterized in that, When the computer program is executed by a processor, it implements the steps of an image compression method according to any one of claims 1-8.

11. An image compression system characterized in that it includes: A feature preprocessing module, configured to: segment the original input image to obtain a plurality of sub-images; Extract features from the original input image and the sub-images respectively by using a convolutional stacking method to obtain the original input image features and the sub-image features; stack the original input image features and the sub-image features to obtain new features; refine the new features to obtain the final feature to be encoded; An encoding module, configured to: the final feature to be encoded first passes through a basic encoder to obtain a latent feature, the latent feature is further sent to a hyperprior encoder to obtain a hyperprior latent feature, and the latent feature and the hyperprior latent feature are respectively quantized; A decoding module, configured to: for the quantized latent feature and hyperprior latent feature, successively perform entropy encoding and entropy decoding to obtain a new latent feature and a new hyperprior latent feature, the new hyperprior latent feature passes through a hyperprior decoder to obtain the hyperprior of the hyperprior latent feature; the new latent feature passes through a basic decoder to generate a decoded feature; A reconstruction module, configured to: the decoded feature undergoes an inverse process operation of the feature preprocessing to obtain a reconstructed feature, thereby realizing image compression; Refining the new features to obtain the final feature to be encoded includes: The feature f′ passes through a 5×5×5 three-dimensional convolution, a PReLU activation function, a 3×3×3 three-dimensional convolution, and a skip connection structure to extract the obtained feature f″; The feature f″ is fed into three cascaded three-dimensional convolutional residual blocks to obtain the feature f 3dr ; Secondly, feature f 3dr passes through a channel attention block with global residuals, and the output feature is added to feature f″ to obtain feature f ca ; Then, the feature f ca passes through a spatial attention block to obtain the output feature f sa ; Finally, feature f sa is generated through a feature attention block att , the feature attention block includes three branches, uses cascaded 3D convolutional residual blocks for feature extraction, and then changes the dimension of the feature output by the feature attention block to W / 2×H / 2×5C to obtain the final encoded feature f after refinement processing; The three-dimensional convolutional residual block includes three different network structures. The first method is that the three-dimensional convolutional residual block includes a residual block; the second method is that the three-dimensional convolutional residual block includes two residual blocks connected by a skip connection; the third method is that the three-dimensional convolutional residual block includes three residual blocks connected in series and connected by a skip connection; a module composed of two three-dimensional convolutions using PReLU as the activation function and a skip connection is called a residual block.

12. An image compression system according to claim 11 characterized in that The feature preprocessing module includes an image segmentation module, a feature extraction module, and a feature stacking attention module; The feature stacking attention module includes a feature stacking module and an attention refinement module; The image segmentation module, configured to: segment the original input image to obtain a plurality of sub-images; The feature extraction module, configured to: extract features from the original input image and the sub-images respectively by using a convolutional stacking method to obtain the original input image features and the sub-image features; The feature stacking module, configured to: stack the original input image features and the sub-image features to obtain new features; The attention refinement module, configured to: refine the new features to obtain the final feature to be encoded.

Images