Image compression method and system based on Swin-Transformer and autoregression

Abstract

The present invention discloses an image compression method and system based on Swin-Transformer and autoregression, comprising the following steps: firstly, extracting image features using an encoder based on Swin-Transformer, and quantizing and arithmetically encoding the image features to generate a binary code stream for storage and transmission. When the image is needed, the binary code stream is arithmetically decoded, and a reconstructed image is finally obtained through a decoder based on Swin-Transformer; an image compression system is designed and implemented, and the system can use the method proposed in the present invention to compress images. The present invention improves the reconstructed image to have a better effect at a specific coding rate, and the effectiveness of the method of the present invention can be proved by test results.

Description

Technical Field

[0001] The present invention relates to the technical field of image compression based on deep learning, and in particular to an image compression method and system based on Swin-Transformer and autoregression. Background Art

[0002] With the rapid development of Internet technology in the 21st century, the information age has arrived. The ways people get information are increasing day by day, especially with the popularity of smart devices. More and more information needs to be transmitted on these devices. Data show that people get about 75% of information from the outside world through vision, and images, as the main way to represent visual information, convey information about things themselves to people in a figurative and vivid form through the intuitive expression of objective things. Therefore, images have become one of the most important ways for people to get information in their lives. When using images to transmit information, due to the large amount of redundancy inside the original image, more network resources and storage space will be occupied during the information transmission process. Therefore, the original image has not been compressed, which limits the storage and transmission of information to a certain extent. Therefore, compressing the original image can alleviate the transmission and storage pressure of the Internet, enabling people to receive information more quickly and conveniently. In recent years, more and more fields need to use images to transmit information. While digital image data brings convenience to information transmission, it also poses challenges to computer storage resources and network transmission bandwidth. The visual effect of uncompressed and compressed images is almost the same, but the memory occupied by the images is dozens of times different. If a large number of original images are stored, the current hardware storage technology will inevitably increase the pressure on the memory. Therefore, uncompressed images greatly increase the burden of daily network transmission and data storage. Therefore, how to study efficient and appropriate image compression methods, by removing redundant data in the image, thereby reducing the image storage space and increasing the image transmission rate in the network, has become the focus of research in the field of images.

[0003] Reference 1 (Han Yifei. Research on image compression algorithm based on deep learning [D]. Beijing University of Posts and Telecommunications, 2020.) is a model based on recurrent neural network. It uses LSTM units to learn and memorize the residual of predicted output and input in each iteration process to improve the expression of feature information and improve the reconstructed image effect. Reference 2 (Mentzer F, Toderici GD, Tschannen M, et al. High-fidelity generative image compression [J]. Advances in Neural Information Processing Systems, 2020, 33: 11913-11924.) uses generative adversarial network as a model to improve the reconstructed image effect.

[0004] In deep learning-based image compression tasks, most models use convolutional neural networks. Although the reconstructed image effect is improved, it also inevitably increases the amount of parameter calculations and industrial implementation. Although convolutional neural networks have made great contributions, one of their major drawbacks is that they are not designed to capture local features, which seriously affects the quality of the reconstructed image. Summary of the invention

[0005] In order to solve the problems that convolutional neural networks are not conducive to capturing local features and the large number of network parameters, the present invention proposes an image compression method based on Swin-Transformer and autoregression. The method uses Swin-Transformer as a feature extraction method; uses an autoregression method to perform Gaussian modeling on feature information; and adopts a decoupling method to model the mean and standard deviation.

[0006] An image compression method based on Swin-Transformer and autoregression, characterized in that it comprises the following steps:

[0007] S1. Collect a variety of different pictures in the natural environment, randomly divide the collected pictures into two parts in a ratio of 9:1, the first part constitutes the training set, and the second part constitutes the test set, and preprocess the training set;

[0008] S2, construct an image compression neural network; randomly horizontally flip, randomly crop and normalize the image obtained in S1, and then input it into the constructed image compression neural network for training; the loss function uses mean square error, perceptual loss and rate distortion loss function, sets hyperparameters to balance loss weights, sets optimization strategies for constructing image compression networks, and continuously reduces the loss function by iteratively training the network until the set number of iterations is completed and the training weight parameters are saved;

[0009] The constructed image compression neural network is specifically as follows: the network structure generally consists of three parts, namely Encoder, Generator and Probability Model. The Encoder sets up 8 Swin-Transformer block stacks in sequence, downsampling once every two Swin-Transformer blocks, and an arithmetic encoder; the Generator is composed of an arithmetic decoder, a 9-layer residual network stack and an 8-layer Swin-Transformer stack. The residual network does not upsample the features, but 8 Swin-Transformers complete the upsampling task, upsampling once every two Swin-Transformer blocks; the Probability Model is mainly composed of two parts. The first part is composed of an arithmetic encoder and a Hyper-Encoder, where the Hyper-Encoder is composed of 3 kernel_size=3, stride=2 convolutions, and the second part is composed of an arithmetic decoder, an autoregressive network Context Model and a Hyper-Decoder, where the Hyper-Decoder is composed of 3 kernel_size=3, stride=2 transposed convolutions and a decoupling head; the autoregressive network Context Model is composed of a 5*5 mask conv convolution;

[0010] During encoding, Encoder, Generator, and Probability Model are referred to as E, G, and P respectively. Image x is downsampled by Encoder to obtain feature y = E(x). Feature y is extracted by Hyper-Encoder in Probability Model to obtain side information z = HE(y). z is quantized to obtain z′ = Quantized(z). The quantized z′ is arithmetically encoded to obtain z bit ; z bit After arithmetic decoding, we get z′. Through Hyper-Decoder, Context Model and decoupling in Probability Model, we get mean u and standard deviation σ, (u,σ)=Concat(HD(z′)+CM()). We quantize y to get y′=Quantized(y). Through u and σ, we perform Gaussian modeling P(y′) on y′, and then we can perform arithmetic coding on P(y′) to get y. bit ;

[0011] When decoding, z bitAfter arithmetic decoding, we get z′, and through Hyper-Decoder, Context Model and decoupling in Probability Model, we get the mean u and standard deviation σ; bit Arithmetic decoding obtains P(y′); y′ is restored from P(y′) and u and σ, and then the Generator reconstructs the image x′=G(y′) after upsampling;

[0012] S3, using the image compression neural network constructed in step S2 and the obtained network training weight parameters to construct an image compression system based on Swin-Transformer and autoregression, and using the compression system to compress the image and store it. When the image needs to be used, the compressed file of the required image is decompressed using the image compression system to obtain the required picture.

[0013] The present invention also proposes an image compression system based on Swin-Transformer and autoregression, which includes the following modules:

[0014] An image compression coding module configured to extract features from the image;

[0015] An image compression hyper-prior module, configured to perform Gaussian modeling of features in the image;

[0016] The image compression generation module is configured to reconstruct an image.

[0017] The beneficial effects of the invention are as follows:

[0018] 1. The present invention proposes an image compression method based on Swin-Transformer and autoregression. Since convolutional neural networks are not conducive to the extraction of local features, an encoder based on Swin-Transformer is used to extract features, and then an autoregressive method is used to perform Gaussian modeling on the features. Finally, the image is reconstructed by a generator based on Swin-Transformer;

[0019] 2. When modeling the feature graph, the present invention adopts the Gaussian modeling method, and uses the autoregression method and decoupling method to obtain the mean and variance required for Gaussian modeling;

[0020] 3. When reconstructing an image, the present invention uses a method combining convolution and Swin-Transformer;

[0021] 4. The present invention proposes and designs an image compression system, which can use the Swin-Transformer and autoregressive method proposed in the present invention to perform image compression. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The overall structure diagram of the model of the image compression method used in the embodiment of the present invention;

[0023] Figure 2 The Swin-Transformer structure diagram of the image compression method used in the embodiment of the present invention;

[0024] Figure 3 A schematic diagram of the image compression system flow in an embodiment of the present invention;

[0025] Figure 4 A diagram showing the composition of an image compression system according to an embodiment of the present invention;

[0026] Figure 5 It is a comparison diagram of the images reconstructed by the method of the present invention and those not reconstructed by the method of the present invention, where original is the original image, sub-image a is the image reconstructed by the method one using the recurrent neural network image compression method, sub-image b is the generative adversarial network image compression method used in the method two, and sub-image ours is the image reconstructed by the method of the present invention;

[0027] Figure 6 This is a comparison chart of the image quality reconstructed using this method, method 1, and method 2. Sub-image a is the comparison result of PSNR under different BPPs, and sub-image b is the comparison result of MS-SSIM under different BPPs. The higher the two indicators, the better. DETAILED DESCRIPTION

[0028] In order to have a clearer understanding of the technical features, purposes and beneficial effects of the present invention, an embodiment of the present invention is further described in conjunction with the accompanying drawings. The embodiment is only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Some non-essential improvements and adjustments made by technicians in this field based on the content of the present invention also fall within the scope of protection of the present invention.

[0029] A Swin-Transformer-based and adaptive image compression method, characterized in that it comprises the following steps:

[0030] S1. Collect a variety of different pictures in the natural environment, and randomly divide the collected pictures into two parts in a ratio of 9:1. The first part constitutes the training set, and the second part constitutes the test set. And preprocess the training set;

[0031] S2, construct an image compression neural network; input the training set images obtained in S1 into the constructed image compression neural network for training; the loss function uses mean square error, perceptual loss and rate-distortion, and sets hyperparameters for constructing the image compression network. By setting the optimization strategy, the network is trained iteratively to continuously reduce the loss function until the set number of iterations is completed and the training weight parameters are saved;

[0032] S3, using the image compression neural network constructed in step S2 and the obtained network training weight parameters to construct an image compression system based on Swin-Transformer and autoregression, and using the compression system to compress the image and store it. When the image needs to be used, the image compression system is used to decompress the required image compression file to obtain the required picture.

[0033] As a specific implementation of the present invention, the steps specifically include the following steps:

[0034] S21, encoder built using Swin-Transformer and generator for reconstructed images;

[0035] S22, using autoregression to assist in Gaussian modeling of the features extracted by the encoder;

[0036] S23, u and σ used in Gaussian modeling are obtained using a decoupling approach;

[0037] S24, Generator uses a combination of convolutional neural network and Swin-Transformer;

[0038] As a specific implementation of the present invention, the image compression neural network constructed in step S3 is specifically:

[0039] The network structure is generally composed of three parts, namely Encoder, Generator and ProbabilityModel. Encoder sets 8 Swin-Transformer blocks and 1 arithmetic encoder in sequence, and downsamples every two Swin-Transformer blocks; Generator is composed of arithmetic decoder, 9 layers of residual network stack and 8 Swin-Transformer block stack. The residual network does not upsample the features, but 8 Swin-Transformers complete the upsampling task, and upsamples every two Swin-Transformer blocks; ProbabilityModel is mainly composed of two parts. The first part is composed of arithmetic encoder and Hyper-Encoder, where Hyper-Encoder is composed of 3 convolutions with kernel_size=3 and stride=2. The second part is composed of an arithmetic decoder, an autoregressive network Context Model and a Hyper-Decoder. The Hyper-Decoder is composed of three transposed convolutions with kernel_size=3 and stride=2 and a decoupling head. The autoregressive network Context Model is composed of a 5*5 mask conv convolution.

[0040] During encoding, Encoder, Generator, and Probability Model are referred to as E, G, and P respectively. Image x is downsampled by Encoder to obtain feature y = E(x), feature y is extracted by Hyper-Encoder in Probability Model to obtain side information z = HE(y), z is quantized to obtain z′ = Quantized(z), and quantized z′ is arithmetic encoded to obtain z bit ; z bit After arithmetic decoding, we get z′. Through Hyper-Decoder, Context Model and decoupling in Probability Model, we get mean u and standard deviation σ, (u, σ) = Concat(HD(zσ) + CM()); y is quantized to get y′ = Quantized(y). Through u and σ, y′ is Gaussian modeled P(y′), and then P(y′) can be arithmetically encoded to get y. bit ;

[0041] When decoding, z bitAfter arithmetic decoding, we get z′, and through Hyper-Decoder, Context Model and decoupling in Probability Model, we get the mean u and standard deviation σ; bit Arithmetic decoding obtains P(y); y′ is restored from P(y′) and u and σ, and then the Generator reconstructs the image x′=G(y′) after upsampling.

[0042] Simulation experiment

[0043] Depend on Figure 5 It can be seen that among the images reconstructed using this method, the ours sub-image performs better than the visual effects of sub-image a of method one and sub-image b of method two, which indicates that the design model in the present invention has a certain enhancement effect and is helpful for reconstructing images.

[0044] The image quality of the method of the present invention and the comparative image compression method is as follows Figure 6 As shown, method one uses a reconstructed image quality index obtained by a recurrent neural network, and method two uses a reconstructed image quality index obtained by a generative adversarial network.

[0045] from Figure 6 It can be seen that: using the Swin-Transformer and autoregressive network method of the present invention, the PSNR and MS-SSIM values ​​are improved to a certain extent, and the reconstructed image effect is better than method 1 and method 2. The above simulation experiment results show that the method of the present invention improves the quality of the reconstructed image while maintaining the same compression ratio.

[0046] The above is a description of the method of the present invention, and those skilled in the art can implement the method of the present invention based on the description of the content. Based on the above content of the present invention, other embodiments obtained by those skilled in the art without creative work should all fall within the scope of protection of the present invention.

Claims

1. An image compression method based on Swin-Transformer and autoregression, Its characteristics include the following steps: S1. Collect a variety of different pictures in the natural environment, randomly divide the collected pictures into two parts in a ratio of 9:1, the first part constitutes the training set, and the second part constitutes the test set, and preprocess the training set; S2, construct an image compression neural network; randomly horizontally flip, randomly crop and normalize the image obtained in S1, and then input it into the constructed image compression neural network for training; the loss function uses mean square error, perceptual loss and rate distortion loss function, sets hyperparameters to balance loss weights, sets optimization strategies for constructing image compression networks, and continuously reduces the loss function by iteratively training the network until the set number of iterations is completed and the training weight parameters are saved; S3, using the image compression neural network constructed in step S2 and the obtained network training weight parameters to construct an image compression system based on Swin-Transformer and autoregression, and using the compression system to compress the image and store it; when the image needs to be used, use the image compression system to decompress the compressed file of the required image to obtain the required picture; The image compression neural network constructed in step S2 is specifically as follows: the network structure is generally composed of three parts, namely, Encoder, Generator and Probability Model; Encoder sequentially sets 8 Swin-Transformer blocks and 1 arithmetic encoder, wherein every two Swin-Transformer blocks are downsampled once; Generator is composed of an arithmetic decoder, a 9-layer residual network stack and 8 Swin-Transformer stacks, the residual network does not perform an upsampling operation on the features, but 8 Swin-Transformers complete the upsampling task, and every two Swin-Transformer blocks are upsampled once; Probability Model is mainly composed of two parts, the first part is composed of an arithmetic encoder and a Hyper-Encoder, wherein the Hyper-Encoder is composed of 3 convolutions with kernel_size=3 and stride=2; and the second part is composed of an arithmetic decoder, an autoregressive network Context The model is composed of a Hyper-Decoder, where the Hyper-Decoder is composed of 3 transposed convolutions with kernel_size=3 and stride=2 and decoupling; the autoregressive network Context Model is composed of a 5*5 mask conv convolution, which only appears at the decoding end; During encoding, Encoder, Generator, and Probability Model are referred to as E, G, and P respectively. Image x is downsampled by Encoder to obtain feature y = E(x). Feature y is extracted by Hyper-Encoder in Probability Model to obtain edge information z = HE(y). z is quantized to obtain z′Quantized(z). The quantized z′ is arithmetically encoded to obtain z bit ; z bit After arithmetic decoding, we get z′. Through Hyper-Decoder, Context Model and decoupling in Probability Model, we get mean u and standard deviation σ, (u, σ) = Concat(HD(z′) + CM()). We quantize y to get y′ = Quantized(y). Through u and σ, we perform Gaussian modeling P(y′) on y′, and then we can perform arithmetic coding on P(y′) to get y. bit ; When decoding, z bit After arithmetic decoding, we get z′, and through Hyper-Decoder, Context Model and decoupling in Probability Model, we get the mean u and standard deviation σ; bit Arithmetic decoding obtains P(y′); y′ is restored from P(y′) and u and σ, and then the Generator reconstructs the image x′=G(y′) after upsampling.

2. The image compression method based on Swin-Transformer and autoregression according to claim 1, characterized in that: The steps include: S21, encoder built using Swin-Transformer and generator for reconstructed images; S22, using autoregression to assist in Gaussian modeling of features extracted by the encoder; S23, u and σ used in Gaussian modeling are obtained using a decoupling approach; S24, Generator uses a combination of convolutional neural network and Swin-Transformer.

3. A system using the Swin-Transformer and autoregressive image compression method according to any one of claims 1 to 2, characterized in that: Includes the following modules: An image compression coding module configured to extract features from the image; An image compression hyper-prior module, configured to perform Gaussian modeling of features in the image; The image compression generation module is configured to reconstruct an image.

Images