Image compression method based on dictionary learning and deep learning

Abstract

The invention provides an end-to-end image compression method based on dictionary learning and deep learning. According to the method, firstly, a learnable network parameter is used as a dictionary, and the dictionary is used for capturing common textures and patterns in a natural image in the gradient descent training process of a neural network; then, in the autoregression coding and decoding process, the dictionary containing the complete texture information is queried by using the features containing the partial texture information through a cross attention method, so that the dictionary features of the complete texture information are extracted from the decoded partial information, and the features are used for probability distribution modeling. The dictionary can summarize useful information in a training data set and provide priori knowledge. During autoregression coding and decoding, probability distribution modeling can be carried out on current coding and decoding information more accurately by utilizing the prior information, so that shorter coding length is realized. The method not only improves the efficiency of image compression, but also enhances the quality of the reconstructed image.

Description

Technical Field

[0001] The present invention belongs to the field of image compression, and in particular relates to an image compression method based on dictionary learning and deep learning. Background Art

[0002] Image compression is an important and long-standing research direction in the field of image signal processing. The widespread demand for high-resolution images in modern society has brought severe challenges to the storage and transmission of high-resolution images. Therefore, a powerful image compression technology is urgently needed to be developed. This image compression technology not only needs to be able to compress the image more tightly to meet the storage and transmission requirements of high-resolution images, but also needs to ensure the fidelity of the reconstructed image to provide users with high-quality high-resolution images.

[0003] Recently, deep learning-based image compression algorithms (LIC) have shown excellent performance, even surpassing the current state-of-the-art image and video coding standard VVC. This is mainly due to its powerful nonlinear transformation capabilities and end-to-end joint optimization methods. Deep learning-based image compression algorithms mainly consist of two key components: a nonlinear encoder-decoder and an entropy model. The encoder and decoder are responsible for converting images and latent representations to each other; while the entropy model is used to estimate the probability distribution of the quantized latent representation. This probability distribution will be further used for entropy coding, encoding the latent representation into a bitstream, or decoding the bitstream back to the latent representation.

[0004] In order to utilize the nonlinear mapping capability to transform the input image into a more compact potential representation, some scholars have studied the design of powerful codecs with nonlinear capabilities. Ballé et al. used convolutional neural networks (CNNs) to locally model the input image. Cheng et al. applied attention modules to enable the model to handle challenging content. Xie et al. used reversible neural networks to alleviate the problem of information loss. Zou et al. and Zhu et al. first used the advantage of transformers that can effectively model long-distance dependencies and used transformers to form their codecs. In order to have both local modeling and non-local modeling capabilities, the present invention uses CNNs and transformers to form our codecs.

[0005] Other scholars studied a more basic module, namely the entropy model, to model the probability distribution of the latent representation. Ballé et al. first proposed the use of a fully decomposed density entropy model to model the probability distribution, and realized the end-to-end joint optimization of the entropy model. In order to further improve the modeling ability of the entropy model, Ballé et al. introduced a super-prior network structure, which captured the spatial dependencies of the latent representation by using side information and modeled the latent representation as a single Gaussian distribution. Cheng et al. further used a more expressive discrete mixed Gaussian distribution to model the latent representation. Kim et al. used the attention mechanism of the transformer to establish the global dependencies between the latent representations. Minnen et al. introduced an autoregressive encoding and decoding method in the spatial pixel dimension, which used the decoded latent representation to assist the probability distribution modeling of the latent representation currently being encoded and decoded. In order to further improve the speed of autoregressive encoding and decoding, Minnen et al. used an autoregressive context model based on the channel dimension to achieve parallel encoding and decoding. He et al. used a checkerboard-shaped spatial context model to achieve two-step decoding.

[0006] Although many advances have been made in improving the modeling capabilities of entropy models, these works mainly focus on how to use the internal dependencies of potential representations to better model the probability distribution of potential representations. For the super-prior network, it uses a small convolutional neural network on the potential representation to extract side information, thereby further capturing the internal spatial dependencies of the potential representation. For the autoregressive decoding method, the decoded potential representation is used to assist in decoding the potential representation currently being decoded, which improves the modeling capabilities of the entropy model by utilizing the internal dependencies of the potential representation. However, relying solely on the internal dependencies between potential representations limits the accurate estimation of probability distribution. In the image compression task, introducing more additional prior information and external dependencies will help achieve higher compression rates and more fidelity reconstructed image quality. The present invention utilizes the ability of dictionary learning to preserve typical textures and patterns, and introduces additional prior information by establishing external dependencies between the potential representation and the dictionary, thereby providing more accurate probability distribution estimates during the compression process, significantly improving the compression rate of the image, and maintaining higher quality when reconstructing the image. Summary of the invention

[0007] Since the entropy model in the existing image compression algorithm mainly focuses on how to use the internal dependency between potential representations to model probability distribution, but lacks the exploration of the external dependency between the potential representation and the natural image, this limits the image compression algorithm to obtain a lower image compression rate and a more fidelity reconstructed image quality. In order to solve the problem that the existing image compression algorithm has limited probability distribution modeling capabilities due to the inability to effectively establish external dependencies, the present invention takes advantage of the fact that dictionary learning can introduce additional information, by using a dictionary to learn the textures and patterns that often appear in natural images, and then when the entropy model is used for probability distribution modeling, the context containing partial texture information is used to query the dictionary containing complete texture information, and an external dependency relationship between the potential representation and the dictionary is established, so that the trained image compression network can obtain a higher image compression rate and a more fidelity reconstructed image quality.

[0008] The technical solution adopted by the present invention to solve the above technical problems is: an image compression method based on dictionary learning and deep learning, comprising the following steps:

[0009] 1) Dictionary construction. In the entropy model of the image compression network, we use a learnable network parameter as a dictionary, which aims to capture common textures and patterns in the training dataset. By introducing this dictionary, we establish an external dependency between the latent representation and the dictionary, bringing in additional prior information. The dictionary is optimized by using a rate-distortion loss function and a gradient descent algorithm.

[0010] 2) Acquisition of dictionary information. When performing autoregressive encoding and decoding, a cross-attention method is used to query the complete texture information in the dictionary. Specifically, context information and side information that carry partial texture information are used as query information. By using the query information to query the dictionary that carries complete texture information, an external dependency relationship between the latent representation and the dictionary is established. The dictionary information is used to more accurately estimate the probability distribution of the latent representation, thereby training an image compression network with better performance.

[0011] The specific steps are as follows:

[0012] Step S1: Obtain training data and preprocess;

[0013] Step S2: construct an end-to-end deep image compression network; the image compression network includes an encoder, a decoder, a dictionary-based entropy model, a hyper-prior module, and a quantizer; the image to be compressed is used as an input to obtain a potential representation to be compressed through the encoder; the potential representation is input into the hyper-prior module and the quantizer; the hyper-prior module generates hyper-prior features to help the dictionary-based entropy model to estimate the probability distribution; the quantizer discretizes the potential representation into a quantized potential representation; the quantized potential representation is encoded into a bit stream or decoded from the bit stream to obtain a decoded representation through the probability distribution output by the dictionary-based entropy model; the decoded representation is dequantized using a quantization error r to obtain a dequantized potential representation; and then the decoder is further used to restore the reconstructed image;

[0014] Step S3: Perform supervised training on the image compression network to obtain a trained image compression network for subsequent image compression.

[0015] The super-prior module extracts a side information z from the potential representation y, which is further quantized and then mapped to the super-prior feature by the super-prior decoder

[0016] Furthermore, the dictionary-based entropy model includes a dictionary-based cross attention module DCA and a probability distribution estimation network f E and the quantization error prediction network f LRP ; Specifically, the super prior feature and the decoded latent representation slice As the input of the dictionary-based cross-attention module to generate a query for the dictionary, and then obtain the features carrying the complete texture after querying the dictionary

[0017]

[0018] Then, and Input into the probability distribution estimation network f E Get the probability distribution parameter Φ i :

[0019]

[0020] Probability distribution parameter Φ i Used for the slice being encoded and decoded Encode it into a bit stream or decode it back from a bit stream.

[0021] Furthermore, the potential representation y is quantized to obtain the quantized potential representation Then it is divided into N slices along the channel dimension. These N slices Perform entropy coding in order;

[0022] slice and Sent together to the quantization error prediction network f LRP Get the slice The predicted slice quantization error r i :

[0023]

[0024] Dequantized slices Feed into decoder g s , and get the reconstructed image

[0025] Furthermore, the workflow of the dictionary-based cross-attention module is as follows:

[0026] Super Prior Features and the decoded and dequantized representation Concatenate along the channel dimension to obtain query features Further pass through a linear layer to generate query tokens Where H×W represents the total number of input pixels, C qk represents the number of channels for each query token; a network-learned parameter is used as a dictionary δ, which is used as the value token Where N is the number of dictionary items, C d The number of channels representing the value token; a linear layer is used to generate the key token The formula is as follows:

[0027]

[0028] K=δW K ,

[0029] V = δ;

[0030] Where W i Q and W K The distribution represents the query features A linear change operation of and dictionary δ;

[0031] Then use the query token Q i And key token K get the similarity matrix between query features and dictionary

[0032]

[0033] Among them, δ iRepresents learning parameters, SoftMax is used to normalize the similarity;

[0034] Using the similarity matrix A i The sum token V further obtains the dictionary feature containing complete texture information

[0035]

[0036] The dictionary features After being enhanced by the feedforward neural network, it will then be sent to the probability distribution estimation network f E .

[0037] Furthermore, during the training process, the rate distortion loss function is used as the loss function of the deep image compression network:

[0038]

[0039] in and Respectively represent the quantized potential representation and quantified side information The bitstream size, Represents the input image x and the reconstructed image ; λ is a hyperparameter.

[0040] Furthermore, the encoder g a Three downsampling modules, three transformer modules and one downsampling convolution layer; the image to be compressed x passes through the downsampling module, transformer module, downsampling module, transformer module, downsampling module, transformer module, downsampling convolution in sequence to obtain the potential representation y to be compressed;

[0041] Decoder g s Includes three upsampling modules, three transformer modules and one upsampling convolutional layer; the potential representation after dequantization The reconstructed image is obtained by sequentially passing through upsampling convolution, transformer module, upsampling module, transformer module, upsampling module, transformer module and upsampling module.

[0042] The downsampling module is composed of a downsampling convolution and three residual blocks connected in series from front to back; the transformer module is composed of several sliding window transformer layers; and the upsampling module is composed of an upsampling convolution layer and three residual blocks connected in series from front to back.

[0043] The beneficial effect of the present invention is that it can realize a deep learning image compression method that can achieve higher image compression effect and more fidelity reconstructed image quality by means of a dictionary that stores textures and patterns that often appear in natural images. First, under the continuous optimization of the rate-distortion loss function and the gradient descent method, learnable network parameters are used as a dictionary to capture the textures and patterns that often appear in the training data set, thereby introducing additional prior information and external dependencies. Secondly, in order to obtain texture information stored in the dictionary related to the potential representation. The present invention uses a cross-attention method to query the dictionary using contextual information and side information containing partial texture information, and uses the obtained dictionary information to more accurately evaluate the probability distribution of the potential representation to obtain a higher image compression rate and a more fidelity reconstructed image quality. The present invention constructs a new deep image compression algorithm based on dictionary learning, which has the advantage of high performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Flowchart of the end-to-end deep image compression method.

[0045] Figure 2 Schematic diagram of the dictionary-based entropy model.

[0046] Figure 3 Schematic diagram of the dictionary-based cross-attention module. DETAILED DESCRIPTION

[0047] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of an image compression method based on dictionary learning and deep learning of the present invention in conjunction with the accompanying drawings.

[0048] This embodiment provides a flowchart of an end-to-end image compression method based on dictionary learning and deep learning. The specific steps of this method are as follows:

[0049] 1) Composition of the training dataset for deep image compression. In order to train the image compression network, the OpenImages v6 open source dataset is used for supervised training. The dataset contains 300,000 images, which are randomly cropped to a resolution of 256×256 during training.

[0050] 2) Design of end-to-end deep image compression network. Figure 1 As shown, the image compression network takes the image to be compressed x as input and passes it through the encoder g a Get the potential representation y to be compressed, and then input the potential representation y into the super prior module to generate super prior features To help the dictionary entropy model estimate the probability distribution. In order to perform entropy coding, the potential representation y will be discretized by the quantizer Q into a quantized potential representation In order to quantize the potential Encode to or decode from a bitstream The dictionary-based entropy model will be used to estimate its probability distribution Φ. after, The quantization error r will be used to dequantize to the dequantized potential representation Further decoded by g s Restore the reconstructed image

[0051] For the encoder g a , which consists of three downsampling modules, three transformer modules and a downsampling convolution layer. Whenever the input is sent to a downsampling module or a downsampling convolution, the height H and width W of the input will become half of the input, and the number of channels C will also change, such as Figure 1 As shown in Figure 1. Each downsampling module consists of a downsampling convolution and three residual blocks connected in series from front to back. Each transformer module consists of several sliding window transformer layers; the image to be compressed x passes through the downsampling module, transformer module, downsampling module, transformer module, downsampling module, transformer module, downsampling convolution in sequence to obtain the potential representation y. For the decoder g s , which consists of three upsampling modules, three transformer modules and an upsampling convolution layer. Whenever the input is sent to an upsampling module or upsampling convolution, the height and width of the input will become twice the input, and the number of channels will also change. The upsampling module consists of an upsampling convolution layer and three residual blocks connected in series from front to back; the potential representation after dequantization After upsampling convolution, transformer module, upsampling module, transformer module, upsampling module, transformer module, upsampling module, the reconstructed image is obtained.

[0052] 3) Design of dictionary-based entropy model. The dictionary-based entropy model is built on the framework of hyper-prior structure and autoregressive structure, and uses dictionary δ to capture the common textures and patterns in natural images. Figure 2 As shown in Figure 2, it is built on the super prior structure and the autoregressive structure. The super prior structure extracts a side information z from the potential representation y, and the side information z is further quantized as and then will be decoded by the super priori decoder h s Mapping to hyper-prior features For the autoregressive structure, we use an autoregressive context model based on the channel dimension, where the latent representation y is first quantized as After that, it is then divided into five slices along the channel dimension These five slices are entropy encoded in order. For example, the decoded dequantized slice Will help Perform probability distribution modeling.

[0053] In getting the super prior features And the decoded dequantized slice Later, since these two features only establish the internal dependency between the latent representations and only contain part of the information of the latent representations, in order to further capture the external dependency between the latent representations and the training dataset, we will and As the input of the dictionary-based cross-attention module (DCA), it generates a query for the dictionary, and then obtains the features carrying the complete texture after querying the dictionary.

[0054]

[0055] Then, and Will be sent to the probability distribution estimation network f E To estimate the slice currently being encoded or decoded The probability distribution parameter Φ i :

[0056]

[0057] The obtained probability distribution parameter Φ i Will be used to Encode it into a bit stream or decode it back from a bit stream.

[0058] In order to reduce the information loss caused by quantization, the decoded slices that have not been dequantized are Will Sent together to the quantization error prediction network f LRP get The predicted slice quantization error r i :

[0059]

[0060] After getting the slice quantization error r i , dequantized slice Will be further fed into the decoder g s To get the reconstructed image

[0061] 4) Design of dictionary-based cross-attention module. Figure 3 As shown in the figure, the cross attention method is used to extract the texture information stored in the dictionary. and decoded and dequantized slices Later, these two features are concatenated along the channel dimension to obtain the query feature The feature is further passed through a linear layer to generate query tokens. (H×W represents the total number of input pixels, C qk represents the number of channels per query token). In addition, a learnable network parameter is used as a dictionary δ, which is used as the value token (N represents the number of dictionary items, C d represents the number of channels of the value token), and a linear layer is used to generate the key token (N represents the number of dictionary items, C qk Indicates the number of key token channels):

[0062]

[0063] K=δW K ,

[0064] V = δ;

[0065] Where W i Q and W K The distribution represents the query features and a linear change operation of the dictionary δ.

[0066] Then use the query token Q i And key token K get the similarity matrix between query features and dictionary

[0067]

[0068] Here τ i is a learnable scaling parameter used to adjust the query token Q i The scale range of the dot product with the key token K, SoftMax is used to normalize the similarity size.

[0069] Using the similarity matrix A i The value token V is further used to obtain the dictionary feature containing complete texture information

[0070]

[0071] The dictionary features After being enhanced by the feedforward neural network, it will then be sent to the probability distribution estimation network f E To estimate the slice currently being encoded or decoded The probability distribution parameter Φ i and is fed into the quantization error prediction network f LRP get The predicted quantization error r i .

[0072] 5) Network training strategy. Since the image compression method requires considering both the image compression rate and the image reconstruction quality, in order to achieve multi-objective optimization, the rate-distortion loss function is used as the loss function of the deep image compression network:

[0073]

[0074] in and Respectively represent the encoded potential representation and quantified side information is the size of the bit stream, Represents the input image x and the reconstructed image λ is a hyperparameter used to control the trade-off between the bitstream size of the encoded image and the distortion of the reconstructed image.

[0075] During the training process of the model, the Adam optimizer was used for optimization, and a total of 60 rounds of training were performed. The learning rate was 1e-4 for the first 50 rounds, and the learning rate was 1e-5 for the next 10 rounds. The batch size was set to 8.

[0076] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. An image compression method based on dictionary learning and deep learning, characterized in that: The following steps are involved: Step S1: Obtain training data and preprocess; Step S2: construct an end-to-end deep image compression network; The image compression network includes an encoder, a decoder, a dictionary-based entropy model, a super-prior module, and a quantizer; the image to be compressed is used as an input to obtain a potential representation to be compressed through the encoder; the potential representation is input into the super-prior module and the quantizer; the super-prior module generates super-prior features to help the dictionary-based entropy model to estimate the probability distribution; the quantizer discretizes the potential representation into a quantized potential representation; the quantized potential representation is encoded into a bit stream or decoded from the bit stream to obtain a decoded representation through the probability distribution output by the dictionary-based entropy model; The decoded representation is dequantized using the quantization error r to obtain a dequantized potential representation; then the decoder is further used to recover the reconstructed image; Step S3: Perform supervised training on the image compression network to obtain a trained image compression network for subsequent image compression.

2. The image compression method based on dictionary learning and deep learning according to claim 1, characterized in that: The super-prior module extracts a side information z from the potential representation y, which is further quantized and then mapped to the super-prior feature by the super-prior decoder 3. The image compression method based on dictionary learning and deep learning according to claim 1, characterized in that: The dictionary-based entropy model includes a dictionary-based cross attention module DCA and a probability distribution estimation network f E and the quantization error prediction network f LRP ; Specifically, the super prior feature and the decoded latent representation slice As the input of the dictionary-based cross-attention module to generate a query for the dictionary, and then obtain the features carrying the complete texture after querying the dictionary Then, and Input into the probability distribution estimation network f E Get the probability distribution parameter Φ i : Probability distribution parameter Φ i Used for the slice being encoded and decoded Encode it into a bit stream or decode it back from a bit stream.

4. The image compression method based on dictionary learning and deep learning according to claim 3, characterized in that: The potential representation y is quantized to obtain the quantized potential representation Then it is divided into N slices along the channel dimension. These N slices Perform entropy coding in order; slice and Sent together to the quantization error prediction network f LRP Get the slice The predicted slice quantization error r i : Dequantized slices Feed into decoder g s , and get the reconstructed image 5. The image compression method based on dictionary learning and deep learning according to claim 4, characterized in that: The workflow of the dictionary-based cross-attention module is as follows: Super Prior Features and the decoded and dequantized representation Concatenate along the channel dimension to obtain query features Further pass through a linear layer to generate query tokens Where H×W represents the total number of input pixels, C qk Indicates the number of channels per query token; A network learned parameter is used as a dictionary δ, which is used as the value token Where N is the number of dictionary items, C d The number of channels representing the value token; Use a linear layer to generate key tokens The formula is as follows: K=δW K , V = δ; Where W i Q and W K The distribution represents the query features A linear change operation of and dictionary δ; Then use the query token Q i And key token K get the similarity matrix between query features and dictionary Among them, τ i Represents learning parameters, SoftMax is used to normalize the similarity; Using the similarity matrix A i The sum token V further obtains the dictionary feature containing complete texture information The dictionary features After being enhanced by the feedforward neural network, it will then be sent to the probability distribution estimation network f E .

6. The image compression method based on dictionary learning and deep learning according to claim 5, characterized in that: During the training process, the rate distortion loss function is used as the loss function of the deep image compression network: in and Respectively represent the quantized potential representation and quantified side information The bitstream size, Represents the input image x and the reconstructed image ; λ is a hyperparameter.

7. The image compression method based on dictionary learning and deep learning according to claim 6, characterized in that: The encoder g a Three downsampling modules, three transformer modules and one downsampling convolution layer; the image to be compressed x passes through the downsampling module, transformer module, downsampling module, transformer module, downsampling module, transformer module, downsampling convolution in sequence to obtain the potential representation y to be compressed; Decoderg s It includes three upsampling modules, three transformer modules and one upsampling convolution layer; the dequantized potential representation y passes through the upsampling convolution, transformer module, upsampling module, transformer module, upsampling module, transformer module, upsampling module in sequence to obtain a reconstructed image.

8. The image compression method based on dictionary learning and deep learning according to claim 7, characterized in that: The downsampling module is composed of a downsampling convolution and three residual blocks connected in series from front to back; the transformer module is composed of several sliding window transformer layers; and the upsampling module is composed of an upsampling convolution layer and three residual blocks connected in series from front to back.

Images