Deep learning image compression method based on importance scale and Gaussian mixture model

Abstract

The invention relates to a deep learning image compression method based on an importance scale and a Gaussian mixture model, and belongs to the technical field of image processing. According to the method, an importance scale network is constructed, the potential representation obtained by a main encoder is processed through the importance scale network, bit distribution of adaptive content is realized, effective extraction of key feature information is ensured, and thus the potential representation with enhanced importance is obtained. A Gaussian mixture model is introduced into both a content stream and a model stream, and the Gaussian mixture model comprises a plurality of Gaussian distributions, so that more accurate probability distributions can be obtained for guiding entropy coding and decoding to obtain more accurate reconstruction potential representation or neurogrammar information. According to the method, content self-adaptive optimization is carried out on the potential representation, a more reasonable and flexible probability distribution model is obtained by using Gaussian mixture distribution prediction, redundant information in the potential representation or neurogrammar is reduced, and the rate distortion performance of a deep learning image compression model is improved while a reconstructed image with a better visual effect is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and relates to a deep learning image compression method based on importance scale and Gaussian mixture model. Background Art

[0002] Image compression refers to the use of a series of coding methods to compress and encode the original image data to be processed, while ensuring the visual effect of the image, to obtain the smallest possible compressed code stream for subsequent storage and transmission. Traditional image compression methods include: Joint Photographic Experts Group (JPEG), High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), etc. These methods all reduce spatial redundancy and improve coding efficiency through intra / inter-frame prediction, transformation, quantization and entropy coding. Each module is designed manually, but the current traditional image compression method has reached a bottleneck in improving compression performance. In recent years, with the continuous development of deep learning, deep neural networks have been widely used in the field of image processing, and various deep learning image compression methods have been continuously proposed. Image compression based on deep learning can process a large amount of image data through training and learning, extract feature information of the image, and has a strong ability to process high-dimensional data. In addition, compared with traditional image compression methods, deep learning image compression methods can jointly optimize various modules such as transformation, quantization, and entropy coding in the compression process, consider the problem of block segmentation as a whole, and obtain better compression performance. However, the current image compression methods based on deep learning rely too much on the data generalization ability of the model itself, and lack consideration of the adaptability of image content. In addition, existing methods usually use a single Gaussian model to predict the probability distribution. Due to the limitation of its fixed shape, the description of the probability distribution model is not accurate enough, and the latent information or neural grammar information after the image transformation is still redundant, which affects the data compression effect. In the existing deep learning image compression model, the image content adaptability is improved by using online update, but this method may increase the computational overhead, and this type of method ignores the refinement of the latent representation within the entire compression model. The latent representation refers to the deeper feature information of the image. By optimizing the existing model, different bit rates can be allocated to the content of different regions of the latent representation, so that the deep learning image compression model can achieve content adaptation at a finer granularity. At the same time, the single Gaussian model is expanded to a Gaussian mixture model to predict a more accurate and flexible probability distribution model to guide entropy coding and decoding, reduce redundant information, and obtain more accurate reconstructed potential representations or neural grammar information. Summary of the invention

[0003] In view of this, an object of the present invention is to provide a deep learning image compression method based on importance scale and Gaussian mixture model.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] A deep learning image compression method based on importance scale and Gaussian mixture model comprises the following steps:

[0006] S1, the image to be processed is analyzed and transformed through the main encoder to obtain a potential representation;

[0007] S2, passing the latent representation through a preset importance scale network, and performing adaptive content bit allocation for different region contents to obtain an importance-enhanced latent representation;

[0008] S3, dividing the importance-enhanced latent representation into a latent representation of a content stream and a latent representation of a model stream in a channel dimension;

[0009] S4, obtaining a probability distribution of a potential representation of the content stream by processing the hyper-prior model, the context model and the preset Gaussian mixture model to obtain a reconstructed potential representation;

[0010] S5. Obtain the neural grammar of the potential representation of the model stream using a neural grammar generator; obtain the probability distribution of the neural grammar through the preset Gaussian mixture model to guide entropy coding and decoding to obtain neural grammar information; feed the neural grammar information to a weight generator to obtain the kernel parameters of the last transposed convolutional layer of the main decoder;

[0011] S6. Performing a synthetic transformation on the reconstructed latent representation and the kernel parameters of the last transposed convolutional layer of the decoder through the main decoder to obtain a reconstructed image.

[0012] Furthermore, in S1: the main encoder includes four two-dimensional zero padding (Zero Padding 2D Layer, ZeroPad2d) layers, four convolutional layers and three generalized division normalization (Generalized Divisive Normalization, GDN) layers.

[0013] Furthermore, in S2: the importance scale network includes a convolutional layer and three residual blocks; an adaptive bit allocation strategy is adopted to allocate a higher bit rate to the area with a larger potential value in the area to extract more feature information, thereby obtaining a potential representation with enhanced importance.

[0014] Further, in S4: the super-a priori model includes a super-a priori encoder and a super-a priori decoder; the super-a priori encoder includes three convolution layer groups; the super-a priori decoder includes three transposed convolution layers; and there is a rectified linear unit (ReLU) function between the convolution layers and between the transposed convolution layers;

[0015] Furthermore, in S4: the context model includes three convolutional layers, and leaky rectified linear unit (LeakyReLU) functions are provided between the convolutional layers; and the context information of the potential representation is obtained through the context model.

[0016] Further, in S4: the Gaussian mixture model is composed of multiple Gaussian distributions with different means, variances and weights, and the Gaussian mixture model is expressed as:

[0017]

[0018] Among them, w represents the weight value of each Gaussian distribution, μ and σ represent the mean and variance of the Gaussian distribution, and K represents the total number of Gaussian models. represents the reconstruction potential representation, Represents the quantitative information of auxiliary information.

[0019] Further, in S5: the neural grammar generator includes a multi-scale network, a global average pooling layer and a fully connected layer; the latent representation of the model stream is extracted features by the multi-scale network, and is connected into a neural grammar feature vector after global average pooling for use by a subsequent weight generator.

[0020] Furthermore, in S5: the weight generator includes three fully connected layers, and there is a leaky rectified linear unit (LeakyReLU) function between the fully connected layers.

[0021] Furthermore, in S6: the main decoder includes five transposed convolutional layers, and there is an inverse GDN layer (Inverse Generalized Divisive Normalization Layer) between the convolutional layers; after the training of the entire deep learning image compression model is completed, the parameters of all transposed convolutional layers except the last transposed convolutional layer are fixed values; the kernel parameters of the last transposed convolutional layer are generated in real time by the weight generator during decoding.

[0022] Furthermore, the total number of Gaussian models K=4.

[0023] The beneficial effects of the present invention are: realizing adaptive content bit allocation, allocating higher bit rates to information-intensive areas, ensuring effective extraction of key feature information, and thereby obtaining a potential representation with enhanced importance; compared with a single Gaussian model, a Gaussian mixture model is more flexible; content-adaptive optimization of the potential representation is performed, and a more reasonable and flexible probability distribution model is obtained by using Gaussian mixture distribution prediction, thereby reducing redundant information in the potential representation or neural grammar, and obtaining a reconstructed image with better visual quality while also improving the rate-distortion performance of the deep learning image compression model.

[0024] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0026] Figure 1 A schematic diagram of a deep learning image compression model based on importance scale and Gaussian mixture model according to an embodiment of the present invention;

[0027] Figure 2 Schematic diagram of an importance scale network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0029] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0030] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0031] See also Figure 1 , is a schematic diagram of a deep learning image compression model based on importance scale and Gaussian mixture model according to an embodiment of the present invention;

[0032] This embodiment provides a deep learning image compression method based on importance scale and Gaussian mixture model, including:

[0033] Step 1: The image to be processed is first analyzed and transformed through the main encoder to obtain the corresponding potential representation. The specific structure of the main encoder consists of four two-dimensional zero padding (Zero Padding 2D Layer, ZeroPad2d) layers, four convolutional layers and three generalized divisive normalization (Generalized Divisive Normalization, GDN) layers.

[0034] Step 2: Please refer to Figure 2 , is a schematic diagram of an importance scale network of an embodiment of the present invention. The main parts of the importance scale network are a convolutional layer and three residual blocks in sequence, and then the hyperbolic tangent function (HyperbolicTangent Function, tanh) and the soft sign function (Softsign Function, softsign) are used in sequence to map the features to the range of [-1,1]. The potential representation obtained by the main encoder is used as input, and the importance scale network is used to obtain a potential representation with enhanced importance. The importance scale network is used to realize adaptive bit rate allocation of feature content, and a higher bit rate is allocated to any area with a larger potential value in any channel to ensure that more feature information is extracted.

[0035] Step 3: The importance-enhanced latent representation obtained by the main encoder and the importance scale network is divided into content stream and model stream in the channel dimension for separate processing.

[0036] Step 4: Process the potential representation of the content stream, and obtain a more accurate probability distribution through the hyper-prior model, the context model, and the preset Gaussian mixture model to guide entropy coding and decoding, so as to obtain a more accurate reconstructed potential representation, which specifically includes the following steps:

[0037] Step 4.1, the latent representation of the content stream is quantized, entropy encoded and decoded by the super-prior codec to obtain the quantized information of the auxiliary information for subsequent probability distribution modeling; the above-mentioned super-prior model is composed of a super-prior encoder and a super-prior decoder. The above-mentioned super-prior encoder is composed of three convolutional layers, each of which is followed by a rectified linear unit (ReLU) function. The purpose is to extract the features of the input data through the convolutional layer, and use the ReLU function to introduce nonlinear characteristics, thereby enhancing the expression and learning capabilities of the model; the above-mentioned super-prior decoder is composed of three transposed convolutional layers arranged in sequence, each of which is followed by a ReLU function. The purpose of this structural design is to upsample the input features through the transposed convolutional layer, and use the ReLU function to introduce nonlinear characteristics, thereby enhancing the expression and reconstruction capabilities of the decoder.

[0038] Step 4.2: The potential representation is passed through the context model to obtain context information, which provides a basis for the subsequent entropy parameter estimation of the probability distribution and improves the accuracy of parameter estimation. The context model consists of three convolutional layers, each of which is followed by a leaky rectified linear unit (LeakyReLU) function. The purpose is to effectively extract features through the convolutional layer and introduce nonlinear characteristics using the LeakyReLU function, thereby enhancing the model's expressive power and its ability to capture context information. In this way, the context model can provide a more accurate and reliable basis for subsequent entropy parameter estimation.

[0039] Step 4.3, the Gaussian mixture model is composed of multiple Gaussian distributions with different means and variances, and different weights are assigned to each type of Gaussian distribution. Using multiple different types of Gaussian distributions can characterize a more complex probability distribution model to ensure that the information obtained is accurate enough. The more accurate probability distribution is then used to guide entropy coding and decoding, reduce redundant information, and obtain a more accurate reconstruction potential representation. The formula of the Gaussian mixture model is as follows:

[0040]

[0041] Among them, w represents the weight value of each Gaussian distribution, μ and σ represent the mean and variance of the Gaussian distribution, and K represents the total number of Gaussian models. Represents the reconstruction potential representation, The quantized information of the auxiliary information is represented. Considering the image compression effect and the convergence speed comprehensively, the Gaussian mixture model with K=4 is selected.

[0042] Step 5: The potential representation of the model flow is obtained through the neural grammar generator and the weight generator, as well as the preset Gaussian mixture model, to obtain the kernel parameters of the last transposed convolutional layer of the main decoder, which specifically includes the following steps:

[0043] Step 5.1, the neural grammar generator is composed of a multi-scale network, a global average pooling layer and a fully connected layer. The potential representation in the model flow is fully extracted by the multi-scale network. The feature information of different scales is then mapped to a one-dimensional potential vector through the global average pooling layer and connected. The connected feature vector is called a neural grammar, which provides feature information for the subsequent weight generator.

[0044] Step 5.2: quantize, entropy encode and decode the neural grammar, and use the more accurate probability distribution predicted by the Gaussian mixture model to guide the entropy encoding and decoding to obtain more accurate neural grammar information; the Gaussian mixture model used in this step is the same as that in step 4.3;

[0045] Step 5.3. The main part of the weight generator is composed of three fully connected layers, each of which is followed by a LeakyReLU function as a nonlinear activation, forming a multi-layer fully connected neural network. The decoded neural grammar information is passed as input to the weight generator, which is then responsible for predicting the kernel parameters required by the main decoder in the last transposed convolutional layer. Since this process is based on the dynamic changes of the input information, the transformation parameters at the decoding end are also dynamically adjusted so that it can flexibly adapt to and process a variety of input images, ensuring that the model can generate outputs that match different images.

[0046] Step 6. The main decoder consists of five transposed convolutional layers, in which an inverse GDN layer (Inverse Generalized Divisive Normalization Layer) is embedded. After the training of the entire deep learning image compression model is completed, the parameters of all layers except the last transposed convolutional layer are fixed, and the kernel parameters of the last transposed convolutional layer are generated in real time by the weight generator during decoding, so they are highly dependent on the input image to be processed. The reconstructed potential representation obtained through the content stream and the kernel parameters of the last transposed convolutional layer of the decoding end obtained through the model stream are synthesized and transformed through the main decoder to obtain a reconstructed image with better visual effects.

[0047] For the convenience of description and concise expression, only the division of the above functional units and modules is used as an example for explanation. In practical applications, the above functions can be assigned to different functional units or modules according to requirements, that is, the internal structure of the model can be divided into different functional units or modules to realize all or part of the functions described above.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A deep learning image compression method based on importance scale and Gaussian mixture model, characterized in that: The following steps are involved: S1, the image to be processed is analyzed and transformed through the main encoder to obtain a potential representation; S2, passing the latent representation through a preset importance scale network, and performing adaptive content bit allocation for different region contents to obtain an importance-enhanced latent representation; S3, dividing the importance-enhanced latent representation into a latent representation of a content stream and a latent representation of a model stream in a channel dimension; S4, obtaining a probability distribution of a potential representation of the content stream by processing the hyper-prior model, the context model and the preset Gaussian mixture model to obtain a reconstructed potential representation; S5. Obtain the neural grammar of the potential representation of the model stream using a neural grammar generator; obtain the probability distribution of the neural grammar through the preset Gaussian mixture model to guide entropy coding and decoding to obtain neural grammar information; feed the neural grammar information to a weight generator to obtain the kernel parameters of the last transposed convolutional layer of the main decoder; S6. Performing a synthetic transformation on the reconstructed latent representation and the kernel parameters of the last transposed convolutional layer of the decoder through the main decoder to obtain a reconstructed image.

2. The deep learning image compression method based on importance scale and Gaussian mixture model according to claim 1, characterized in that: In S1: the main encoder includes four two-dimensional zero padding (Zero Padding 2D Layer, ZeroPad2d) layers, four convolutional layers and three generalized division normalization (Generalized Divisive Normalization, GDN) layers.

3. The deep learning image compression method based on importance scale and Gaussian mixture model according to claim 1, characterized in that: In S2: the importance scale network includes a convolutional layer and three residual blocks; an adaptive bit allocation strategy is adopted to allocate a higher bit rate to the area with a larger potential value in the area to extract more feature information, thereby obtaining a potential representation with enhanced importance.

4. The deep learning image compression method based on importance scale and Gaussian mixture model according to claim 1, characterized in that: In S4: the super-a priori model includes a super-a priori encoder and a super-a priori decoder; the super-a priori encoder includes three convolution layer groups; the super-a priori decoder includes three transposed convolution layers; and there is a rectified linear unit (ReLU) function between the convolution layers and between the transposed convolution layers; 5. The deep learning image compression method based on importance scale and Gaussian mixture model according to claim 1, characterized in that: In S4: the context model includes three convolutional layers, and leaky rectified linear unit (LeakyReLU) functions are provided between the convolutional layers; the context information of the potential representation is obtained through the context model.

6. The deep learning image compression method based on importance scale and Gaussian mixture model according to claim 1, characterized in that: In S4: the Gaussian mixture model is composed of multiple Gaussian distributions with different means, variances and weights, and the Gaussian mixture model is expressed as: Among them, w represents the weight value of each Gaussian distribution, μ and σ represent the mean and variance of the Gaussian distribution, and K represents the total number of Gaussian models. represents the reconstruction potential representation, Represents the quantitative information of auxiliary information.

7. The deep learning image compression method based on importance scale and Gaussian mixture model according to claim 1, characterized in that: In S5: the neural grammar generator includes a multi-scale network, a global average pooling layer and a fully connected layer; the latent representation of the model flow is extracted features by the multi-scale network, and is connected into a neural grammar feature vector after global average pooling for use by a subsequent weight generator.

8. The deep learning image compression method based on importance scale and Gaussian mixture model according to claim 1, characterized in that: In S5: the weight generator includes three fully connected layers, and there is a leaky rectified linear unit (LeakyReLU) function between the fully connected layers.

9. The deep learning image compression method based on importance scale and Gaussian mixture model according to claim 1, characterized in that: In S6: the main decoder includes five transposed convolutional layers, and there is an inverse GDN layer (Inverse Generalized Divisive Normalization Layer) between the convolutional layers; after the training of the entire deep learning image compression model is completed, the parameters of all transposed convolutional layers except the last transposed convolutional layer are fixed values; the kernel parameters of the last transposed convolutional layer are generated in real time by the weight generator during decoding.

10. The deep learning image compression method based on importance scale and Gaussian mixture model according to claim 6, characterized in that: The total number of Gaussian models K=4.

Images