Image compression method based on mixed neural representation

Abstract

The invention discloses an image compression method based on mixed neural representation. In order to solve the problem of high decoding complexity of the existing image compression method based on deep learning, the invention provides an image compression method based on mixed neural representation, and each image to be compressed is independently over-fitted with a lightweight auto-encoder network. Specifically, in an auto-encoder network, a packet feature aggregation network is designed for aggregating features of different groups; a local modulation network is designed for enhanced representation of local features; and a Gaussian mixture model is used, so that the accuracy of code rate estimation is improved. Experimental results show that the complexity of image decoding can be effectively reduced through the hybrid neural representation method, and the light weight of image compression is realized.

Description

Technical Field

[0001] The present invention relates to the fields of image compression and deep learning, and in particular to an image compression method based on hybrid neural representation. Background Art

[0002] Image compression has been an important research topic in the field of signal processing for decades. Classic image compression methods, such as JPEG and JPEG2000, have been widely used in image processing software. However, the explosive growth of multimedia content in the digital age still urgently requires finding more effective compressors to solve the storage cost problem.

[0003] With the development of deep learning, image compression methods based on autoencoders have been widely studied. This type of method uses an end-to-end autoencoder structure as a nonlinear transformation to achieve better performance. However, the structure of the autoencoder is very large and the computational complexity is high, which limits the application of this type of method on resource-constrained devices.

[0004] Recently, the theory of implicit neural representation has been proposed and improved. Implicit neural representation aims to use a continuous function to fit the mapping of spatial coordinates to corresponding position values. Unlike the traditional discrete grid storage form, implicit neural representation can use the representation power of continuous functions to represent complex scenes with compact neural networks, and thus has natural compression potential. This type of method uses a compact implicit network to overfit an image, thereby storing image information as model parameter information. This method has the advantage of low decoding complexity and can be decoded quickly on edge devices. However, there is still a certain gap between the compression performance of this type of method and the learning-based method.

[0005] The closest existing technology and its review:

[0006] Dupont et al. (Dupont E, Goliński A, Alizadeh M, et al. Coin: Compression with implicit neural representations [J]. arXiv preprint arXiv: 2103.03123, 2021.) proposed an image compression method based on implicit neural representation. For an image, a neural network is used to overfit and train the mapping from position coordinates to RGB values, and the trained network parameters are used to represent the compressed image. In this way, we transform the image compression problem into a model compression problem. Strumpler et al. (Strümpler Y, Postels J, Yang R, et al.: Implicit neural representations for image compression [C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland. 2022: 74–91.) applied the meta-learning method to the image compression task based on implicit neural representation, obtained a better initial value of its network parameters, and only needed a small number of iterations to achieve the compression quality that could only be achieved by training from scratch multiple times. Girish et al. (Girish S, Shrivastava A, Gupta K: Shacira: Scalable hash-grid compression for implicit neural representations. [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2023: 17513-17524.) improved the implicit neural representation based on hash tables. For each layer of hash tables, an implicit hash table with a dimension smaller than the feature dimension is used for decoding and generation.Catania et al. (Catania L, Allegra D: Nif: A fast implicit image compression with bottleneck layers and modulated sinusoidal activations [C] / / Proceedings of the 31st ACM International Conference on Multimedia. 2023: 9022–9031.) improved the implicit neural representation based on the periodic activation function, and adjusted the period of the hidden layer features through a dedicated neural network, thereby adapting to the changes in frequency in different regions of the image. The model uses a base network and a modulation network. The base network generates pixel features at the coordinate position, and the modulation network adjusts the period of the activation function. Summary of the invention

[0007] The object of the present invention is to provide an image compression method based on hybrid neural representation.

[0008] The technical solution of the present invention aims at the problem of poor decoding quality in the existing technology based on implicit neural representation, and proposes an image compression method based on hybrid neural representation. The decoder part of the autoencoder structure is used, and the overfitting idea of ​​implicit neural representation is utilized. Each image is represented by embedded features and a separate decoder. The decoder parameters are also transmitted as a bit stream, but since only one image needs to be decoded, a large decoder structure is not required. In order to improve the decoder's fitting of features of different scales, the method proposes a grouped feature aggregation network, which associates different channels of embedded features with different scale features, and extracts multi-scale features by connecting multiple feature aggregation blocks in parallel. At the same time, in order to improve the decoder's capture of local information, the method proposes a local feature modulation network, and the extracted multi-scale features are subjected to multiple local modulation blocks in series, which enhances the perception of local information. Finally, in order to improve the accuracy of bit rate estimation, the method uses a context-based mixed Gaussian entropy model, and uses a mixed Gaussian model with stronger fitting ability to fit the probability distribution of embedded features.

[0009] The technical solution of the present invention is:

[0010] An image compression method based on hybrid neural representation, characterized by comprising the following steps:

[0011] Step 1: Preprocess the original image to be compressed, including pixel normalization, as a training set;

[0012] Step 2: construct a lightweight image hybrid neural representation network, wherein the lightweight image hybrid neural representation network includes an encoder, a separable decoder and a bit rate estimation module. The separable decoder includes a group feature aggregation network and a local feature modulation network.

[0013] Step 3: Input the image of the training set obtained in step 1 into the lightweight image hybrid neural representation network constructed in step 2 to obtain the reconstructed image.

[0014] Step 4: Compare the reconstructed image obtained in step 3 with the input image, calculate the reconstruction loss, use the bitrate estimation module in step 2 to calculate the bitrate of the embedded features obtained after the original image passes through the encoder, and weightedly sum the reconstruction loss and bitrate to get the total loss;

[0015] Step 5: According to the total loss obtained in step 4, the lightweight image hybrid neural representation network is optimized using the Adam optimizer;

[0016] Step 6: Loop through steps 3 to 5, train for a specified number of rounds, save the trained model file, and compress the decoder network parameters in the trained image hybrid neural representation network and the embedded features of the original image after passing through the encoder as compressed data to obtain an image code stream;

[0017] Step 7: Send the image code stream to the decoding end for decompression, obtain the embedded features of the decoder and the original image after passing through the encoder, input the embedded features into the decoder, output the reconstructed image, denormalize the reconstructed image, and obtain the decoded image.

[0018] Furthermore, the step 2:

[0019] For the encoder, the input image x is taken as input and the embedded feature y is obtained:

[0020] y=g a (x)

[0021] The embedded features are quantized to obtain discrete embedded feature representations:

[0022]

[0023] In order to avoid the gradient being undifferentiable due to rounding operations, uniform noise is added during training. To approximate quantization:

[0024]

[0025] The separable decoder comprises a group feature aggregation network g m and local feature modulation network g ψThe discrete embedded feature representation is input to a separable decoder and outputs a reconstructed image.

[0026] Specifically, the grouped feature aggregation network contains 5 parallel implicit feature blocks. The discrete embedded features are first divided into 5 groups of features along the channel dimension, and the number of channels in each group is 1, 4, 16, 64, and 256 respectively. Then, the i-th group of features Input the i-th implicit feature block, where i is the index number of the group, and calculate the reorganized features:

[0027]

[0028] Among them, pixel shuffle is a pixel reorganization operation, 2 i-1 is the upsampling factor. Then, the reorganized features are implicitly aggregated with the pixel coordinates to obtain the aggregated features:

[0029] F i =gridsample(G i ,v)

[0030] Where gridsample is the grid sampling operation, and v is the pixel coordinate of the input image. Finally, the features of each group of aggregations are connected to obtain the output of the group feature aggregation network:

[0031]

[0032] The local feature modulation network includes three local modulation blocks connected in series. For the lth local modulation block, its input is the output D of the previous local modulation block. l-1 and a learnable modulation vector M l , the output is the feature D of the current local modulation block l For the first block, the output of the previous local modulation block is the output of the group feature aggregation network, that is: D0 = F.

[0033] The local modulation block first undergoes a linear transformation to obtain the intermediate features:

[0034] I l =W l D l-1 +b l

[0035] Where W l and b l are the weights and biases of the linear transformation respectively. Then, the intermediate feature is dot-producted with the upsampled modulation vector to obtain the output modulation feature:

[0036] D l =ReLU(I l ⊙upsample(Ml ))

[0037] Among them, ⊙ is the element-by-element multiplication, and upsample is the upsampling operation of bicubic interpolation.

[0038] Finally, a linear layer is used to map the modulated features to the image domain to obtain the reconstructed image:

[0039]

[0040] Furthermore, the step 4:

[0041] The context entropy model is used to estimate the probability of discrete embedded features and is modeled using a Gaussian mixture distribution:

[0042] [μ j ,σ j ,ω j ] = g φ (ctx j )

[0043] Among them, ctx j represents the context information of the jth position, μ j ,σ j ,ω j They represent the mean, standard deviation and weight of the Gaussian mixture distribution respectively.

[0044] Next, calculate the probability of the discrete embedding feature at the jth position:

[0045]

[0046] Bit rate loss is used to control the size of the compressed file. The calculation formula is as follows:

[0047]

[0048] The reconstruction loss is used to make the reconstructed image as similar as possible to the original image, and the calculation formula is as follows:

[0049]

[0050] The total loss function is calculated by weighting the bit rate loss and the reconstruction loss. The calculation formula is as follows:

[0051]

[0052] In the formula, λ is a weight parameter, which can be set by the user according to needs.

[0053] By adopting the above scheme, the beneficial effects of the present invention are:

[0054] 1. The present invention proposes an image compression method based on hybrid neural representation, which combines the autoencoder structure with implicit neural representation, effectively reducing the complexity of image decoding;

[0055] 2. The present invention improves the autoencoder network structure, wherein:

[0056] A grouped feature aggregation network is proposed to improve the decoder's fitting of features of different scales;

[0057] A local feature modulation network is proposed, which can effectively enhance the decoder's capture and perception of local information and improve the performance of image compression based on implicit neural representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a basic flow chart of the image compression method based on hybrid neural representation of the present invention.

[0059] Figure 2 It is a structural diagram of the lightweight image hybrid neural representation network of the present invention.

[0060] Figure 3 It is a structural diagram of the implicit aggregation block and the local modulation block of the present invention.

[0061] Figure 4 It is a visual comparison diagram of the present invention and the existing method.

[0062] Figure 5 This is the image compression comparison result between the present invention and the existing method.

[0063] Figure 6 This is the decoding complexity comparison result between the present invention and the existing method.

[0064] Figure 7 The following are the results of the network aggregation without and with grouping features.

[0065] Figure 8 These are the results without and with the local feature modulation network. DETAILED DESCRIPTION

[0066] The technical solution of the present invention is further described below in conjunction with the accompanying drawings. The described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0067] Figure 1 The flowchart shown:

[0068] Step 1: Preprocess the original image to be compressed, including pixel normalization. For a 512*768 RGB three-channel image, normalize the pixel values ​​from 0 to 255 to 0 to 1 as a training set.

[0069] Step 2: Build a lightweight image hybrid neural representation network (such as Figure 2 As shown), the lightweight image hybrid neural representation network includes an encoder, a separable decoder and a bit rate estimation module.

[0070] The encoder takes the input image x as input and outputs the embedded feature y:

[0071] y=g a (x)

[0072] The embedded feature y is quantized to obtain a discrete embedded feature representation

[0073]

[0074] In order to avoid the gradient being undifferentiable due to rounding operations, uniform noise is added during training. To approximate quantization:

[0075]

[0076] The separable decoder comprises a group feature aggregation network g m and local feature modulation network g ψ Discrete Embedded Feature Representation Input is a separable decoder, output is a reconstructed image

[0077] Specifically, the group feature aggregation network g m Contains 5 parallel implicit feature blocks, its structure is as follows Figure 3 As shown in a. Discrete embedding feature representation First, the features are divided into 5 groups along the channel dimension, with the number of channels in each group being 1, 4, 16, 64, and 256. Then, the i-th group of features Input the i-th implicit feature block, where i is the index number of the group, and calculate the reorganized feature G i :

[0078]

[0079] Among them, pixel shuffle is a pixel reorganization operation, 2 i-1 is the upsampling factor. Then, the reorganized feature G i Implicitly aggregate with pixel coordinates to obtain aggregated features F i :

[0080] F i =gridsample(G i ,v)

[0081] Where gridsample is the grid sampling operation, and v is the pixel coordinate of the input image. Finally, each group of aggregated features F i Connect them to get the output F of the group feature aggregation network:

[0082]

[0083] The local feature modulation network g ψ It contains three local modulation blocks connected in series, and its structure is as follows Figure 3 As shown in b. For the lth local modulation block, its input is the output D of the previous local modulation block l-1 and a learnable modulation vector M l , the output is the feature D of the current local modulation block l For the first block, the output of the previous local modulation block is the output F of the group feature aggregation network, that is: D0 = F.

[0084] The three local modulation blocks connected in series first undergo linear transformation to obtain the intermediate features:

[0085] I l =W l D l-1 +b l

[0086] Where W l and b l are the weights and biases of the linear transformation respectively. Then, the intermediate feature is dot-producted with the upsampled modulation vector to obtain the output modulation feature D l :

[0087] D l =ReLU(I l ⊙upsample(M l ))

[0088] Among them, ⊙ is the element-by-element multiplication, and upsample is the upsampling operation of bicubic interpolation.

[0089] Finally, a linear layer is used to map the modulated features to the image domain to obtain the reconstructed image

[0090]

[0091] Step 3: Input the image of the training set obtained in step 1 into the lightweight image hybrid neural representation network constructed in step 2 to obtain the reconstructed image.

[0092] Step 4: Compare the reconstructed image obtained in step 3 with the input image, calculate the reconstruction loss, use the bitrate estimation module in step 2 to calculate the bitrate of the embedded features obtained after the original image passes through the encoder, and weightedly sum the reconstruction loss and bitrate to get the total loss;

[0093] The context entropy model is used to estimate the probability of discrete embedded features and is modeled using a Gaussian mixture distribution:

[0094] [μ j ,σ j ,ω j ] = g φ (ctx j )

[0095] Among them, μ j ,σ j ,ω j Respectively represent the mean, standard deviation and weight of Gaussian mixture distribution, g φ () indicates that ctx j represents the context information of the jth position. Specifically, g φ A 3-layer MLP is used with 12 neurons in the hidden layer.

[0096] Next, calculate the probability of the discrete embedding feature at the jth position:

[0097]

[0098] Bit rate loss Used to control the size of the compressed file. The calculation formula is as follows:

[0099]

[0100] Reconstruction loss It is used to make the reconstructed image as similar to the original image as possible. The calculation formula is as follows:

[0101]

[0102] Total loss function Bit rate loss and reconstruction loss The weighted calculation formula is as follows:

[0103]

[0104] Wherein λ is a weight parameter, which can be set by the user according to the needs. In this embodiment, λ=0.003.

[0105] Step 5: Based on the total loss obtained in step 4, the Adam optimizer is used to optimize the lightweight image hybrid neural representation network described in step 2. The learning rate is initialized to 0.001. The learning rate decay optimizer is used. When the loss does not decrease after 10 training rounds, the learning rate is returned to the original 0.5.

[0106] Step 6: Loop through steps 3 to 5, and train for a specified number of rounds. In this embodiment, the number of training rounds is 20,000. After the training is completed, save the trained model file, and compress the decoder network parameters in the trained image hybrid neural representation network and the embedded features of the original image after the encoder as compressed data to obtain an image code stream. Specifically, arithmetic coding is used to compress the embedded features, and 8-bit quantization and entropy coding are used to compress the decoder parameters.

[0107] Step 7: Send the image code stream to the decoding end for decompression, obtain the embedded features of the decoder and the original image after passing through the encoder, input the embedded features into the decoder, output the reconstructed image, denormalize the reconstructed image, denormalize each pixel of the obtained image from 0 to 1 to the range of 0 to 255, and obtain the decoded image.

[0108] The method of the present invention was compared with the closest prior art in an experiment, and the data used in the experiment were Kodak and CelebA datasets. The Kodak dataset contains 24 natural images with a resolution of 512*768 or 768*512, and the dataset is widely used in image compression. For the CelebA dataset, which contains more than 200,000 face images with a resolution of 178×218, the method of the present invention was evaluated using the same test images as the Strumpler method.

[0109] The comparison results of image compression of this method with existing technologies such as COIN, NIF, Strumpler and image coding standards on Kodak and CelebA datasets are shown in the rate-distortion curves. Figure 4 As shown in Figure 2, under the same bit rate condition on the Kodak dataset, the PNSR of this method exceeds the baseline method NIF by about 1.0 dB. The image reconstruction results of this method and the prior art are compared as follows: Figure 5 As shown in the figure, from the visualization results, the visual quality of this method is better than the existing implicit neural representation based methods. In order to verify that this method has lower decoding complexity, the code rate is 0.3bpp, and the Kodak dataset image is compressed as an example. The complexity and decoding time of various methods using CPU decoding are calculated respectively, as shown in the figure below. Figure 6As shown. The decoding complexity of this method is significantly lower than that of the learning-based method, and is also somewhat lower than that of the implicit neural representation-based method. Specifically, the decoding complexity of this method is only 1% of the Ballé method and 5.3% of the NIF method. In terms of decoding time, the time of this method is close to that of the implicit neural representation-based method, and is much shorter than that of the learning-based method, and the decoding speed is increased by about 20 times.

[0110] The above-mentioned image compression implementation method based on hybrid neural representation proposes an attention loss function. For this purpose, this implementation method also conducts additional comparative experiments, which are the test results obtained without and with attention loss during training, such as Figure 4 As shown, the experimental results show that the attention loss makes the content structure clearer. This implementation method also conducts additional comparative experiments with the closest prior art. Figure 5 The content and style loss comparison results of 360 stylized images generated by the implementation method of the present invention and the prior art on 18 content images and 20 style images are shown in Figure 2. The experimental results show that the content and style loss of the implementation method of the present invention is the lowest in this test data. Figure 6 As shown: A comparison of the style transfer results of 8 content-style image pairs using the implementation method and the prior art. The experimental results show that the implementation method can effectively retain the significant structure of the content and the detailed texture of the style, and obtain a higher quality stylized image.

[0111] In order to prove the effectiveness of the module design introduced in the solution of the present invention, an additional comparative experiment was conducted in this embodiment. An ablation experiment was conducted on the decoder to compare the test results with and without the introduction of the group feature aggregation network. Figure 7 As shown in Figure 2, an ablation experiment was conducted on the decoder with the local feature modulation network introduced, and the test results with and without the local feature modulation network introduced were compared. Figure 8 shown.

[0112] The above description is only a description of the preferred embodiments of the present application, and is not intended to limit the scope of the present application. Any changes or modifications made by any person skilled in the art based on the above disclosed technical contents shall be deemed as equivalent effective embodiments and shall fall within the scope of protection of the technical solution of the present application.

Claims

1. An image compression method based on hybrid neural representation, characterized in that: The steps include: Step 1: Preprocess the original image to be compressed, including pixel normalization, as a training set; Step 2: constructing a lightweight image hybrid neural representation network, wherein the lightweight image hybrid neural representation network includes an encoder, a separable decoder and a bit rate estimation module; the separable decoder includes a group feature aggregation network and a local feature modulation network; Step 3: Input the training set image obtained in step 1 into the lightweight image hybrid neural representation network constructed in step 2 to obtain a reconstructed image; Step 4: Compare the reconstructed image obtained in step 3 with the input image, calculate the reconstruction loss, use the bitrate estimation module in step 2 to calculate the bitrate of the embedded features obtained after the original image passes through the encoder, and weightedly sum the reconstruction loss and bitrate to get the total loss; Step 5: According to the total loss obtained in step 4, the lightweight image hybrid neural representation network is optimized using the Adam optimizer; Step 6: Loop through steps 3 to 5, train for a specified number of rounds, save the trained model file, and compress the decoder network parameters in the trained image hybrid neural representation network and the embedded features of the original image after passing through the encoder as compressed data to obtain an image code stream; Step 7: Send the image code stream to the decoding end for decompression, obtain the embedded features of the decoder and the original image after passing through the encoder, input the embedded features into the decoder, output the reconstructed image, denormalize the reconstructed image, and obtain the decoded image.

2. The image compression method based on hybrid neural representation as claimed in claim 1, characterized in that: Step 2: For the encoder, the input image x is taken as input and the embedded feature y is obtained: y=g a (x) The embedded features are quantized to obtain discrete embedded feature representations: In order to avoid the gradient being undifferentiable due to rounding operations, uniform noise is added during training. To approximate quantization: The separable decoder comprises a group feature aggregation network g m and local feature modulation network g ψ The discrete embedded feature representation inputs a separable decoder and outputs a reconstructed image; Specifically, the grouped feature aggregation network contains 5 parallel implicit feature blocks. The discrete embedded features are first divided into 5 groups of features along the channel dimension, with the number of channels in each group being 1, 4, 16, 64, and 256 respectively; then, the i-th group of features Input the i-th implicit feature block, where i is the index number of the group, and calculate the reorganized features: Among them, pixel shuffle is a pixel reorganization operation, 2 i-1 is the upsampling factor. Then, the reorganized features are implicitly aggregated with the pixel coordinates to obtain the aggregated features: F i =gridsample(G i ,v) Where gridsample is the grid sampling operation, and v is the pixel coordinate of the input image; finally, each group of aggregated features is connected to obtain the output of the grouped feature aggregation network: The local feature modulation network comprises three local modulation blocks connected in series; for the lth local modulation block, its input is the output D of the previous local modulation block. l-1 and a learnable modulation vector M l , the output is the feature D of the current local modulation block l ; For the first block, the output of the previous local modulation block is the output of the group feature aggregation network, that is: D0 = F; The local modulation block first undergoes a linear transformation to obtain the intermediate features: I l =W l D l-1 +b l Where W l and b l are the weights and biases of the linear transformation respectively; then, the intermediate feature is dot-producted with the upsampled modulation vector to obtain the output modulation feature: D l =ReLU(I l ☉upsample(M l )) Among them, ⊙ is the element-by-element multiplication, upsample is the upsampling operation of bicubic interpolation; Finally, a linear layer is used to map the modulated features to the image domain to obtain the reconstructed image:

3. The image compression method based on hybrid neural representation as claimed in claim 1, characterized in that: Step 4: The context entropy model is used to estimate the probability of discrete embedded features and is modeled using a Gaussian mixture distribution: [m j ,s j ,oh j ]=g φ (ctx j ) Among them, ctx j represents the context information of the jth position, μ j ,σ j ,ω j Respectively represent the mean, standard deviation and weight of Gaussian mixture distribution; Next, calculate the probability of the discrete embedding feature at the jth position: Bit rate loss is used to control the size of the compressed file. The calculation formula is as follows: The reconstruction loss is used to make the reconstructed image as similar as possible to the original image, and the calculation formula is as follows: The total loss function is calculated by weighting the bit rate loss and reconstruction loss, and the calculation formula is as follows: In the formula, λ is a weight parameter, which can be set by the user according to needs.

Images