Image encoding, decoding and compression method based on frequency domain feature perception learning

Abstract

The application discloses an image coding, decoding and compression method based on frequency domain feature perception learning, comprising the following steps: obtaining first and second feature maps of a to-be-coded image through an encoding network composed of a frequency domain feature perception learning module; obtaining a hyper-prior feature binary code stream and hyper-prior information of the second feature map; dividing the second feature map into multiple channel groups; modeling the distribution of each channel group as a Gaussian distribution, and predicting the mean and variance of the Gaussian distribution through a channel autoregressive entropy model based on a Transformer; performing probability estimation on the second feature map according to the Gaussian distribution, and performing arithmetic coding to obtain a feature binary code stream of each channel group and combine the feature binary code streams into a feature binary code stream, which is combined with the hyper-prior feature binary code stream to obtain a compressed image binary code stream. The application can extract more compact latent variables, accurately model the distribution of three-dimensional features and estimate the code rate, and improve the rate-distortion performance of image compression.

Description

Technical Field

[0001] This invention belongs to the field of image compression and relates to an end-to-end image compression method. Specifically, it relates to an image encoding, decoding, and compression method based on frequency domain feature perceptual learning. Background Technology

[0002] Recently, deep learning-based image compression models have emerged as a promising solution for image storage and transmission, outperforming traditional codecs in rate-distortion metrics. Theoretically, the nonlinear transformations in deep learning-based image compression models provide more adaptive quantization units for multidimensional quantizers, overcoming the construction limitations of transform coding schemes. Therefore, this nonlinear transformation has become an important topic in end-to-end image compression to reduce the encoded bitrate.

[0003] A literature search of existing technologies revealed that Minnen et al.'s paper "Joint Autoregressive and Hierarchical Priors for Learned Image Compression," presented at the 2018 Conference and Workshop on Neural Information Processing Systems, employed convolutional neural networks (CNNs) to achieve nonlinear analysis and synthetic transformations. However, the local receptive field of CNNs limits their representativeness and leads to redundant latent representations. To address this issue, recent research has employed attention modules and Transformers, such as Zou et al.'s "Learned image compression with mixed transformer-CNN architectures," presented at the 2023 Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. This paper uses Transformers to capture nonlocal spatial relationships, thereby achieving better RD performance.

[0004] However, we note that while the frequency characteristics of natural images are very important in traditional image compression methods, no research has specifically addressed frequency characteristics in deep learning-based image compression models. This can easily lead to coding redundancy and increased coding overhead. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes an image encoding, decoding, and compression method based on frequency domain feature-aware learning. This method can extract more compact latent variables and accurately model the distribution of three-dimensional features and estimate the bit rate, thereby improving the rate-distortion performance of image compression.

[0006] In a first aspect, the present invention proposes an image coding method for image coding based on frequency domain feature-aware learning, comprising:

[0007] A first feature map and a second feature map of the image to be encoded are obtained, wherein the first feature map is a frequency decomposition feature map of the image to be encoded obtained by passing the image through a coding network composed of frequency-knowledgeable Transformer blocks; the second feature map is obtained by quantizing each feature value in the first feature map into an integer.

[0008] Obtain the binary code stream of the super-prior features of the second feature map and the corresponding super-prior information;

[0009] The second feature map is uniformly divided into multiple channel groups; the distribution of each channel group is parameterized and modeled as a Gaussian distribution, and a channel autoregressive model based on Transformer is used to establish the channel context relationship of each group. Combined with prior information, the mean and variance of the Gaussian distribution of each group are predicted.

[0010] Probability estimation is performed on each channel group based on the predicted mean and variance of the Gaussian distribution, and arithmetic coding is performed based on the estimated probability to obtain the characteristic binary code stream of each channel group;

[0011] Each of the channel group feature binary streams is merged into a feature binary stream, and then further merged with the prior feature binary stream to obtain a compressed image binary stream.

[0012] Optionally, the step of obtaining a frequency decomposition feature map as the first feature map by passing the image to be encoded through an encoding network composed of multiple frequency domain feature-aware learning modules includes:

[0013] The image to be encoded is passed through the frequency domain feature perception learning modules in sequence. The input of the first frequency domain feature perception learning module is the image to be encoded, the input of the kth frequency domain feature perception learning module is the output of the (k-1)th frequency domain feature perception learning module, and the output of the last frequency domain feature perception learning module is used as the first feature map. Here, k is a positive integer greater than 1.

[0014] For any frequency domain feature-aware learning module, it includes:

[0015] The input is processed through a frequency-decomposable window multi-head attention mechanism. This mechanism divides the heads evenly into four groups, each using a rectangular window of a different size to partition the input features. An attention mechanism is then applied within each local window to extract features of specific frequencies. Specifically: the first group's windows are used to extract high-frequency features; the second group's windows are used to extract low-frequency features; the third group's windows are used to extract horizontal frequency features; and the fourth group's windows are used to extract vertical frequency features. Finally, the outputs of each group are concatenated to obtain the final output.

[0016] The output and input of the frequency-decomposable window multi-head attention mechanism are short-circuited and fed into a frequency-selective feedforward network. This network first passes the input features through a simple feedforward network consisting of a GELU nonlinear mapping layer and two multilayer perceptron layers, and then performs a Fast Fourier Transform (FFT) on the output. The result of the block-wise FFT is then multiplied by a learnable filter to achieve adaptive frequency selection. Finally, the frequency-selective features obtained through an inverse FFT are used as the output. The learnable filter is a three-dimensional tensor with dimensions of length, width, and channels, and the number of channels is twice the number of channels in the input features.

[0017] The features of different frequency components are connected to the frequency selection features through a short connection operation, and the result is used as the output of the frequency domain feature perception learning module.

[0018] Optionally, the step of sequentially passing the second feature map through a super-coding neural network to obtain a super-prior feature binary code stream, and passing the super-prior feature binary code stream through a super-decoding neural network to obtain super-prior information, includes:

[0019] The first hyperprior feature map is obtained by applying a hypercoding neural network to the second feature map;

[0020] Each feature in the first hyper-prior feature map is quantized into an integer to obtain the second hyper-prior feature map;

[0021] An independent distribution estimate is established for each feature in the second hyperprior feature map, and this distribution estimate is used to perform arithmetic coding on the second hyperprior feature map to obtain the hyperprior feature binary code stream.

[0022] The second hyperprior feature map is obtained by arithmetic decoding of the hyperprior feature binary code stream using the independent distribution estimation among the features;

[0023] The recovered second hyperprior feature map is processed by a hyperdecoding neural network to obtain hyperprior information.

[0024] A second aspect of the present invention provides an image decoding method for image coding obtained by frequency domain feature-aware learning, for correspondingly decoding the coding obtained by the above method, comprising:

[0025] Obtain the compressed image binary bitstream, and decompose it to obtain the feature binary bitstream and the prior feature binary bitstream;

[0026] Super-prior information is obtained from the super-prior feature binary code stream using a super-decoding neural network;

[0027] The channel context information of each group is established by the Transformer-based channel autoregressive model, and the channel context features of each group channel are output; the mean and variance of the Gaussian model in each channel group are predicted for decoding the second feature map.

[0028] The Transformer-based channel autoregressive model consists of L Transformer network layers. The input to the first Transformer layer is the second feature map, and the inputs to the remaining L-1 layers are the outputs of the previous Transformer layer. The output of the last layer is the final channel context feature for each grouped channel. Each Transformer layer includes a grouped mask channel multi-head attention mechanism layer, a GELU nonlinear mapping layer, and two grouped convolutional layers. The number of groups in the grouped convolutional layers is equal to the number of groups of the second feature map. For example, there can be two grouped convolutional layers, where the output of the first grouped convolutional layer is the input of GELU, and the output of GELU is the input of the second grouped convolutional layer.

[0029] Optionally, the channel multi-head attention mechanism layer of the group masking masks the attention weights of the undecoded group channels to the decoded group channels; through the channel attention mechanism, the decoded group channels can provide context information for other group channels; the main feature of the GELU nonlinear mapping layer is that when the input (the output of the previous group convolutional layer of GELU) is close to zero, its output is close to a linear transformation; while when the input is far from zero, its output is close to a saturated nonlinear transformation.

[0030] The channel context features and prior information of each group channel are concatenated, and the mean and variance of the Gaussian distribution of each group are predicted by a parameter estimation network.

[0031] Arithmetic decoding of the feature binary code stream is performed based on the mean and variance of the Gaussian model, and the second feature map is obtained by grouping the channels one by one in the order of channel grouping.

[0032] The second feature map is passed through a decoding neural network to obtain a reconstructed image.

[0033] A third aspect of the present invention provides an image compression method for image coding based on frequency domain feature-aware learning, comprising:

[0034] Obtain the image to be compressed;

[0035] Based on the above frequency-based Transformer image coding method, a compressed image binary bitstream is obtained;

[0036] Based on the frequency-based Transformer image decoding method described above, the compressed image binary code stream is decompressed to obtain the reconstructed image.

[0037] A fourth aspect of the present invention provides an image compression system, the image compression system comprising an encoder and a decoder, comprising:

[0038] Obtain the image to be compressed;

[0039] The encoder uses the frequency-based Transformer image encoding method described above to compress the image to be compressed, thereby obtaining a compressed image binary bitstream;

[0040] The decoder uses the frequency-based Transformer image decoding method described above to decompress the compressed image binary code stream to obtain the reconstructed image;

[0041] A fifth aspect of the present invention provides an image compression apparatus, the image compression apparatus comprising a memory and a processor, the memory storing a program executable by the processor, the processor executing the program and performing any of the following methods:

[0042] -The image coding method described above for frequency domain feature-aware learning, or,

[0043] -The image decoding method for image encoding based on frequency domain feature-aware learning described above, or,

[0044] -The image compression method described above is based on frequency domain feature-aware learning for image coding.

[0045] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0046] The image encoding, decoding, and compression method of the frequency domain feature-aware learning described above in this invention utilizes frequency-aware Transformer blocks to extract feature information of different frequency components, which can reduce information redundancy by generating more compact implicit representations.

[0047] The image encoding, decoding, and compression method of the frequency domain feature-aware learning described above in this invention, based on the channel autoregressive model of Transformer, can better capture the relevant information of changing channels, thereby increasing the efficiency of entropy bit rate and achieving better compression performance.

[0048] The image encoding, decoding, and compression method of the frequency domain feature-aware learning described above in this invention can save 2.78% of the bitstream overhead and improve rate-distortion performance compared with current methods, while maintaining the same image reconstruction quality.

[0049] The image encoding, decoding, and compression methods described above using frequency domain feature-aware learning in this invention can improve the accuracy of bitrate estimation and enhance image compression rate-distortion performance on natural images of different resolutions acquired in various scenarios. This invention has significant practical application value, especially in actual image compression, where it can flexibly estimate feature map distribution based on different image content and computational complexity constraints to achieve bitrate estimation and improve rate-distortion performance. Attached Figure Description

[0050] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0051] Figure 1 This is a flowchart of an image encoding method according to an embodiment of the present invention.

[0052] Figure 2 This is a flowchart of an image encoding method in a preferred embodiment of the present invention.

[0053] Figure 3 This is a flowchart of an image decoding method in a preferred embodiment of the present invention.

[0054] Figure 4 This is a flowchart of an image compression method in a preferred embodiment of the present invention.

[0055] Figure 5 This is a flowchart of an image compression model training method in a preferred embodiment of the present invention. Detailed Implementation

[0056] The present invention will now be described in detail with reference to specific embodiments and accompanying drawings. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0057] Reference Figure 1The diagram shows a flowchart of an image coding method based on frequency domain feature-aware learning according to an embodiment of the present invention. This embodiment provides an image coding method based on frequency domain feature-aware learning, including steps S100-S600. Specifically:

[0058] S100, Obtain the first feature map and the second feature map of the image to be encoded;

[0059] In this step, the image to be encoded is first determined, and then the first and second feature maps of the image are obtained through an encoding neural network. Specifically, the encoding neural network consists of multiple frequency-aware Transformer blocks (frequency domain feature-aware learning modules).

[0060] S200, obtain the binary code stream of the super-prior features of the second feature map and the corresponding super-prior information;

[0061] In this step, a neural network can be used. For example, the second feature map can be passed through a super-coding neural network to obtain a super-prior feature binary code stream. Furthermore, the super-prior feature binary code stream can be passed through a super-decoding neural network to obtain super-prior information.

[0062] S300, the second feature map is evenly divided into multiple channel groups to obtain grouped second feature maps;

[0063] S400 parameterizes the distribution of each channel group in S300 into a Gaussian distribution, establishes the channel context information of each group based on the channel autoregressive model of Transformer, and outputs the channel context features of each group channel; the channel context features and prior information of each group channel are concatenated, and the mean and variance of the Gaussian distribution of each group are predicted by a parameter estimation network.

[0064] In this step, a channel autoregressive model based on Transformer is used to establish the channel context information of each group, which can better capture the changing channel-related information, thereby increasing the efficiency of entropy bit rate and achieving better compression performance.

[0065] S500 performs probability estimation for each channel group based on the predicted mean and variance of the Gaussian distribution, and performs arithmetic coding based on the estimated probability to obtain the characteristic binary code stream of each channel group.

[0066] S600 merges the grouped feature binary bitstreams of each channel into a feature binary bitstream, and further merges it with the prior feature binary bitstream to obtain a compressed image binary bitstream.

[0067] In the above embodiments of the present invention, the distribution of three-dimensional features can be accurately modeled and the bit rate estimated. This can be used in practical image compression, and the feature map distribution can be flexibly estimated according to different image content and different computational complexity constraints to achieve bit rate estimation and improve rate-distortion performance. Of course, the above embodiments do not require strict adherence to the above steps in order. For example, steps S200 and S300 can be performed in any order to achieve the purpose of the present invention.

[0068] In one specific embodiment, the frequency-aware Transformer block includes a frequency-decomposable window multi-head attention mechanism, a frequency-selective feedforward network, and performs corresponding short connection operations; the encoding network consists of multiple frequency-aware Transformer blocks and extracts the first feature map of frequency decomposition.

[0069] In some possible implementations, the frequency-decomposable windowed multi-head attention mechanism divides the heads evenly into four groups. Each group uses a rectangular window of a different size to partition the input features, and an attention mechanism is implemented within the local window to extract features of specific frequencies. For example, the first group of heads has a 4x4 window size for extracting high-frequency features; the second group has a 16x16 window size for extracting low-frequency features; the third group has a 4x16 window size for extracting horizontal frequency features; and the fourth group has a 16x4 window size for extracting vertical frequency features. Finally, the outputs of each group are concatenated to obtain the final output. Of course, other window sizes can be used in other embodiments.

[0070] In some possible implementations, different frequency component features are input into a frequency-selective feedforward network. This network first passes the input features through a simple feedforward network consisting of a GELU nonlinear mapping layer and two multilayer perceptron layers, and then performs a 16x16 block Fast Fourier Transform (FFT) on the output. The result of the block FFT is multiplied by a learnable filter to achieve adaptive frequency selection. Finally, a 16x16 block inverse FFT is performed as the output. The learnable filter is a three-dimensional tensor with dimensions of length, width, and channels, where the length is 16, the width is 9, and the number of channels is twice the number of input feature channels. Optionally, each feature value in the first feature map is quantized into an integer to obtain the second feature map. The second feature map is then evenly divided into multiple channel groups to obtain grouped second feature maps. Of course, in other embodiments, other block sizes are also possible.

[0071] Figure 2 This is a flowchart of an image coding method based on frequency domain feature-aware learning in a preferred embodiment of the present invention. In this preferred embodiment, steps S101-S109 may be included, specifically:

[0072] S101, 3D feature extraction: The image to be encoded obtains frequency decomposition features through an encoding neural network composed of frequency-aware Transformer blocks;

[0073] S102, Prior Feature Extraction and Quantization: Super-prior features are obtained by using a super-coding neural network on the 3D features; the prior features are rounded and quantized into integers;

[0074] S103, Super-prior feature encoding: Establish a distribution estimate that is independent of each feature for the super-prior features, and use this distribution estimate to perform arithmetic encoding on the super-prior features to obtain the super-prior feature binary code stream;

[0075] S104, Super-prior feature decoding: Arithmetic decoding of the super-prior feature binary code stream is performed to obtain the recovered super-prior features; the recovered super-prior features are then processed by a super-decoding neural network to obtain super-prior information;

[0076] S105, Quantization: Rounds the three-dimensional features to integers;

[0077] S106, Three-dimensional feature channels: Divide the second feature map into channel groups evenly along the channel dimension.

[0078] S107, Probability Entropy Modeling Based on Transformer: The distribution of each channel group is parameterized and modeled as a Gaussian distribution. A channel autoregressive model based on Transformer is used to establish the channel context relationship of each group and output the channel context features of each group channel. The channel context features and prior information of each group channel are concatenated and the mean and variance of the Gaussian distribution of each group are predicted through a parameter estimation network.

[0079] S108, Three-dimensional feature encoding: Probability estimation is performed on each channel group based on the predicted mean and variance of the Gaussian distribution, and arithmetic encoding is performed based on the estimated probability to obtain the feature binary code stream of each channel group;

[0080] S109, Bitstream synthesis: Combine the feature binary bitstreams of each channel group into a feature binary bitstream, and further combine it with the prior feature binary bitstream to obtain a compressed image binary bitstream.

[0081] In the above embodiments of the present invention, by grouping specific three-dimensional feature channels and estimating the parameters of the entropy model based on Transformer, the distribution of three-dimensional features can be modeled and the bit rate estimated more accurately, thereby improving the rate-distortion performance of image compression.

[0082] Based on the same inventive concept, another embodiment of the present invention provides an image decoding method using frequency domain feature-aware learning, the flowchart of which is shown below. Figure 3 As shown, it includes:

[0083] S201, Bitstream decomposition: Decompose the compressed image binary bitstream to obtain the feature binary bitstream and the prior feature binary bitstream;

[0084] S202, Super-prior feature decoding: Perform arithmetic decoding on the binary code stream of super-prior features to obtain the recovered super-prior features; apply a super-decoding neural network to the recovered super-prior features to obtain super-prior information;

[0085] S203, Probabilistic Entropy Modeling Based on Transformer: The structure and parameter values ​​of this module are similar to... Figure 2 The Transformer-based probability entropy modeling in the illustrated embodiments is exactly the same.

[0086] For example, the three-dimensional features are first initialized with all-zero tensors; the three-dimensional features are non-uniformly divided into K channel groups along the channel dimension; the channel context information of each group is established by probabilistic entropy modeling based on Transformer, and the channel context features of each group channel are output; the channel context features and prior information of each group channel are concatenated, and the mean and variance of the Gaussian distribution of each group are predicted by a parameter estimation network.

[0087] S204, 3D feature decoding: Arithmetic decoding of the feature binary code stream is performed based on the mean and variance of the Gaussian model, and the quantized 3D features are obtained by grouping channels one by one in the order of channel grouping.

[0088] S205, Image Reconstruction: The quantized 3D features of the reconstruction are processed using a decoding neural network to obtain the reconstructed image.

[0089] The frequency-based Transformer-based image decoding method in this embodiment is used to correspond to... Figure 2 The frequency-aware Transformer-based image coding method shown utilizes a Transformer-based channel autoregressive model to estimate the channel context information for each group. This, combined with prior information, predicts the mean and variance of the Gaussian model, resulting in a more accurate distribution estimate. For details not elaborated above, please refer to [link to relevant documentation]. Figure 2 The corresponding descriptions in the illustrated embodiments will not be repeated here.

[0090] Based on the same inventive concept, in another embodiment, a frequency domain feature-aware learning-based image compression method is also provided. Taking the compression of 24 images with a resolution of 512×768 in the Kodak test set as an example, the flowchart of the method is as follows: Figure 4 As shown, it includes:

[0091] S301 Acquisition Module: Sequentially acquires the images to be compressed;

[0092] S302 encoder: Encodes the image to be compressed according to steps S101-S109 to obtain the compressed image binary code stream.

[0093] For example, the parameters are as follows: the number of channels for the three-dimensional feature is 320, the height is 32, the width is 48, and it is evenly divided into K=5 channel groups along the channel dimension; the specific values ​​of the preset parameters of the channel autoregressive model based on Transformer are: number of layers L=12;

[0094] S303 Decoder: Decompresses the compressed image binary stream according to S201-S205 to obtain the reconstructed image.

[0095] The number of channels, height, width, number of channel groups, and frequency-based preset parameters of the Transformer are exactly the same as those of the encoder for the three-dimensional features.

[0096] The compression method in the above embodiments of the present invention is based on Figure 1 , 2 Frequency-based Transformer image coding methods and Figure 3 The image encoding method based on frequency-based Transformer is shown. For parts not described in detail above, please refer to the corresponding descriptions in the above embodiments, which will not be repeated here.

[0097] The compression method in the above embodiments of the present invention can improve the accuracy of bitrate estimation and improve the rate-distortion performance of image compression on natural images of different resolutions collected in different scenarios. It can flexibly estimate the feature map distribution according to different image content and different computational complexity constraints to achieve bitrate estimation and improve rate-distortion performance.

[0098] Based on the same inventive concept, another embodiment of the present invention provides an image compression method, comprising: acquiring an image to be compressed; obtaining a compressed image binary code stream using the image encoding method of frequency domain feature perception learning in any of the above embodiments; and decompressing the compressed image binary code stream using the image decoding method of frequency domain feature perception learning in any of the above embodiments to obtain a reconstructed image.

[0099] The image compression methods described above can be improved by training a model. For example, the training set consists of one million natural images of different resolutions collected in various natural scenes; the trained model can then be used for practical image encoding and decoding. Specifically, such as... Figure 5 As shown, in a preferred embodiment, the image compression method includes:

[0100] S401, 3D Feature Extraction: Randomly crop the images in the training set to a resolution of 256×256, and input them into an encoding neural network to obtain 3D features. The encoding neural network consists of a residual block, two frequency-aware Transformer blocks, another residual block, two frequency-aware Transformer blocks, another residual block, two frequency-aware Transformer blocks, and a convolutional layer cascaded sequentially. The number of 3D feature channels is 320, and the height and width are both 16. Of course, the image resolution and 3D feature parameters described above in this embodiment can be adjusted according to actual conditions and are not limited to those described in this step. The same applies to the parameters in the following steps. Correspondingly, the encoding neural network can also adopt other network structures, such as four cascaded convolutional layers, and is not limited to the encoding neural network in this step.

[0101] S402, Super-prior feature extraction and noise addition: The 3D features are input into the super-prior coding neural network to obtain super-prior features. To avoid the gradient being zero everywhere due to quantization, noise with a uniform distribution from -0.5 to 0.5 is added to the super-prior features to simulate actual quantization. The super-prior coding neural network consists of three concatenated convolutional layers. Of course, the super-prior coding neural network can also be other network structures, such as three concatenated residual modules, and is not limited to the super-prior coding neural network in this step.

[0102] S403, Super-prior feature code rate estimation: Establish a distribution estimate that is independent of each feature for the super-prior features, and use this distribution estimate to obtain the super-prior feature code rate estimate;

[0103] S404, Acquisition of super-prior information: Input the noise-added super-prior features into the super-prior decoding neural network to obtain super-prior information, wherein: the super-prior decoding neural network is composed of three concatenated convolutional layers;

[0104] S405, Feature Noise Addition: To avoid the gradient being zero everywhere due to quantization, noise with a uniform distribution from -0.5 to 0.5 is added to the 3D features to simulate actual quantization;

[0105] S406, Feature Channels: Divide the 3D features evenly into K=5 channel groups along the channel dimension.

[0106] S407, Probabilistic Entropy Modeling Based on Transformer: The structure of this module is similar to... Figure 2The channel autoregressive model based on Transformer in the illustrated embodiment is exactly the same; by parameterizing and modeling the distribution of each channel group as a Gaussian distribution, a channel autoregressive model based on Transformer is used to establish the channel context relationship of each group and output the channel context features of each group channel; the channel context features and prior information of each group channel are concatenated, and the mean and variance of the Gaussian distribution of each group are predicted by a parameter estimation network: number of layers L = 12;

[0107] S408, Feature bitrate estimation: Obtain the bitrate estimate for each channel group of the three-dimensional feature based on the mean, variance and weight of the predicted Gaussian distribution; add the bitrate estimates of each channel group to obtain the feature bitrate estimate;

[0108] S409, Total Bitrate Estimation: Add the prior feature bitrate estimate and the feature bitrate estimate to obtain the total bitrate estimate;

[0109] S410, Image Reconstruction: Input the noisy 3D features into the decoding neural network to obtain the reconstructed image, wherein: the decoding neural network consists of a residual block, two frequency-aware Transformer blocks, a residual block, two frequency-aware Transformer blocks, a residual block, two frequency-aware Transformer blocks, and a convolutional layer cascaded in sequence.

[0110] In some possible implementations, the above image compression method can be used as an image compression model, which can be further optimized using stochastic gradient descent, with a rate-distortion loss function as follows:

[0111]

[0112] Among them, distortion x is the image to be compressed. This is a reconstructed image. λ>0 is a preset parameter used to balance distortion and bitrate. The bitrate R is calculated based on the Gaussian distribution of each channel group in each channel group of the second feature map.

[0113]

[0114] in, The characteristic of the k-th channel group θ is the predicted distribution in the k-th channel group. k ={φ k ,ψ k} is a set of learnable parameters.

[0115] Implementation results:

[0116] In the above embodiment, the frequency-aware Transformer-based image compression model was trained using the ImageNet dataset. λ was set to 0.0018, 0.0035, 0.0067, 0.0130, 0.0250, and 0.0483, corresponding to six different bitrate image compression models. The Adam optimizer was used for 2,000,000 optimization steps, with an initial learning rate of 10 for the first 1,800,000 steps. -4 It then dropped to 3*10 -5 Finally, it dropped to 10 at 1,950,000 steps. -5 .

[0117] In the above embodiment, image encoders based on six frequency-aware Transformer image compression models were trained to compress 24 images with a resolution of 512×768 in the Kodak test set. The average bitrates of the 24 compressed images were 0.1284 bpp, 0.1983 bpp, 0.2963 bpp, 0.4332 bpp, 0.6108 bpp, and 0.8360 bpp, respectively. In the above embodiment, image encoders based on six frequency-aware Transformer image compression models were trained to compress 24 images with a resolution of 512×768 in the Kodak test set to obtain reconstructed images. The average peak signal-to-noise ratios (PSNR) of the 24 reconstructed images were 29.650, 31.142, 32.712, 34.430, 36.180, and 37.928, respectively.

[0118] Compared to methods using baseline neural networks, the frequency-aware Transformer-based image compression method provided in the above embodiments of the present invention exhibits better rate-distortion performance. Specifically, it can save 2.78% of bitstream overhead under the same reconstructed image distortion. The frequency-aware Transformer-based image compression method of the present invention can improve the accuracy of bitrate estimation and enhance image compression rate-distortion performance on natural images of different resolutions acquired in different scenarios. The present invention has strong practical application value, especially in actual image compression, where it can flexibly estimate feature map distribution based on different image content and different computational complexity constraints to achieve bitrate estimation and improve rate-distortion performance.

[0119] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0121] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0122] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0123] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.

Claims

1. An image coding method based on frequency domain feature-aware learning, characterized in that, include: The image to be encoded is passed through an encoding network composed of multiple frequency domain feature perception learning modules to obtain a frequency decomposition feature map, which is used as the first feature map. Each feature value in the first feature map is quantized into an integer to obtain the second feature map; Obtain the binary code stream of the super-prior features of the second feature map and the corresponding super-prior information; The second feature map is evenly divided into multiple channel groups, and the distribution of each channel group is parameterized and modeled as a Gaussian distribution. A channel autoregressive model based on Transformer is used to establish the channel context relationship of each group, and the mean and variance of the Gaussian distribution of each group are predicted by combining the prior information. Probability estimation is performed on each channel group based on the predicted mean and variance of the Gaussian distribution, and arithmetic coding is performed based on the estimated probability to obtain the characteristic binary code stream of each channel group; All the channel group feature binary streams are merged into a feature binary stream, and then further merged with the prior feature binary stream to obtain a compressed image binary stream; The step of obtaining a frequency decomposition feature map, which serves as the first feature map, by passing the image to be encoded through an encoding network composed of multiple frequency domain feature perception learning modules includes: The image to be encoded is passed through the frequency domain feature perception learning modules sequentially. The input of the first frequency domain feature perception learning module is the image to be encoded, the input of the kth frequency domain feature perception learning module is the output of the (k-1)th frequency domain feature perception learning module, and the output of the last frequency domain feature perception learning module is used as the first feature map. Here, k is a positive integer greater than 1. For any frequency domain feature-aware learning module, it includes: The input is processed through a frequency-decomposable window multi-head attention mechanism. This mechanism divides the heads evenly into four groups, each using a rectangular window of a different size to partition the input features. An attention mechanism is then applied within each local window to extract features of specific frequencies. Specifically: the first group's windows are used to extract high-frequency features; the second group's windows are used to extract low-frequency features; the third group's windows are used to extract horizontal frequency features; and the fourth group's windows are used to extract vertical frequency features. Finally, the outputs of each group are concatenated to obtain the final output. The output and input of the frequency-decomposable window multi-head attention mechanism are short-circuited and fed into a frequency-selective feedforward network. This network first passes the input features through a simple feedforward network consisting of a GELU nonlinear mapping layer and two multilayer perceptron layers, and then performs a Fast Fourier Transform (FFT) on the output. The results of the block-wise FFT are then multiplied by a learnable filter to achieve adaptive frequency selection. Finally, an inverse FFT is performed to obtain the frequency-selective features as the output. The learnable filter is a three-dimensional tensor with dimensions of length, width, and channels, and the number of channels is twice the number of channels in the input features. The features of different frequency components are connected to the frequency selection features through a short connection operation, and the result is used as the output of the frequency domain feature perception learning module.

2. The image coding method for frequency domain feature-aware learning according to claim 1, characterized in that, The step of obtaining the prior feature binary code stream and prior information of the second feature map includes: The second feature map is processed by a super-coding neural network to obtain the first super-prior feature map; Each feature in the first hyper-prior feature map is quantized into an integer to obtain the second hyper-prior feature map; An independent distribution estimate is established for each feature in the second hyperprior feature map, and this distribution estimate is used to perform arithmetic coding on the second hyperprior feature map to obtain the hyperprior feature binary code stream. The second hyperprior feature map is obtained by arithmetic decoding of the hyperprior feature binary code stream using the independent distribution estimation among the features; The recovered second hyperprior feature map is processed by a hyperdecoding neural network to obtain hyperprior information.

3. The image coding method for frequency domain feature-aware learning according to claim 1, characterized in that, The method employs a Transformer-based channel autoregressive model to establish the channel context relationships for each group, and combines prior information to predict the mean and variance of the Gaussian distribution for each group, including: The second feature map of the group is input into the channel autoregressive model based on Transformer to establish the channel context information of each group and output the channel context features of each group channel. The Transformer-based channel autoregressive model comprises an L-layer Transformer network. The input to the first layer of the Transformer network is the second feature map, and the inputs to the remaining L-1 layers are the outputs of the previous Transformer network. The output of the last layer is the final channel context features for each grouped channel. The channel context features and prior information of each group channel are concatenated, and the mean and variance of the Gaussian distribution of each group are predicted by a parameter estimation network.

4. The image coding method for frequency domain feature-aware learning according to claim 3, characterized in that, Any Transformer network layer includes a grouped mask channel multi-head attention mechanism layer, a grouped convolutional layer, a GELU nonlinear mapping layer, and another grouped convolutional layer, wherein... The channel multi-head attention mechanism layer of the group masking masks the attention weights of undecoded group channels to decoded group channels. Through the channel attention mechanism, the decoded group channels can provide context information to other group channels. The GELU nonlinear mapping layer has an output close to a linear transformation when the input is close to zero, and an output close to a saturated nonlinear transformation when the input is far from zero. The number of groups in the grouped convolutional layer is equal to the number of groups in the second feature map of the group.

5. An image decoding method based on frequency domain feature-aware learning, used to decode the encoding obtained according to any one of claims 1-4, characterized in that, include: Obtain the compressed image binary bitstream, and decompose the compressed image binary bitstream to obtain the feature binary bitstream and the prior feature binary bitstream; Super-prior information is obtained from the super-prior feature binary code stream using a super-decoding neural network; The channel context information obtained by the Transformer-based channel autoregressive model is combined with the prior information to predict the mean and variance of the Gaussian model in each channel group for decoding the second feature map. Arithmetic decoding of the feature binary code stream is performed based on the mean, variance, and weight of the Gaussian model, and the second feature map is obtained by grouping channels one by one in the order of channel grouping. The second feature map is passed through a decoding neural network composed of frequency-determined Transformer blocks to obtain a reconstructed image.

6. An image compression method based on frequency domain feature-aware learning, characterized in that, include: Obtain the image to be compressed; A compressed image binary bitstream is obtained based on the image encoding method according to any one of claims 1-4; The image decoding method described in claim 5 is used to decompress the compressed image binary code stream to obtain a reconstructed image.

7. The image compression method based on frequency domain feature-aware learning according to claim 6, characterized in that, The parameters of encoding neural networks, decoding neural networks, super-encoding neural networks, super-decoding neural networks, and channel autoregressive models based on Transformer are obtained through parameter pass-through distortion optimization. ； Among them, distortion , It is the image to be compressed. It is to reconstruct the image. These are preset parameters used to balance distortion and bitrate; bitrate The following is calculated based on the Gaussian distribution of each channel group within each channel group of the second feature map: ； in, It is the first Features of channel grouping It is the first Predicted distribution in each channel group It is a set of learnable parameters.

8. An image compression system, characterized in that, include: Encoder: The encoder uses the image encoding method according to any one of claims 1-4 to encode the image to be compressed to obtain a compressed image binary bitstream; Decoder: The decoder decodes the compressed image binary code stream based on the image decoding method of claim 5 to obtain the reconstructed image.

9. An image compression apparatus, the image compression apparatus comprising a memory and a processor, the memory storing a program executable by the processor, characterized in that, The processor performs any of the following methods when executing the program: - The image encoding method according to any one of claims 1-4, or, - The image decoding method according to claim 5, or, - The image compression method according to any one of claims 6-7.

Images