A deep learning image compression method based on a frequency domain visual state model
By combining a frequency domain visual state model with a high-frequency reversible neural network, the limitations of existing image compression methods in processing global information and high-frequency details are overcome, achieving a more efficient image compression effect that is superior to traditional methods.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU DIANZI UNIV
- Filing Date
- 2026-03-02
- Publication Date
- 2026-06-26
AI Technical Summary
Existing image compression methods have limitations in capturing global information and high-frequency details. In particular, CNNs are limited in local feature processing, INNs are difficult to preserve fine details in reconstructing details and textures, and Transformers have high computational cost and resource consumption. Furthermore, different architectures neglect the processing of high-frequency details in the frequency domain.
A frequency domain visual state model is adopted, and image features are decomposed into low-frequency and high-frequency components by joint wavelet transform. The low-frequency and high-frequency components are processed by the frequency domain visual state space model and high-frequency reversible neural network respectively. The feature representation is optimized by spatial channel entropy model, and bit rate and distortion are optimized by combining end-to-end training and Lagrange multipliers.
It significantly improves the ability to capture low-frequency information, enhances the preservation of high-frequency details, reduces computational complexity and the number of parameters, and improves compression efficiency. It outperforms traditional methods, especially surpassing the VVC standard in terms of bit rate distortion.
Smart Images
Figure CN122289406A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image / video compression technology, and relates to a deep learning image compression method based on a frequency domain visual state model. Background Technology
[0002] High-performance image compression methods that balance quality and efficiency are crucial for addressing the ever-increasing demands for data storage and transmission. Among existing learning-based image compression (LIC) methods, convolutional neural networks (CNNs), invertible neural networks (INNs), and transformer architectures have all demonstrated strong performance in image compression tasks. CNNs excel at processing local image features, but their inherently limited receptive field presents challenges in capturing global information. INNs perform well in capturing rich high-frequency information, but their reconstructions often fail to preserve finer details and textures, especially for highly complex or noisy data. Transformer architectures excel at capturing long-term dependencies and global context, performing global dependency modeling, making them highly effective in high-resolution image processing. However, their high computational cost and resource consumption, especially in real-time applications or environments with limited hardware capabilities, become limiting factors. Recent research employs state-space models (SSMs) to reduce computational complexity and capture the global dependencies of images. However, these methods often neglect the frequency domain characteristics of different architectures. While they excel at modeling low-frequency features, they struggle with handling nuanced high-frequency information, which is crucial for achieving an effective trade-off in compression. These limitations underscore the need for novel models. Summary of the Invention
[0003] To address the aforementioned technical problems in existing technologies, this invention proposes a deep learning-based image compression method based on a frequency domain visual state model, the specific technical solution of which is as follows:
[0004] A deep learning-based image compression method based on a frequency domain visual state model includes the following steps:
[0005] S1. The input image is processed by the analysis and transformation module to extract multi-scale frequency domain features and convert them into a latent representation;
[0006] S2. The latent representation is quantized and entropy encoded using a frequency-domain-aware spatial channel entropy model to obtain the encoded bitstream and the recovered latent representation.
[0007] S3. Input the recovered latent representation into the synthesis transformation module to reconstruct the image and output the reconstructed image;
[0008] The analysis and transformation module includes a joint wavelet transform unit. In the joint wavelet transform unit, the frequency domain features of the input image are decomposed into low-frequency components and high-frequency components. The low-frequency components are processed by a frequency domain visual state space model to capture global low-frequency information of the image. The high-frequency components are processed by a high-frequency reversible neural network model to enhance and preserve the high-frequency details of the image. The features processed by the frequency domain visual state space model and the high-frequency reversible neural network model are fused to output an enhanced feature representation.
[0009] Furthermore, the analysis and transformation module transforms the features of the input image into approximation coefficients and wavelet detail coefficients in multiple directions through wavelet transform; the approximation coefficients are combined with the downsampling features of the input image to extract low-frequency features, thereby obtaining the low-frequency component; the wavelet detail coefficients in multiple directions are concatenated and high-frequency features are extracted through a convolutional layer to obtain the high-frequency component.
[0010] Furthermore, in the high-frequency reversible neural network model: the input high-frequency components are divided into multiple partial features according to channels, and weights are dynamically assigned according to the importance of channels. The weighted partial features interact in a chain manner through the conditional reversible neural network structure to gradually enhance the detail representation of high-frequency components, so as to selectively capture and enhance high-frequency components.
[0011] Furthermore, in the spatial channel entropy model: context information is calculated for the latent representation in both the spatial and channel dimensions to generate corresponding spatial context features and channel context features, and a super-prior encoder-decoder is introduced to obtain a super-prior feature representation. The spatial context features, channel context features, and super-prior feature representation are then concatenated and fused, and subsequently, a parameter aggregation network is used to predict the distribution parameters used for entropy encoding.
[0012] Furthermore, in the super-prior encoder-decoder: the latent representation is input into the super-prior encoder to generate additional information, the additional information is quantized, and the super-prior feature representation is reconstructed through the super-prior decoder.
[0013] Furthermore, in the spatial dimension, a symbol set is formed by latent representations from all preceding positions. The symbol set is divided into anchor groups and non-anchor groups. First, the anchor groups are encoded, and then the non-anchor groups are encoded based on the context obtained from the anchor groups. Subsequently, the symbol set is input into a spatial context model, which uses chessboard mask convolution to generate context information.
[0014] At the channel dimension, the channels of the latent representation are divided into multiple blocks, and the encoding of the current block is guided by the context of the previously decoded blocks. For each block, context information is computed using the channel context model.
[0015] Fourier domain attention mechanism is used in both spatial and channel dimensions to adaptively focus on frequency content.
[0016] Furthermore, the parameter aggregation network integrates a frequency domain visual state space model.
[0017] Furthermore, in the frequency domain visual state space model, a directional scanning strategy is adopted: 2D-Selective-Scan operation is used to extract important features in the spatial domain, and then frequency domain optimization processing is performed through a frequency selective feedforward network to highlight low-frequency components and suppress high-frequency components.
[0018] Furthermore, the frequency-selective feedforward network optimizes the feature map and processes spatial features in the frequency domain through convolution, channel separation, tensor reshaping, and fast Fourier transform.
[0019] Furthermore, the analysis transformation module, spatial channel entropy model, and synthesis transformation module are optimized through end-to-end training. The loss function used in training introduces Lagrange multipliers to balance bit rate and distortion, where bit rate includes the bit rate of the latent representation and the bit rate of the additional information, and distortion is a measure of the difference between the input image and the reconstructed image.
[0020] Beneficial effects:
[0021] The frequency domain visual state space (FVSS) model of this invention can effectively highlight the low-frequency components at the center of the spectrum while suppressing the high-frequency components, thus outperforming the traditional visual state space (VSS) method in capturing low-frequency features.
[0022] The high-frequency reversible neural network (HINN) model of this invention emphasizes the high-frequency components at the edge of the spectrum, significantly improving the ability to capture high-frequency details, which is superior to the traditional reversible neural network (INN) method.
[0023] This invention demonstrates superior performance in bit rate distortion (BD-Rate), surpassing the traditional compression standard VVC on the dataset with an improvement of 13.59%. Furthermore, compared to existing learned image compression (LIC) methods, this method reduces the number of parameters by 45.03%, FLOPs by 59.87%, and decoding time by 42.28% on the Kodak dataset. Attached Figure Description
[0024] Figure 1 This is a flowchart of a deep learning image compression method based on a frequency domain visual state model according to an embodiment of the present invention.
[0025] Figure 2 This is a schematic diagram of the structure of the joint wavelet transform unit according to an embodiment of the present invention;
[0026] Figure 3 This is a schematic diagram of the structural principle of the frequency domain visual state space model in an embodiment of the instinctive invention;
[0027] Figure 4 This is a schematic diagram of the structure of a frequency-selective feedforward network according to an embodiment of the invention.
[0028] Figure 5 This is a schematic diagram of the structure of a high-frequency reversible neural network model in an embodiment of the invention.
[0029] Figure 6 This is a schematic diagram of the reversible residual block in an embodiment of the invention.
[0030] Figure 7 This is a structural principle diagram of the spatial channel entropy model in an embodiment of the instinctive invention;
[0031] Figure 8 This is a schematic diagram of the structure of a priori codec, which is an embodiment of an instinctive invention. Detailed Implementation
[0032] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.
[0033] This embodiment discloses a deep learning image compression method based on a frequency domain visual state model. It utilizes three key modules—an analysis and transformation module, a synthesis and transformation module, and a frequency domain-aware spatial channel entropy model—to implement an end-to-end optimization process. Figure 1 As shown, the process includes:
[0034] S1: The input image x is first processed by the analysis and transformation module g. a It is transformed into a latent representation y to capture multi-scale frequency domain features.
[0035] S2: Next, the latent representation y is quantized and entropy encoded using a frequency-domain-aware spatial channel entropy model. Specifically, the latent representation y is processed by subtracting the mean μ, and y-μ is rounded to the nearest integer and encoded as a bitstream. After encoding, the mean μ is added back to restore the representation. This ensures alignment with the original potential representation.
[0036] S3: Finally, the restored latent representation Transmitted to the synthesis and transformation module g s Perform image reconstruction and output the reconstructed image. .
[0037] The above processes S1~S3 can be represented as:
[0038] ,
[0039] The quantization process Q is achieved by adding uniform noise (AUN) or using a pass-through estimator (STE).
[0040] like Figure 2 As shown, in the analysis and transformation module g of S1 a An internal Joint Wavelet Transform (JWT) unit is designed, featuring a bi-branch structure with low-frequency and high-frequency branches to decompose the frequency domain features of an image into low-frequency and high-frequency components. Specifically, the transform module g is analyzed. a Wavelet transform is used to convert the features of the input image into frequency components. This transform generates approximation coefficients A and horizontal, vertical, and diagonal wavelet detail coefficients (H, V, D), which represent the frequency information of the image in different directions, respectively.
[0041] ,
[0042] Subsequently, approximation coefficients Downsampling features of the input image Combined, processed through a convolutional layer, and then passed through a residual block. Extracting low-frequency features yields low-frequency components. Meanwhile, the wavelet detail coefficients (H, V, D) are concatenated and high-frequency features are extracted through a 1×1 convolutional layer to obtain the high-frequency components. :
[0043] ,
[0044] ,
[0045] ,
[0046] in, Indicates the splicing of channels. This indicates average pooling.
[0047] Next, while optimizing the high-frequency components, multi-scale feature fusion techniques are used to enhance the expressive power of the low-frequency components. Specifically, the low-frequency components are input into a frequency domain visual state space (FVSS) model, and the high-frequency components are input into a high-frequency reversible neural network (HINN) model for further processing. Then, the features from the FVSS model and the HINN model are fused to enhance the representational power. The fusion process can be described as follows:
[0048] .
[0049] In image compression tasks, capturing low-frequency information is crucial for effectively modeling the global content of an image. However, traditional networks often focus on local features, making it difficult to capture long-range dependencies and low-frequency information. Due to their inherent architecture, state-space models (SSMs) concentrate more on capturing low-frequency information while neglecting high-frequency components. The Frequency Domain Visual State Space (FVSS) model described in this embodiment is specifically designed to handle information in the low-frequency branches, such as... Figure 3 As shown. The FVSS model employs a directional scanning strategy: a 2D-Selective-Scan (SS2D) operation extracts important features by traversing the spatial domain and unfolding the input image into multiple patches. These patches are organized along four different scanning directions, enabling the FVSS model to effectively capture low-frequency patterns by emphasizing global coarse features of the image. The output is then processed by a Frequency Selective Feedforward Network (FSFFN), as shown. Figure 4 As shown, FSFFN performs convolution and Fast Fourier Transform (FFT) to further optimize the feature map and process spatial features in the frequency domain. Given low-frequency features of the input... The processing procedure of the FVSS model can be represented as follows:
[0050] ,
[0051] ;
[0052] in, This refers to layer normalization. and These are the learned weights used to scale the input feature map. Given an input feature map... FSFFN can be represented as:
[0053] ,
[0054] ,
[0055] ;
[0056] in, This refers to separation by channel. , It is a feature map obtained after separating by channel. This indicates a depthwise separable convolution using a 3×3 kernel; Represents element-wise multiplication; Reshaping a tensor refers to patch folding and unfolding; FFT and IFFT refer to Fast Fourier Transform and Inverse Fast Fourier Transform, respectively. Frequency domain processing is achieved by modulating the frequency components of the tensor after the FFT operation.
[0057] The FVSS model effectively highlights the low-frequency components at the center of the spectrum while suppressing the high-frequency components, thus outperforming the traditional visual state space (VSS) method in capturing low-frequency features.
[0058] While traditional reversible neural networks (INNs) are highly effective in ensuring reversibility and lossless reconstruction, they often distort or blur high-frequency details. This results in compressed images that are less sharp and more prone to artifacts, negatively impacting visual quality. To address this issue, this invention designs the aforementioned High-Frequency Reversible Neural Network (HINN) model, incorporating a channel dynamic importance weighting mechanism to progressively enhance the detail representation of high-frequency components. This aims to selectively capture and preserve high-frequency elements, such as... Figure 5 As shown. To enhance this capability, weights are dynamically assigned to channels based on their importance. Given the high-frequency features of the input. The process can be represented as follows:
[0059] ,
[0060] ,
[0061] ,
[0062] ,
[0063] ;
[0064] Among them, high-frequency features have The channel is divided into three parts: , and The number of channels in each part is In order to achieve a balanced frequency distribution, Set as . It is a weight vector that is applied to each tensor part after the segmentation. express The normalized version. It is a small value used to avoid division by zero errors. This represents channels 1 to c. The tensor after each segmentation is multiplied element-wise by a normalized weight vector. Multiplication is weighted. By dynamically adjusting the importance of different channels, this weighting mechanism allows the model to focus on the most relevant information and discard unimportant features, thereby significantly reducing computational overhead. To ensure adaptability to different input data, this invention utilizes a Conditional Invertible Neural Network (CINN) structure to interact with the weighted channels in a chain-like manner through successive bottleneck operations. These interactions are controlled by learned weights, and the entire process can be represented as follows:
[0065] ,
[0066] ,
[0067] ,
[0068] ,
[0069] ;
[0070] Where exp represents exponential scaling. This represents the sigmoid activation function. For example... Figure 6 As shown, This refers to a reversible residual block that processes the input and applies a learnable transformation. The final output is... , , By stitching together the processed feature maps, the CINN structure highlights key high-frequency components, thus providing a more compact and efficient feature representation for image compression.
[0071] The HINN model emphasizes high-frequency components at the edges of the spectrum, significantly improving the ability to capture high-frequency details, which is superior to the traditional invertible neural network (INN) method.
[0072] To remove the correlation frequency redundancy between different channels and spatial content, a spatial channel entropy model was designed in S2 to model the conditional distribution of the latent representation, such as... Figure 7 As shown. In the spatial dimension, a symbol set is formed by the latent representations from all preceding positions 1 to i-1. The symbol set is divided into anchor point groups. Non-anchor group First, the anchor group is encoded, then the non-anchor group is encoded based on the context obtained from the anchor group. Subsequently, the symbol set... Input into the spatial context model The model uses checkerboard mask convolution to generate contextual information. This process can be represented as:
[0073] ;
[0074] ;
[0075] Furthermore, a parallel back-adaptation strategy is introduced to reduce redundancy between channels. The channels of the latent representation are divided into K blocks to construct a channel-level context. Each block is encoded sequentially, using the context of the previously decoded blocks. This guides the encoding of the current block. For each block K, contextual information is computed using the channel context model. Spatial context and channel context are concatenated with the hyperprior context to predict Gaussian distribution parameters:
[0076] ;
[0077] Because initial blocks are referenced more frequently by subsequent blocks, they tend to carry more critical information. This relates to the latent representation with M channels. Divided into 5 blocks, with the number of channels allocated as follows: Unlike treating all frequency components equally, this invention employs a Fourier domain attention mechanism (FFTAtten) to adaptively focus on frequency content, thereby more effectively reducing frequency redundancy.
[0078] Spatial branches and channel branches in position The outputs on are respectively labeled as and , with prior representation The data is then fused. This combination is then fed into a location-specific parameter aggregation network to compute the entropy parameter. The parameter aggregation network incorporates the FVSS model to optimize the predicted mean and variance values in a more accurate and compact form.
[0079] ;
[0080] These parameters are used to predict values in subsequent encoding and decoding steps. New predicted value Used as contextual information, so that the next spatial feature can be calculated. or channel characteristics This continues until the entire symbol set has been encoded and decoded.
[0081] In summary, this embodiment combines the FVSS model with the Fourier domain attention mechanism and integrates it into the spatial channel entropy model, effectively reducing frequency redundancy in the latent representation, thereby achieving more accurate image reconstruction and effectively reducing the bit rate.
[0082] like Figure 8 As shown, in order to more accurately model the latent representation, y is input into a super-prior encoder h. aAdditional information z is generated. Then, the additional information z undergoes quantization to obtain quantized additional information. Quantified additional information Through the hyper-prior decoder h s Processing for reconstructing prior knowledge The process is represented as:
[0083] .
[0084] Latent variables Modeled as following a conditional Gaussian distribution , expressed as mean μ and standard deviation . The Gaussian distribution is predicted by the spatial channel context entropy model based on the frequency visual state model.
[0085] Lagrange multipliers were introduced during the training of the image compression model. To balance the trade-off between bit rate and distortion, the loss function aims to optimize bit rate and distortion in an end-to-end manner, expressed as:
[0086] ;
[0087] Among them, Lagrange multipliers Balance between bit rate and distortion. Bit rate Representing potential representations Number of bits, additional information bit rate Derived from a priori model. Distortion. The difference between the input image and the reconstructed image is typically measured using mean squared error (MSE).
[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the implementation process of the present invention has been described in detail above, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A deep learning image compression method based on a frequency domain visual state model, characterized in that, Includes the following steps: S1. The input image is processed by the analysis and transformation module to extract multi-scale frequency domain features and convert them into a latent representation; S2. The latent representation is quantized and entropy encoded using a frequency-domain-aware spatial channel entropy model to obtain the encoded bitstream and the recovered latent representation. S3. Input the recovered latent representation into the synthesis transformation module to reconstruct the image and output the reconstructed image; The analysis and transformation module includes a joint wavelet transform unit. In the joint wavelet transform unit, the frequency domain features of the input image are decomposed into low-frequency components and high-frequency components. The low-frequency components are processed by a frequency domain visual state space model to capture global low-frequency information of the image. The high-frequency components are processed by a high-frequency reversible neural network model to enhance and preserve the high-frequency details of the image. The features processed by the frequency domain visual state space model and the high-frequency reversible neural network model are fused to output an enhanced feature representation.
2. The method as described in claim 1, characterized in that, The analysis and transformation module transforms the features of the input image into approximation coefficients and wavelet detail coefficients in multiple directions through wavelet transform. The approximation coefficient is combined with the downsampling features of the input image and low-frequency features are extracted to obtain the low-frequency component; The wavelet detail coefficients from multiple directions are concatenated and high-frequency features are extracted through a convolutional layer to obtain the high-frequency components.
3. The method as described in claim 1, characterized in that, In the high-frequency reversible neural network model, the input high-frequency components are divided into multiple feature parts according to channels, and weights are dynamically assigned according to the importance of channels. The weighted feature parts interact in a chain manner through the conditional reversible neural network structure to gradually enhance the detail representation of high-frequency components, so as to selectively capture and enhance high-frequency components.
4. The method as described in claim 1, characterized in that, In the spatial channel entropy model: context information is calculated for the latent representation in both the spatial and channel dimensions to generate corresponding spatial context features and channel context features. A super-prior encoder-decoder is introduced to obtain a super-prior feature representation. The spatial context features, channel context features, and super-prior feature representation are concatenated and fused. Then, the distribution parameters used for entropy encoding are predicted through a parameter aggregation network.
5. The method as described in claim 4, characterized in that, In the super-prior encoder-decoder: the latent representation is input into the super-prior encoder to generate additional information, the additional information is quantized, and the super-prior feature representation is reconstructed through the super-prior decoder.
6. The method as described in claim 4, characterized in that, In the spatial dimension, a symbol set is formed by latent representations from all preceding positions. The symbol set is divided into anchor groups and non-anchor groups. First, the anchor groups are encoded, and then the non-anchor groups are encoded based on the context obtained from the anchor groups. Subsequently, the symbol set is input into a spatial context model, which uses chessboard mask convolution to generate context information. At the channel dimension, the channels of the latent representation are divided into multiple blocks, and the encoding of the current block is guided by the context of the previously decoded blocks. For each block, context information is computed using the channel context model. Fourier domain attention mechanism is used in both spatial and channel dimensions to adaptively focus on frequency content.
7. The method as described in claim 4, characterized in that, The parameter aggregation network integrates a frequency domain visual state space model.
8. The method as described in claim 7, characterized in that, In the frequency domain visual state space model, a directional scanning strategy is adopted: 2D-Selective-Scan operation is used to extract important features in the spatial domain, and then frequency domain optimization processing is performed through a frequency selective feedforward network to highlight low-frequency components and suppress high-frequency components.
9. The method as described in claim 8, characterized in that, The frequency-selective feedforward network optimizes feature maps and processes spatial features in the frequency domain through convolution, channel separation, tensor reshaping, and fast Fourier transform.
10. The method as described in claim 4, characterized in that, The analysis transformation module, spatial channel entropy model, and synthesis transformation module are optimized through end-to-end training. The loss function used in training introduces Lagrange multipliers to balance bit rate and distortion, where bit rate includes the bit rate of the latent representation and the bit rate of the additional information, and distortion is a measure of the difference between the input image and the reconstructed image.