Cross-granularity online image compression method and system based on mask compensation

Abstract

The invention discloses a cross-granularity online image compression method and system based on mask compensation, and belongs to the technical field of digital image compression. In order to solve the problem that the existing image compression model is easy to generate a spreading effect when processing input image distribution and cannot fully model local structure features, a data dependence transformation module is mainly adopted to carry out coarse-grained global optimization, and a mask selectivity compensation module is mainly adopted to carry out fine-grained local optimization. And cross-granularity collaborative modeling is realized through sparse representation, codebook selection and progressive joint training. According to the method, the expression capability of an end-to-end image compression model on global and local features of the image can be improved, and high-precision image reconstruction under a low code rate is realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of digital image compression, and relates to a cross-granularity online image compression method and system based on mask compensation. BACKGROUND

[0002] Image compression aims to reduce the storage space and transmission bandwidth required for images while maintaining the content and quality of the images. Image compression has been a focus of both industry and academia.

[0003] Traditional image compression frameworks utilize key steps such as transformation, quantization, and entropy coding to estimate the real distribution and remove redundant information for efficient transmission. With the rapid development of deep learning, many neural network-based image compression methods have been proposed. Early deep learning-based image compression methods achieve variable-rate compression by embedding generalized divisive normalization (GDN) transformation networks or developing recurrent architectures. Later, more advanced module architectures such as attention mechanisms and Transformer-based architectures further enhance compression performance. In addition, joint autoregressive and hyperprior models and their related variants are developed to improve entropy coding efficiency. Unlike traditional methods, deep learning-based image compression methods use end-to-end learning networks to replace manual optimization, achieving better rate-distortion optimization performance.

[0004] However, whether based on manually designed traditional methods or end-to-end learning methods optimized for rate-distortion cost, there is a common challenge in handling input image distribution: models tend to fit average attributes rather than individual image characteristics, a phenomenon known as the dilution effect. Specifically, this distribution gap manifests in two aspects: global gap caused by image content and theme differences; and local structure and texture information gap caused by quantization processes and sampling randomness.

[0005] To address this challenge, traditional coding techniques use auxiliary information to achieve sample adaptive modeling, such as using adaptive loop filters (ALF) to compensate for global distribution bias and using multiple transform selection (MTS) to handle local structure distribution differences, thereby reducing the rate-distortion cost. For learning-based image compression, a sample-by-sample optimization method based on online gradient descent is proposed, which adjusts the image-level parameters stored at the decoder end by auxiliary code streams to reduce the global distribution gap. However, in existing online gradient descent optimization methods, due to fixed and overly strict code rate constraints, the auxiliary code stream often fails to learn enough information at the beginning of optimization, resulting in a tendency to abandon learning related information due to code rate limitations, which cannot be further optimized in the later stage, resulting in information collapse and poor information transmission as expected. In addition, such methods generally ignore the optimization of fine-grained local structure distribution, making it difficult to achieve deep integration and enhancement of fine signal representation. SUMMARY

[0006] The application aims to provide a cross-granularity online image compression method and system based on mask compensation, which is used to solve the problems of information collapse and insufficient local structure compensation in global and local image distribution optimization of the existing method, and realize efficient compression and fine reconstruction of images.

[0007] To achieve the above-mentioned purpose, the technical scheme adopted by the application is as follows:

[0008] A cross-granularity online image compression method based on mask compensation, comprising the following steps:

[0009] 1) Constructing an image compression model, setting a mask-based selective compensation module MSC and a data-dependent transformation module DDT in an end-to-end image compression network;

[0010] 2) Encoding the input image, generating a latent representation through an end-to-end encoder, and obtaining a quantized latent representation through a quantizer;

[0011] 3) Reconstructing the quantized latent representation using a hyper-prior model and a probability model to obtain a reconstructed quantized latent representation;

[0012] 4) Calculating residual features based on the difference before and after quantization, inputting the residual features into the MSC module, generating compensation features through the MSC encoder, obtaining sparse compensation features through codebook and sparse mask processing, and decoding the sparse compensation features using the MSC decoder to obtain distribution information compensation features;

[0013] 5) Fusing the reconstructed quantized latent representation and the distribution information compensation features to obtain a fused latent representation;

[0014] 6) Inputting the input image into the DDT module, generating adaptive convolution kernel parameters through the DDT encoder and the DDT decoder;

[0015] 7) Decoding the fused latent representation under the condition of adaptive convolution kernel parameters using an end-to-end decoder to obtain a reconstructed image;

[0016] 8) Inputting training data into the image compression model, training the DDT module and the MSC module, calculating a loss function, and optimizing model parameters according to the loss function;

[0017] 9) Compressing the image to be compressed using the trained image compression model to generate a compressed reconstructed image.

[0018] Further, the step of reconstructing in step 3) comprises:

[0019] quantizing the quantized latent representation using a hyper-prior model to obtain a quantized hyper-prior representation;

[0020] The quantized hyper-prior representation is input into the probability model to obtain an estimated mean and variance of a Gaussian distribution of the quantized latent representation;

[0021] Based on the Gaussian distribution, the quantized latent representation is generated into a code stream through an arithmetic encoder and is decoded into a reconstructed quantized latent representation through an arithmetic decoder.

[0022] Further, the MSC module in step 4) includes an MSC encoder, an MSC decoder, a vector quantization encoder, a vector quantization decoder, and a codebook, wherein the MSC encoder and the MSC decoder are both constructed based on a convolutional network; the processing steps of the MSC module include:

[0023] Pre-selection operation: the MSC encoder is used to obtain a compensation feature of the residual feature; each pixel of the compensation feature is used to find a corresponding quantization vector in the codebook through the vector quantization encoder to obtain a quantized compensation feature and an index mapping;

[0024] Sparse mask operation: a sparse representation mask is generated through fine-grained optimization, the index mapping is operated through the sparse mask to calculate a sparse compensation representation; and the sparse compensation feature is generated by searching the codebook through the vector quantization decoder;

[0025] Decoding operation: the MSC decoder is used to decode the sparse compensation feature to obtain a distribution information compensation feature.

[0026] Further, the DDT module in step 6) includes a DDT encoder, a DDT decoder, a quantizer, an arithmetic encoder, and an arithmetic decoder, wherein the DDT encoder and the DDT decoder are constructed based on a multi-layer perceptual mechanism with residual connection; the processing steps of the DDT module include:

[0027] The input image is encoded by the DDT encoder, multi-scale features are extracted through multi-layer convolution, and are spliced into a compact one-dimensional vector after global average pooling;

[0028] The one-dimensional vector is quantized through the quantizer with an adjustable quantization parameter;

[0029] After quantization, the arithmetic encoder is used to perform arithmetic encoding based on a standard Gaussian distribution to generate a syntax vector;

[0030] The syntax vector is decoded by the arithmetic decoder based on the standard Gaussian distribution to obtain a decompressed syntax vector;

[0031] The decompressed syntax vector is decoded by the DDT decoder to generate adaptive convolution kernel parameters.

[0032] Further, the image compression model is trained by using a progressive multi-stage training strategy in step 8), and the steps include:

[0033] First, the baseline model of the end-to-end image compression network is pre-trained, and the pre-trained parameters are loaded.

[0034] Then, the DDT module is trained for coarse-grained optimization;

[0035] Next, the MSC module is trained and fine-grained optimization is performed.

[0036] Furthermore, the coarse-grained optimization steps for training the DDT module in step 8) include:

[0037] Relax the bitrate constraint of the syntax vector during gradient descent training;

[0038] Minimize the difference between the image-level estimated distribution and the true distribution of a single sample;

[0039] Calculate the loss function The DDT module parameters are then optimized based on this loss function.

[0040] Furthermore, the fine-grained optimization steps for training the MSC module in step 8) include:

[0041] A sparse representation strategy is used to compensate for local structural features;

[0042] Sparse compensated representations are obtained through pre-selection and sparse masking optimization operations;

[0043] Distribution information compensation features are generated using the MSC decoder;

[0044] Calculate the loss function The parameters of the MSC module are then optimized based on this loss function.

[0045] Furthermore, the loss function The calculation formula is as follows:

[0046]

[0047] Where x represents the input image, The image is represented by y, where y represents the bitstream corresponding to the quantized latent representation, and -log Q. Y (y) represents the negative log-likelihood of the bitstream y under a fixed distribution Y, λ represents the loss tradeoff parameter, Δ(,) represents the measure of image degradation, and F E Indicates DDT encoder, F D Indicates DDT decoder, φ D represents the parameters of the DDT decoder, and s represents the syntax vector.

[0048] Furthermore, the loss function The calculation formula is as follows:

[0049]

[0050] Where x represents the input image, This represents the image reconstructed using the MSC module, y res E represents the residual characteristics before and after quantization. MSC Indicates MSC encoder, z q This represents the quantization vector selected from the codebook, and sg[·] indicates stopping the gradient operation. This represents the square of the Euclidean norm.

[0051] A cross-granularity online image compression system based on mask compensation includes:

[0052] An end-to-end compression network includes an end-to-end encoder and decoder, a quantizer, a super-prior model, a probabilistic model, an arithmetic encoder, and a decoder. The end-to-end encoder encodes the input image into a latent representation; the quantizer quantizes the latent representation into a quantized latent representation; the super-prior model quantizes the quantized latent representation to generate a quantized super-prior representation; the probabilistic model processes the quantized super-prior representation to obtain an estimated mean and variance of the Gaussian distribution of the quantized latent representation; the arithmetic encoder encodes the quantized latent representation into a bitstream based on the Gaussian distribution; the arithmetic decoder decodes the bitstream to generate a reconstructed quantized latent representation; and the end-to-end decoder decodes the fused latent representation under adaptive convolutional kernel parameters to generate a reconstructed image.

[0053] The MSC module includes an MSC encoder, an MSC decoder, a vector quantization encoder, a vector quantization decoder, and a codebook. The MSC encoder extracts compensation features from the residual features. The vector quantization encoder matches the quantization vector in the codebook for each pixel of the compensation feature, generating quantized compensation features and an index mapping. The vector quantization decoder applies a sparse representation mask to the index mapping, generating a sparse compensation representation and generating sparse compensation features based on the codebook. The MSC decoder decodes the sparse compensation features to obtain distribution information compensation features.

[0054] The DDT model includes a DDT encoder, a DDT decoder, a quantizer, an arithmetic encoder, and an arithmetic decoder. The DDT encoder encodes the input image, extracting multi-scale features through multiple convolutional layers and generating a compact one-dimensional vector via global average pooling. The quantizer performs adjustable parameter quantization on the one-dimensional vector. The arithmetic encoder performs arithmetic encoding on the quantized vector based on a standard Gaussian distribution to generate a syntax vector. The arithmetic decoder performs arithmetic decoding on the syntax vector to obtain a decompressed syntax vector. The DDT decoder decodes the decompressed syntax vector to generate adaptive convolutional kernel parameters.

[0055] The beneficial effects achieved by this invention are as follows:

[0056] 1. This invention introduces a DDT module for coarse-grained online optimization, enabling adaptive modeling of global image features and improving the end-to-end compression model's ability to match the distribution of the input image.

[0057] 2. This invention uses the MSC module for fine-grained optimization and employs sparse representation and mask selection strategies to effectively compensate for local structural features and texture information, thereby achieving fine signal reconstruction at low bit rates.

[0058] 3. This invention adopts a cross-granularity joint optimization strategy. By first optimizing the DDT module with coarse granularity and then optimizing the MSC module with fine granularity, the invention achieves collaborative modeling of global and local features, thereby improving the adaptability and reconstruction accuracy of the model.

[0059] 4. This invention enhances the model's ability to represent residual information and diverse structural distributions by combining end-to-end training with codebook selection, sparsity compensation, and masking operations, while also improving the activation rate of codebook vectors, thus achieving more efficient information transmission.

[0060] 5. This invention significantly improves image compression performance through a cross-granularity joint optimization strategy while maintaining network lightweightness. It outperforms existing deep learning methods and traditional coding standards on both standard datasets and professional benchmark sets, achieving a superior rate-distortion balance. Using a coarse-grained online optimization method, on the Kodak professional benchmark dataset, it improves the peak signal-to-noise ratio (PSNR) BD-rate by 1.447% compared to the baseline method; using a fine-grained online optimization method, the improved performance is further enhanced to 2.524%. Attached Figure Description

[0061] Figure 1 This is a flowchart of a cross-granularity online image compression method based on mask compensation, according to an embodiment of the present invention.

[0062] Figure 2 This is a schematic diagram illustrating the optimization strategies for coarse-grained and fine-grained optimization in an embodiment of the present invention. Detailed Implementation

[0063] To make the above-mentioned features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings. It should be noted that the specific number of layers, modules, functions, and layer settings given in the following embodiments are merely preferred implementations and are not intended to limit the scope. Those skilled in the art can select the appropriate number and set different layers according to actual needs, which should be understood.

[0064] This embodiment discloses a cross-granularity online image compression method based on mask compensation. First, high-quality natural image data is collected as a training dataset for training and testing the compression network. Then, an end-to-end compression network framework is constructed based on the baseline model. This framework introduces a mask-based selective compensation module (MSC module) and a data-dependent transformation module (DDT module), and cross-granularity online image compression is achieved based on this framework.

[0065] This method can be divided into a training phase and an inference phase. In the training phase, the model is trained using a training dataset, and optimization strategies are employed to refine the model parameters. In the inference phase, the trained model is used to process the image to be compressed, outputting a compressed bitstream and performing image reconstruction.

[0066] I. Method Processing Flow

[0067] like Figure 1 As shown, the specific processing procedure of this method is as follows:

[0068] For the input image x, it is processed through the baseline branch, MSC branch, and DDT branch respectively. The transmitted bitstream includes three types: quantized latent representation. Index mapping id t and syntax vectors s.

[0069] (I) Baseline branching:

[0070] Baseline branch processing is handled by the baseline model, which consists of an end-to-end encoder E, a quantizer Q, a super-prior model, a probabilistic model, an arithmetic encoder (AE), an arithmetic decoder (AD), and an end-to-end encoder.

[0071] 1. Encode the input image x using an end-to-end encoder E to generate a latent representation y = E(x);

[0072] 2. The latent representation y is quantized by the quantizer Q to obtain the quantized latent representation.

[0073] 3. Quantify the latent representation The input hyperprior model is quantized to obtain the quantized hyperprior representation.

[0074] 4. Quantify the prior representation The input probability model is processed to obtain the quantized latent representation. The estimated mean μ and variance σ of the Gaussian distribution;

[0075] 5. Based on the Gaussian distribution described above, the latent representation will be quantized. The bitstream is obtained through arithmetic encoding, and then the original quantized latent representation is reconstructed through corresponding arithmetic decoding. For distinction, the reconstructed quantized latent representation is denoted as .

[0076] 5. Through the end-to-end decoder D, adaptive convolutional kernel parameters are used. Fusion features Decode the image and reconstruct it. Among them, fusion features From the reconstructed quantized latent representation and distribution information compensation feature y t Composition, i.e., fusion characteristics And the distribution information compensation feature y t Obtained by the following MSC branch processing; adaptive convolution kernel parameters Obtained by processing the following DDT branch.

[0077] (II) MSC branching:

[0078] MSC branching is handled by the MSC module, which includes MSC encoders E, all based on convolution. MSC and MSC decoder D MSC Vector quantization encoder (VQ encoder) E VQ and Vector Quantization Decoder (VQ-decoder) VQ and codebooks for storing diverse structural distribution characteristics.

[0079] 1. Pre-selection operation: This involves selecting residual characteristics before and after quantization. The input is processed by the MSC module, first through the MSC encoder E. MSC Obtain the compensation feature y c ∈R h×w×n Then compensate for the feature y c Each pixel of the feature map is quantized by the vector encoder E VQ Find the corresponding quantization vector in the codebook Z to obtain the quantization compensation feature. and index mapping id c This process can be represented as:

[0080]

[0081] Where, λ c To determine the trade-off parameters through online optimization, In codebook Z, it means that The smallest z value k , R(id) k ) for id k The required bitstream is compressed using Huffman coding. For y c The value at each pixel position (i,j).

[0082] 2. Sparse Masking Operation: Generate a sparse representation mask M through fine-grained optimization, and then apply this to the ID. c Perform sparse masking operations and calculate the sparse compensation representation id. t Selectively transmitting the most valuable information to achieve greater performance gains. This process can be represented as:

[0083] id t =id c ⊙M

[0084] Here, ⊙ represents element-wise multiplication.

[0085] Considering id t Due to its sparsity, a global probability distribution Q is used during transmission. t Encoding is performed. The final vector quantization decoder D... VQ By ID t Searching for codebook Z to generate sparse compensated features The calculation process and id c The calculation process is the complete reverse process, the only difference being that it starts with an ID that has been masked. t rather than the ID before the mask was overwritten. c .

[0086] 3. Decoding operation: Sparse compensation features are processed through the MSC decoder. Decoding is performed to obtain distribution information compensation features.

[0087] (III) DDT branch processing:

[0088] DDT branching is handled by the DDT module. This DDT module includes DDT encoders E, each with a multilayer perceptron featuring residual connections. DDT and DDT decoder D DDT It also includes the quantizer Q, the arithmetic encoder (AE), and the arithmetic decoder (AD).

[0089] 1. Input image x Input DDT encoder E DDT In the processing, the input image x is processed through multiple convolutions to extract multi-scale features E. DDT(x) is then concatenated into a compact one-dimensional vector s0 after global average pooling. This method allows the extracted features to possess both multi-scale information and global consistency, making it more suitable for fitting image-level distributions.

[0090] 2. Then, through the adjustable quantization parameter λ t The quantizer Q(·,·) quantizes the one-dimensional vector s0;

[0091] 3. Arithmetic coding compression based on a standard Gaussian distribution (μ = 0, σ = 1) is performed using AE to obtain the syntax vector s = Q(E). DDT (x),λ t Then, the syntax vector s is arithmetically decoded and decompressed based on a standard Gaussian distribution (μ=0, σ=1) using AD.

[0092] 4. The decompressed syntax vector s is processed by the DDT decoder D. DDT Generate adaptive convolution kernel parameters These parameters will serve as the weights for the final convolutional layers of the end-to-end decoder D.

[0093] II. Optimization strategies, such as Figure 2 As shown

[0094] In terms of optimization strategy, this method adopts an online optimization strategy that combines coarse-grained optimization (global alignment) and fine-grained optimization (local correction) during the inference stage, aiming to minimize the gap between the predicted distribution and the true distribution at the two signal distribution levels: image level and local structure level.

[0095] (I) Coarse-grained optimization

[0096] At the coarse-grained level, this method improves upon existing gradient descent methods by introducing relaxed bitrate constraints and an adaptive quantization parameter (QP) mechanism to avoid information collapse and optimize the image-level distribution. At this level, the optimization objective is to minimize the difference between the general distribution estimated at the image level and the true distribution of a single sample. Similar to most existing methods, this method employs a gradient descent strategy for coarse-grained online optimization of network parameters, thereby achieving parameter optimization at the image level.

[0097] To further prevent information collapse, this method splits the original gradient descent strategy into two stages: First, during the gradient optimization process of the Data-Dependent Transformation (DDT) encoder, the bitrate constraint on the syntax vector s is relaxed, and its optimization objective can be expressed as:

[0098] s = arg min s Δ(x,F D(y|φ D D DDT (s)))

[0099] Where, arg min s Let s represent the value that minimizes the objective function; Δ(a,b) represents the metric for image degradation, used to measure the difference between image a and image b; F D Represents the decoder of the network; φ D Indicates decoder F D The parameters of D; DDT (s) represents the output of the syntax vector s after processing by the DDT decoder; F D (y|φ D D DDT (s) represents the condition given parameter φ D and D DDT Under the condition of (s), the bitstream y is input into the decoder to obtain the output image.

[0100] Subsequently, this method employs a binary search algorithm to iteratively adjust the quantization parameter λ starting from a larger initial value. t This achieves an optimal trade-off between the bitrate overhead of the syntax vectors and the image reconstruction performance.

[0101] (II) Fine-grained optimization

[0102] At a fine-grained level, this method proposes a sparse representation strategy and designs a mask-based selective compensation module. By encoding diverse structural distribution features, sparse signal reconstruction under low bitrate conditions is achieved, thereby effectively compensating for the local structural information of the image. In this optimization process, the focus is on the local structural details of the image. Due to the randomness of the quantization process and image sampling itself, these local detail distributions typically exhibit discrete characteristics, making traditional gradient descent methods unable to achieve further fitting based on offline training data. To achieve online optimization of the local structure, this method employs a sparse representation strategy for compensation, the specific process of which can be divided into the following two steps.

[0103] Step 1: Pre-selection trade-off optimization

[0104] In the sparsity compensation process of the MSC module, a balance needs to be struck between the transmission cost of the compensated features and the performance gains they bring. During the pre-selection phase, features... This represents the candidate vector that yields the maximum performance gain without considering transmission costs. For a given feature... Matching vector z in the codebook k The closer the image is to its counterpart, the greater the potential benefit. This benefit can be approximated by the L1 norm of the two. However, the L1 norm is only an approximate measure of performance gain and not a reliable indicator of image degradation. There is no strict positive correlation. Therefore, when making rate-gain tradeoffs, it is necessary to find a local tradeoff parameter λ that is aligned with the target parameter λ. c λ is the hyperparameter input during the training phase. Using a binary search strategy, the λ that minimizes the global rate-distortion (RD) loss can be determined. c .

[0105] Step 2: Sparse Mask Optimization

[0106] During the pre-selection process, although a corresponding quantization vector can be found for each feature location, not all vectors produce a sufficiently positive effect on local structure compensation. To achieve optimal transmission cost-effectiveness, this method designs a sparse masking optimization strategy to mask inefficient vectors. This optimization targets quantization compensation features. Each pixel in the feature map The process proceeds independently: First, compensation features are injected into the corresponding positions (i,j) using element-wise addition; then, the end-to-end decoder D generates a reconstructed image based on the modified features. The difference in loss before and after compensation is calculated. Based on this performance difference, a binary mask decision is made: if the position is valid (mask value is 1), the corresponding feature transmission is activated; if the position is invalid (mask value is 0), the feature is pruned to optimize the overall bit rate overhead.

[0107] III. Model Training and Loss Calculation

[0108] To enhance the matching between end-to-end neural network parameters and actual data distribution, and to improve the scalability of cross-granularity online optimization strategies, this method proposes a progressive multi-stage training strategy. This strategy includes the following steps: first, loading the pre-trained model parameters of the baseline end-to-end network; then, training the DDT module based on these parameters; and finally, training the MSC module.

[0109] (I) DDT Module Training

[0110] In the training of the DDT module, similar to coarse-grained optimization, this method relaxes the bitrate constraint on the syntax vector s during gradient descent-based training. Its loss function is defined as:

[0111]

[0112] Where x represents the input image, Represents the reconstructed image; y represents the bitstream corresponding to the quantized latent representation; -log Q Y(y) represents the negative log-likelihood of the bitstream y under a fixed distribution Y, used to approximate the bitrate distribution; λ represents the loss trade-off parameter, used to balance the bitrate and distortion terms; Δ(,) represents the measure of image degradation; F E Indicates DDT encoder, F D Indicates DDT decoder; φ D represents the parameters of the DDT decoder; s represents the syntax vector.

[0113] (II) MSC Module Training

[0114] The training process of the MSC module is divided into two stages: MSC encoder / decoder training and codebook joint training.

[0115] 1. MSC encoder / decoder training

[0116] First, train the MSC encoder and decoder. During training, remove the vector quantization encoder E from the network. VQ and vector quantization decoder D VQ To eliminate the influence of y c Vector quantization operation. With the rest of the network fixed, only the MSC encoder and decoder are trained independently, using the mean squared error (MSE) between the reconstructed image and the input image as the loss function. After training, the MSC encoder / decoder can extract the values ​​from the residual y. res Extract effective compensation information from it.

[0117] 2. Codebook Joint Training

[0118] After the MSC encoder / decoder training is completed, joint training of the codebook and MSC encoder is performed. This stage involves backpropagating gradients from the decoder to the encoder, achieving end-to-end joint optimization of the model and codebook. The loss function is defined as:

[0119]

[0120] Where x represents the input image, This represents the image reconstructed using the MSC module, y res E represents the residual characteristics before and after quantization. MSC Indicates MSC encoder, z q This represents the quantization vector selected from the codebook, and sg[·] indicates stopping the gradient operation. Represents the square of the Euclidean norm; This represents the mean square error between the input image and the MSC-reconstructed image, used to measure reconstruction accuracy. This means that the MSC encoder output is compared with the codebook selection vector, while preventing gradients from being backpropagated to the encoder, for codebook training. This means comparing the codebook vector with the MSC encoder output while preventing gradients from being backpropagated to the codebook, for encoder training.

[0121] To maximize the activation rate of the codebook vector and facilitate the learning of diverse structural distributions, this method formally selects z... q Previously, a warm-up training phase was introduced. In this phase, a codebook vector z was randomly selected. q To minimize the sum of all vectors in the codebook and y c The distance effectively alleviates the problem of low codebook activation rate.

[0122] Experimental test:

[0123] This invention significantly improves the performance of learned image compression by employing a cross-granularity joint optimization strategy for training and inference of the image compression model. Taking the CLIC professional benchmark set as an example, this method achieves a 43.86% improvement in the BD-Rate of Peak Signal-to-Noise Ratio (PSNR) compared to the traditional coding standard BPG. With only a 2% increase in network parameters, this method outperforms the baseline method Liu (CVPR-23) by approximately 2%, and performs best among various comparison methods. Experimental results demonstrate that this invention achieves superior rate-distortion performance while maintaining a lightweight model.

[0124] Table 1: Comparison of BD-Rate between the Method of the Invention and the Comparative Method

[0125] Method BD-Rate (lower values are better) VTM-12.1 -34.64％ Cheng (CVPR-20) -29.93％ Xie (ACMMM-21) -33.37％ He (CVPR-22) -38.26％ Zou (CVPR-22) -36.84％ Liu (CVPR-23) -41.65％ Li (ICLR-24) -42.80％ The method of the invention -43.86％

[0126] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention should be determined by the claims.

Claims

1. A cross-granularity online image compression method based on mask compensation, characterized in that, Includes the following steps: 1) Construct an image compression model, and set up a mask-based selective compensation module MSC and a data dependency transformation module DDT in the end-to-end image compression network; 2) Encode the input image by generating a latent representation through an end-to-end encoder, and then obtain a quantized latent representation through a quantizer; 3) The quantization latent representation is reconstructed using a priori model and a probabilistic model to obtain the reconstructed quantization latent representation; 4) Calculate residual features based on the difference before and after quantization, input the residual features into the MSC module, generate compensation features through the MSC encoder, obtain sparse compensation features through codebook and sparse mask processing, and decode the sparse compensation features using the MSC decoder to obtain distribution information compensation features. 5) The reconstructed quantized latent representation is fused with the distribution information compensation features to obtain the fused latent representation; 6) Input the input image into the DDT module, where the DDT encoder and DDT decoder generate adaptive convolution kernel parameters; 7) The fused latent representation is decoded using an end-to-end decoder under adaptive convolution kernel parameters to obtain the reconstructed image; 8) Input the training data into the image compression model, train the DDT module and MSC module, calculate the loss function, and optimize the model parameters based on the loss function; 9) Compress the image to be compressed using the trained image compression model to generate the compressed reconstructed image.

2. The method as described in claim 1, characterized in that, The steps for reconstruction in step 3) include: The quantized latent representation is quantized using a super-prior model to obtain the quantized super-prior representation. The quantized prior representation is input into the probability model and processed to obtain the estimated mean and variance of the Gaussian distribution of the quantized latent representation. Based on the Gaussian distribution described above, the quantized latent representation is generated into a bitstream by an arithmetic encoder and then decoded by an arithmetic decoder to obtain the reconstructed quantized latent representation.

3. The method as described in claim 1, characterized in that, In step 4), the MSC module includes an MSC encoder, an MSC decoder, a vector quantization encoder, a vector quantization decoder, and a codebook. The MSC encoder and MSC decoder are both built based on convolutional networks. The processing steps of the MSC module include: Pre-selection operation: Obtain compensation features for residual features through the MSC encoder; for each pixel of the compensation feature, find the corresponding quantization vector in the codebook through the vector quantization encoder to obtain the quantization compensation feature and index mapping; Sparse masking operation: A sparse representation mask is generated through fine-grained optimization. This mask is then used to perform a sparse masking operation on the index mapping to calculate the sparse compensation representation. Finally, a vector quantization decoder is used to look up the codebook to generate sparse compensation features. Decoding operation: The sparse compensation features are decoded by the MSC decoder to obtain the distribution information compensation features.

4. The method as described in claim 1, characterized in that, In step 6), the DDT module includes a DDT encoder, a DDT decoder, a quantizer, an arithmetic encoder, and an arithmetic decoder, wherein the DDT encoder and DDT decoder are constructed based on a multilayer perceptron with residual connections; The processing steps of the DDT module include: The input image is encoded by a DDT encoder, multi-scale features are extracted through multi-layer convolution, and then concatenated into a compact one-dimensional vector after global average pooling. Quantize a one-dimensional vector using a quantizer with adjustable quantization parameters; After quantization, the syntax vector is generated by arithmetic encoding based on a standard Gaussian distribution using an arithmetic encoder. The syntax vector is arithmetically decoded based on a standard Gaussian distribution using an arithmetic decoder to obtain the decompressed syntax vector; The decompressed syntax vectors are decoded by the DDT decoder to generate adaptive convolution kernel parameters.

5. The method as described in claim 1, characterized in that, Step 8) employs a progressive multi-stage training strategy to train the image compression model. The steps include: First, the baseline model of the end-to-end image compression network is pre-trained, and the pre-trained parameters are loaded. Then, the DDT module is trained for coarse-grained optimization; Next, the MSC module is trained and fine-grained optimization is performed.

6. The method as described in claim 5, characterized in that, Step 8) includes the coarse-grained optimization steps for training the DDT module: Relax the bitrate constraint of the syntax vector during gradient descent training; Minimize the difference between the image-level estimated distribution and the true distribution of a single sample; Calculate the loss function The DDT module parameters are then optimized based on this loss function.

7. The method as described in claim 6, characterized in that, loss function The calculation formula is as follows: Where x represents the input image, The image is represented by y, which represents the bitstream corresponding to the quantized latent representation, and -logQ. Y (y) represents the negative log-likelihood of the bitstream y under a fixed distribution Y, λ represents the loss tradeoff parameter, Δ(,) represents the measure of image degradation, and F E Indicates DDT encoder, F D Indicates DDT decoder, φ D represents the parameters of the DDT decoder, and s represents the syntax vector.

8. The method as described in claim 5, characterized in that, Step 8) includes the following fine-grained optimization steps for training the MSC module: A sparse representation strategy is used to compensate for local structural features; Sparse compensated representations are obtained through pre-selection and sparse masking optimization operations; Distribution information compensation features are generated using the MSC decoder; Calculate the loss function The MSC module parameters are then optimized based on this loss function.

9. The method as described in claim 8, characterized in that, loss function The calculation formula is as follows: Where x represents the input image, This represents the image reconstructed using the MSC module, y res E represents the residual characteristics before and after quantization. MSC Indicates MSC encoder, z q This represents the quantization vector selected from the codebook, and sg[·] indicates stopping the gradient operation. This represents the square of the Euclidean norm.

10. A cross-granularity online image compression system based on mask compensation, characterized in that, include: An end-to-end compression network includes an end-to-end encoder and decoder, a quantizer, a super-prior model, a probabilistic model, an arithmetic encoder, and a decoder. The end-to-end encoder encodes the input image into a latent representation; the quantizer quantizes the latent representation into a quantized latent representation; the super-prior model quantizes the quantized latent representation to generate a quantized super-prior representation; the probabilistic model processes the quantized super-prior representation to obtain an estimated mean and variance of the Gaussian distribution of the quantized latent representation; the arithmetic encoder encodes the quantized latent representation into a bitstream based on the Gaussian distribution; the arithmetic decoder decodes the bitstream to generate a reconstructed quantized latent representation; and the end-to-end decoder decodes the fused latent representation under adaptive convolutional kernel parameters to generate a reconstructed image. The MSC module includes an MSC encoder, an MSC decoder, a vector quantization encoder, a vector quantization decoder, and a codebook. The MSC encoder extracts compensation features from the residual features. The vector quantization encoder matches the quantization vector in the codebook for each pixel of the compensation feature, generating quantized compensation features and an index mapping. The vector quantization decoder applies a sparse representation mask to the index mapping, generating a sparse compensation representation and generating sparse compensation features based on the codebook. The MSC decoder decodes the sparse compensation features to obtain distribution information compensation features. The DDT model includes a DDT encoder, a DDT decoder, a quantizer, an arithmetic encoder, and an arithmetic decoder. The DDT encoder encodes the input image, extracting multi-scale features through multiple convolutional layers and generating a compact one-dimensional vector via global average pooling. The quantizer performs adjustable parameter quantization on the one-dimensional vector. The arithmetic encoder performs arithmetic encoding on the quantized vector based on a standard Gaussian distribution to generate a syntax vector. The arithmetic decoder performs arithmetic decoding on the syntax vector to obtain a decompressed syntax vector. The DDT decoder decodes the decompressed syntax vector to generate adaptive convolutional kernel parameters.

Images