Distortion and perceptual code rate adaptive variable code rate depth image compression method

Abstract

The invention provides a distortion and perceptual code rate self-adaptive variable code rate depth image compression method, which is used for constructing an encoder model, and comprises the following steps of: dividing an image to be encoded into image blocks with different granularities, extracting hierarchical features, and filtering by using an image mask to obtain non-repetitively represented hierarchical features; vector quantization is carried out on the hierarchical features based on the hierarchical vector quantization codebook, and discrete index representation is generated and coded into a code stream file; constructing a decoder model, namely decoding a code stream file into discrete index representation, recovering layered features through a layered vector quantization codebook, fusing and decoding by using a mixed condition decoder, and generating a reconstructed image; carrying out joint training on the encoder model, the decoder model and the layered vector quantization codebook; and encoding an input image into a code stream file through the trained encoder model, and decoding the code stream file into a reconstructed image through the trained decoder model. According to the invention, fine bit rate control and adaptive switching between perception and distortion optimization can be realized.

Description

Technical Field

[0001] The present invention belongs to the field of digital image processing, and in particular relates to a variable bit rate image compression scheme based on deep learning. Background Art

[0002] Since the advent of digital imaging, lossy image compression has been a fundamental research topic in computer vision and image processing. Over the past few decades, a series of classic image compression standards such as JPEG, JPEG2000, BPG, and VVC have been developed and widely used in various practical scenarios, significantly improving storage and transmission efficiency. At the forefront of academic research, learning-based image compression methods have made significant progress and demonstrated excellent rate-distortion performance. Although these deep learning methods have achieved excellent rate-distortion performance, they are still subject to the constraint of fixed bitrate. Specifically, deep models are trained for discrete bitrate levels, which inherently lack adaptive compression capabilities across different bitrates, resulting in high training complexity and high deployment costs in multi-bitrate scenarios.

[0003] To address this issue, a series of variable-rate compression methods have been proposed, achieving bitrate adjustment within a single model through methods such as rate control hyperparameter injection, feature editing, and non-uniform quantization. These methods achieve comparable performance to single-rate methods while maintaining coding flexibility. However, as the allocated bitrate decreases, they inevitably suffer from a gradual degradation in visual quality, such as over-smoothing of textures and severe smearing artifacts.

[0004] Driven by the rapid development of generative models, perception-oriented compression methods have emerged as a promising solution for mitigating blurring artifacts. These methods prioritize preserving key perceptual features over pixel-level fidelity, aiming to reconstruct visually natural and semantically consistent content at a low bitrate. Building on this foundation, generative variable-rate compression methods have also been proposed. However, due to the inherent distortion-perception tradeoff, these perception-oriented methods inevitably exhibit compromised fidelity in texture reconstruction.

[0005] In summary, existing methods face a fundamental conflict in bitrate adaptability between fidelity and perceptual quality due to the "distortion-perception" trade-off. Specifically, while distortion-oriented methods can achieve pixel-accurate reconstruction at high bitrates, their signal-fidelity-based optimization mechanisms can easily blur key semantic structures when compressed at low bitrates. While perception-oriented methods can maintain visual plausibility at low bitrates, the uncontrollable nature of their generation mechanisms can lead to structural deviations or semantic distortion at high bitrates. This mutually exclusive drawback makes it difficult for existing methods to achieve an adaptive balance in reconstruction quality during continuous bitrate changes. Therefore, there is an urgent need to develop a variable-rate deep image compression method that is both distortion- and perceptual-adaptive, preserving faithful textures at high bitrates while maintaining semantic content and generative details at low bitrates, thereby enabling adaptive reconstruction of visually pleasing images at varying bitrates. Currently, no distortion- and perceptual-adaptive variable-rate deep image compression method exists, either domestically or internationally. No relevant papers have been published in domestic or international journals. At present, there is no distortion and perceptual bit rate adaptive variable bit rate deep image compression method and authorized patent in China. Summary of the Invention

[0006] In view of the deficiencies in the prior art, the present invention provides a distortion-perception bit rate adaptive variable bit rate deep image compression method.

[0007] The technical solution of the present invention is a variable bit rate deep image compression method with adaptive distortion and perceptual bit rate, which performs the following process: The encoder model is constructed, which includes dividing the image to be encoded into image blocks of different granularities, extracting hierarchical features through a hierarchical encoder, filtering the hierarchical features using image masks to obtain non-repeated hierarchical features; vector quantizing the hierarchical features based on a hierarchical vector quantization codebook, generating a discrete index representation, and encoding it into a bitstream file; Decoder model construction, including decoding the code stream file into a discrete index representation, recovering the hierarchical features through the layered vector quantization codebook, fusing and decoding the hierarchical features using a hybrid conditional decoder, and generating a reconstructed image; The model training process includes joint training of the encoder model, decoder model, and layered vector quantization codebook; The output results include encoding the input image into a bitstream file through the trained encoder model and decoding the bitstream file into a reconstructed image through the trained decoder model.

[0008] Moreover, the image blocks are divided based on information density, and the image is divided into three non-overlapping image blocks of different sizes: fine-grained, medium-grained, and coarse-grained, and the division results are marked by image masks.

[0009] Moreover, the proportion of the number of fine-grained, medium-grained, and coarse-grained image blocks is controlled by hyperparameters.

[0010] Moreover, the hierarchical encoder adopts the backbone network of vector quantized variational autoencoder to extract hierarchical features from different network depths.

[0011] Moreover, the hybrid conditional decoder first fuses features from different levels, and the obtained aggregated features are subjected to hierarchical decoding and feature updating to reconstruct a decoded image.

[0012] Moreover, the model training is implemented using a training method based on a vector quantized variational autoencoder to achieve an adaptive balance between distortion and perception trade-offs.

[0013] Moreover, the code rate control of the layered vector quantization codebook is achieved by adjusting the granularity distribution ratio of the image blocks. Fine-grained image blocks are allocated to high information density areas to retain texture details, and coarse-grained image blocks are allocated to low information density areas to reduce the code rate.

[0014] On the other hand, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and runnable on the processor, wherein when the processor executes the program, the variable bit rate deep image compression method with adaptive distortion and perceptual bit rate as described above is implemented.

[0015] On the other hand, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the variable bit rate deep image compression method with adaptive distortion and perceptual bit rate is implemented as described above.

[0016] On the other hand, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the variable bit rate deep image compression method with adaptive distortion and perceptual bit rate as described above.

[0017] The present invention simultaneously considers faithful texture restoration at high bit rates and generation of missing details at low bit rates, utilizes the perception-oriented characteristics of deep features and distortion-oriented characteristics of shallow features, decomposes the image into hierarchical feature sets, learns a multi-level codebook for vector quantization, and performs bit rate control based on the division of hierarchical features, thereby achieving fine bit rate control and adaptive switching between perception and distortion optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 3 is a structural diagram of a distortion-perception bitrate adaptive variable bitrate deep image compression method according to an embodiment of the present invention.

[0019] Figure 2 This is a performance comparison chart of the method according to the embodiment of the present invention and mainstream image compression methods. DETAILED DESCRIPTION

[0020] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0021] See also Figure 1 The embodiment of the present invention provides a distortion-perception bitrate adaptive variable bitrate deep image compression method, comprising the following steps: Step 1: Encoder model construction, which involves dividing the image to be encoded into blocks of different sizes according to information density, extracting feature representations of non-repetitive representations through a layered encoder, quantizing the feature representations into discrete index representations using a layered vector quantization codebook, and finally storing the discrete index representations as a bitstream file; The code rate control of the layered vector quantization codebook is achieved by adjusting the granularity distribution ratio of the image blocks. Fine-grained image blocks are allocated to high-information-density areas to retain texture details, and coarse-grained image blocks are allocated to low-information-density areas to reduce the code rate.

[0022] The encoder model of the embodiment is specifically implemented as follows: 1) The image to be encoded The image is divided into three non-overlapping blocks of different sizes according to the information density, and the division results are represented by the image mask. express; The present invention further proposes that the three sizes of image blocks are all square, and their pixel sizes are all powers of 2, and can form a geometric progression with a ratio of 2, such as 4, 8, and 16.

[0023] The embodiment preferably uses the information entropy of the image block as a measure to divide the image block, and sets three different image block sizes of 4×4, 8×8, and 16×16, which are respectively called fine-grained image blocks, medium-grained image blocks, and coarse-grained image blocks. As a hyperparameter, it controls the image division method, which in turn affects the bit rate and the distortion and perceptual characteristics of the decoded image. % of the image blocks are divided into fine-grained blocks, and then % of the image blocks are divided into medium-sized blocks, and finally % of the image blocks are divided into coarse-grained image blocks. In specific implementation, it can be pre-set according to needs or experience The value of + + =100.

[0024] Mask the image Set to all zeros, the area marked as fine-grained block is in the image mask The corresponding position is set to 1, and the area marked as a medium-sized block is in the image mask. The corresponding position of is set to 1, and the area marked as a coarse-grained block is in the image mask. The corresponding position of is set to 1, and the image mask representing the final image division mode is obtained. .

[0025] 2) Image to be encoded Send it to the layered encoder to get features of different depths , whose spatial dimension corresponds to the spatial dimension of the image mask, and then the image mask is used to map the depth features Perform spatial filtering to obtain hierarchical features with non-repeated representations ; In the embodiment, the encoder backbone network of the vector quantization variational autoencoder VQ-VAE is preferably used as the hierarchical encoder, and hierarchical features of different network depths and different spatial dimensions are obtained from the last stage, the second to last stage, and the third to last stage. The specific implementation of the vector quantization variational autoencoder VQ-VAE can adopt existing technology, which will not be described in detail in the present invention.

[0026] Then, according to the image mask , the hierarchical features are filtered by element-wise multiplication, and this process can be expressed as:

[0027] in, is element-by-element multiplication. Finally, we get the hierarchical features of non-repeated representation .

[0028] 3) Based on the layered vector quantization codebook , for hierarchical features Perform vector quantization to obtain its discrete index representation and encode it into a binary code stream file; In the embodiment, firstly the hierarchical features The spatial dimension is expanded and the all-zero features filtered by the image mask are removed. Subsequently, vector quantization is used to convert the high-dimensional continuous feature vector into a one-dimensional discrete index representation. For example, the entry in the codebook is , the feature to be vector quantized is ,in is the codebook index, is the feature index, the vector quantization process can be expressed as:

[0029]

[0030] in, is the feature after vector quantization, It is in the code book Discrete index in , is the Euclidean distance. Each feature in is vector quantized to obtain its corresponding index representation By performing the above operations on the features and codebook of each layer, the discrete index representation is obtained Finally, through Huffman lossless coding, and Encode into binary file for storage or transmission.

[0031] Step 2: Decoder model construction, which includes restoring the code stream file to a discrete index representation, then restoring the discrete index representation to a feature representation using a layered vector quantization codebook, and then decoding the image using a hybrid conditional decoder; The encoder model of the embodiment is specifically implemented as follows: 1) Decode the binary code stream file into a discrete index representation and use the layered vector quantization codebook Reconstruct the hierarchical features after vector quantization ; In the embodiment, the binary file is first decoded by Huffman to obtain and .pass The index value in the corresponding codebook Query the corresponding quantized features in and decode the index representation vector into feature representation.

[0032] Subsequently, through The spatial position information of the features saved in is used to place the decoded features in the correct spatial position and restore them to the original feature dimensions. .

[0033] 2) Layered features Send it to the hybrid conditional decoder to reconstruct the decoded image .

[0034] In the embodiment, a hybrid conditional decoder solution is provided. First, features from different levels are fused through a feature aggregation module. This process can be expressed as:

[0035] in, represents a downsampling module with convolutional layers, It is the fusion information feature obtained by feature aggregation.

[0036] Subsequently, the aggregated features are decoded and updated hierarchically to reconstruct the decoded image. , this process can be expressed as:

[0037]

[0038]

[0039] in, represents the sub-networks at different decoding stages, For the negation operation, They are the first layer decoding features, the second layer decoding features and the decoded image respectively.

[0040] Step 3, model training, includes training the encoder model constructed in step 1 and the decoder model constructed in step 2, involving joint training of the layered encoder in the encoder model, the hybrid conditional decoder in the decoder model, and the shared layered vector quantization codebook; In this embodiment, a general training method for a vector quantized variational autoencoder (VQ-VAE) is preferably employed to optimize distortion and perceptual rate adaptation. In specific implementations, the training platform can be based on the open-source Pytorch framework, written in Python. It is also recommended to use a high-performance computer equipped with a high-performance graphics card for graphics computing.

[0041] Step 4, outputting the result, includes encoding the input image into a bitstream file according to the encoder model trained in step 3, and decoding the bitstream file into a reconstructed image according to the decoder model trained in step 3.

[0042] In the embodiment, the training results in step 3 are presented. Image encoding is achieved by inputting an image to be encoded and outputting a corresponding bitstream file. Image decoding is achieved by inputting a bitstream file and outputting a corresponding reconstructed image.

[0043] To facilitate understanding of the technical effects of the present invention, a performance comparison between the present invention and mainstream image compression algorithms is provided. The comparison algorithms include distortion-oriented methods (traditional algorithm VTM, deep learning algorithms M&S, CTC) and perception-oriented methods (deep learning algorithms HiFiC, MS-ILLM, CDC, CGIC). All experimental data were obtained by testing on the Kodak dataset. The results are shown in the figure below. Figure 2As can be seen, at low bitrates (bpp < 0.2 on the horizontal axis), the present invention achieves comparable performance to the perception-oriented single-rate algorithms MS-ILLM and CDC in terms of the perceptual quality metrics FID and LPIPS. At high bitrates (bpp > 0.2 on the horizontal axis), the present invention approaches the distortion-oriented algorithms in terms of PSNR, while outperforming perception-oriented methods. Furthermore, the present invention achieves finer-grained bitrate control, simultaneously balancing distortion quality at high bitrates and perceptual quality at low bitrates, and is able to adaptively balance distortion and perceptual trade-offs across fine-grained bitrate variations.

[0044] The following describes the variable bit rate deep image compression electronic device with adaptive distortion and perceptual bit rate provided by the present invention. The variable bit rate deep image compression electronic device with adaptive distortion and perceptual bit rate described below and the variable bit rate deep image compression method with adaptive distortion and perceptual bit rate described above can be referenced to each other.

[0045] The electronic device may include: a processor, a communications interface, a memory, and a communications bus, wherein the processor, the communications interface, and the memory communicate with each other via the communications bus. The processor may invoke logic instructions in the memory to execute a variable bitrate deep image compression method with adaptive distortion and perceptual bitrate, which primarily includes the software processing portion of the above steps.

[0046] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0047] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the software processing part of the variable bit rate deep image compression method with adaptive distortion and perceptual bit rate provided by the above methods.

[0048] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the software processing part of the variable bit rate deep image compression method with adaptive distortion and perceptual bit rate provided by the above methods.

[0049] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0050] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A variable bit rate deep image compression method with adaptive distortion and perceptual bit rate, characterized in that: Follow the process below: The encoder model is constructed, which includes dividing the image to be encoded into image blocks of different granularities, extracting hierarchical features through a hierarchical encoder, and filtering the hierarchical features using image masks to obtain hierarchical features with non-repetitive representations; Perform vector quantization on hierarchical features based on hierarchical vector quantization codebook, generate discrete index representation and encode it into a bitstream file; Decoder model construction, including decoding the code stream file into a discrete index representation, recovering the hierarchical features through the layered vector quantization codebook, fusing and decoding the hierarchical features using a hybrid conditional decoder, and generating a reconstructed image; The model training process includes joint training of the encoder model, decoder model, and layered vector quantization codebook; The output results include encoding the input image into a bitstream file through the trained encoder model and decoding the bitstream file into a reconstructed image through the trained decoder model.

2. The distortion and perceptual bitrate adaptive variable bitrate deep image compression method according to claim 1, characterized in that: The image blocks are divided based on information density, and the image is divided into three non-overlapping image blocks of different sizes: fine-grained, medium-grained, and coarse-grained, and the division results are marked by image masks.

3. The distortion and perceptual bitrate adaptive variable bitrate deep image compression method according to claim 2, characterized in that: The proportion of the number of fine-grained, medium-grained, and coarse-grained image blocks is controlled by hyperparameters.

4. The distortion and perceptual bitrate adaptive variable bitrate deep image compression method according to claim 1, characterized in that: The hierarchical encoder uses a backbone network of vector quantized variational autoencoders to extract hierarchical features from different network depths.

5. The distortion and perceptual bitrate adaptive variable bitrate deep image compression method according to claim 1, characterized in that: The hybrid conditional decoder first fuses features from different levels, and the obtained aggregated features are subjected to hierarchical decoding and feature updating to reconstruct the decoded image.

6. The distortion and perceptual bitrate adaptive variable bitrate deep image compression method according to claim 1, characterized in that: The model training is implemented using a training method based on a vector quantized variational autoencoder to achieve an adaptive balance between distortion and perception trade-offs.

7. The distortion and perceptual bitrate adaptive variable bitrate deep image compression method according to claim 1, characterized in that: The code rate control of the layered vector quantization codebook is achieved by adjusting the granularity distribution ratio of the image blocks. Fine-grained image blocks are allocated to high-information-density areas to retain texture details, and coarse-grained image blocks are allocated to low-information-density areas to reduce the code rate.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the variable bit rate deep image compression method with adaptive distortion and perceptual bit rate as claimed in any one of claims 1 to 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the distortion and perceptual bit rate adaptive variable bit rate deep image compression method according to any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program, characterized in that: When the computer program is executed by a processor, the distortion and perceptual bit rate adaptive variable bit rate deep image compression method according to any one of claims 1 to 7 is implemented.

Images