Variable bit rate depth image compression method with distortion and perceptual bit rate adaptation

Abstract

The application provides a distortion and perceptual code rate adaptive variable code rate depth image compression method, encoder model construction is carried out, including dividing the image to be encoded into image blocks of different granularities, extracting hierarchical features, and obtaining non-repeated hierarchical features by filtering using an image mask; hierarchical vector quantization codebooks are used to quantize the hierarchical features, discrete index representations are generated, and code stream files are encoded; decoder model construction, including decoding the code stream files into discrete index representations, recovering the hierarchical features through the hierarchical vector quantization codebooks, fusing and decoding using a hybrid conditional decoder, and generating a reconstructed image; the encoder model, the decoder model and the hierarchical vector quantization codebook are jointly trained; the input image is encoded into a code stream file through the trained encoder model, and the code stream file is decoded into a reconstructed image through the trained decoder model. The application can realize fine bit rate control and adaptive switching between perceptual and distortion optimization.

Description

Distortion and perceptual bitrate adaptive variable bitrate depth image compression method Technical Field

[0001] This invention belongs to the field of digital image processing, and particularly relates to a variable bitrate image compression scheme based on deep learning. Background Technology

[0002] Since the advent of digital imaging, lossy image compression has been a fundamental research topic in computer vision and image processing. Over the past few decades, a series of classic image compression standards, such as JPEG, JPEG2000, BPG, and VVC, have been developed and widely applied in various practical scenarios, significantly improving storage and transmission efficiency. At the forefront of academic research, learning-based image compression methods have made significant progress and exhibited excellent rate-distortion performance. Despite these deep learning methods achieving excellent rate-distortion performance, they are still constrained by fixed bitrates. Specifically, deep models are trained at discrete bitrate levels, and discrete bitrates inherently lack adaptive compression capabilities across different bitrates, resulting in high training complexity and deployment costs in multi-bitrate scenarios.

[0003] To address this issue, a series of variable-rate compression methods have been proposed, achieving bitrate adjustment within a single model through techniques such as bitrate-controlled hyperparameter injection, feature editing, and non-uniform quantization. These methods achieve performance comparable to single-bitrate methods while maintaining coding flexibility. However, as the allocated bitrate decreases, they inevitably experience a gradual decline in visual quality, such as over-smoothing of textures and severe smearing artifacts.

[0004] Driven by the rapid development of generative models, perceptual compression methods have emerged as promising solutions for mitigating blur artifacts. These methods prioritize preserving key perceptual features rather than pixel-level fidelity, aiming to reconstruct visually natural and semantically consistent content at low bitrates. Building upon this, generative variable-rate compression methods have also been proposed. However, due to the inherent distortion-perceptual tradeoff, these perceptual methods inevitably exhibit compromised fidelity in texture reconstruction.

[0005] In summary, existing methods suffer from a fundamental contradiction in bitrate adaptability between fidelity and perceptual quality due to the "distortion-perception" trade-off. Specifically, distortion-oriented methods, while achieving accurate pixel-level reconstruction at high bitrates, are prone to causing blurring and distortion of key semantic structures at low bitrates due to their signal fidelity-based optimization mechanisms. Conversely, perceptual-oriented methods, while maintaining visual plausibility in low bitrate scenarios, suffer from structural deviations or semantic distortions in high bitrate reconstruction due to the uncontrollability of the generation mechanism. This mutually exclusive defect indicates that existing methods struggle to achieve an adaptive balance of reconstruction quality across continuous bitrate variations. Therefore, there is an urgent need to research a distortion-perceptual bitrate-adaptive variable bitrate depth image compression method that maintains faithful texture at high bitrates while preserving semantic content and generation details at low bitrates, thereby achieving visually pleasing adaptive image reconstruction at different bitrates. Currently, there is no distortion-perceptual bitrate-adaptive variable bitrate depth image compression method available domestically or internationally. No related papers have been published in domestic or international journals. Currently, there are no distortion- and perceptual bitrate adaptive variable bitrate depth image compression methods or authorized patents in China. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a distortion-aware bitrate adaptive variable bitrate depth image compression method.

[0007] The technical solution of this invention is a variable bitrate depth image compression method that adapts to both distortion and perceptual bitrate, comprising the following steps:

[0008] The encoder model construction includes dividing the image to be encoded into image blocks of different granularities, extracting layered features through a layered encoder, filtering the layered features using an image mask to obtain non-repetitive layered features, and performing vector quantization on the layered features based on the layered vector quantization codebook to generate discrete index representations and encode them into a bitstream file.

[0009] Decoder model construction includes decoding the bitstream file into a discrete index representation, recovering the hierarchical features through a hierarchical vector quantization codebook, and using a hybrid conditional decoder to fuse and decode the hierarchical features to generate a reconstructed image;

[0010] The model training process includes joint training of the encoder model, decoder model, and hierarchical vector quantization codebook;

[0011] The output includes encoding the input image into a bitstream file using the trained encoder model, and decoding the bitstream file into a reconstructed image using the trained decoder model.

[0012] Moreover, the image block division described herein is based on information density, dividing the image into three different sizes of non-overlapping image blocks: fine-grained, medium-grained, and coarse-grained, and identifying the division results through image masks.

[0013] Furthermore, the proportion of fine-grained, medium-grained, and coarse-grained image blocks is controlled by hyperparameters.

[0014] Moreover, the hierarchical encoder uses a vector quantization variational autoencoder backbone network to extract hierarchical features from different network depths.

[0015] Moreover, the hybrid conditional decoder first fuses features from different levels, and the resulting aggregated features are then subjected to hierarchical decoding and feature updates to reconstruct the decoded image.

[0016] Moreover, the model training is achieved using a training method based on vector quantization variational autoencoder, which realizes an adaptive balance between distortion and perception tradeoffs.

[0017] Moreover, the bitrate control of the layered vector quantization codebook is achieved by adjusting the granularity distribution ratio of image blocks. Fine-grained image blocks are allocated to high information density regions to preserve texture details, while coarse-grained image blocks are allocated to low information density regions to reduce the bitrate.

[0018] On the other hand, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the variable bitrate depth image compression method with distortion and perceptual bitrate adaptation as described above.

[0019] On the other hand, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the variable bitrate depth image compression method described above for distortion and perceptual bitrate adaptation.

[0020] On the other hand, the present invention provides a computer program product, comprising a computer program that, when executed by a processor, implements the variable bitrate depth image compression method described above for distortion and perceptual bitrate adaptation.

[0021] This invention simultaneously considers faithful texture restoration at high bit rates and the generation of missing details at low bit rates. It utilizes the characteristics of deep features oriented towards perception and shallow features oriented towards distortion. By decomposing the image into layered feature sets and learning multi-level codebooks for vector quantization, and controlling the bit rate according to the division of layered features, it achieves fine bit rate control and adaptive switching between perception and distortion optimization. Attached Figure Description

[0022] Figure 1 is a schematic diagram of the structure of the distortion-aware bitrate adaptive variable bitrate depth image compression method according to an embodiment of the present invention.

[0023] Figure 2 is a performance comparison chart of the method of the present invention and mainstream image compression methods. Detailed Implementation

[0024] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0025] Referring to Figure 1, this embodiment of the invention provides a distortion-aware bitrate adaptive variable bitrate depth image compression method, comprising the following steps:

[0026] Step 1, encoder model construction, includes dividing the image to be encoded into image blocks of different sizes according to information density, then extracting the feature representation of non-repetitive representation through a hierarchical encoder, then quantizing the feature representation into discrete index representation through a hierarchical vector quantization codebook, and finally storing the discrete index representation as a bitstream file;

[0027] The bitrate control of the hierarchical vector quantization codebook is achieved by adjusting the granularity distribution ratio of image blocks. Fine-grained image blocks are allocated to high information density regions to preserve texture details, while coarse-grained image blocks are allocated to low information density regions to reduce the bitrate.

[0028] The encoder model in this embodiment is implemented as follows:

[0029] 1) The image to be encoded The image is divided into three non-overlapping sizes based on information density, and the division result is determined by the image mask. express;

[0030] The present invention further proposes that the three types of image blocks are all squares, and their pixel size is a power of 2, and they can form a geometric sequence with a ratio of 2, such as 4, 8, and 16.

[0031] In this embodiment, the information entropy of image blocks is preferably used as the criterion for dividing image blocks. Three different image block sizes are set: 4×4, 8×8, and 16×16, respectively referred to as fine-grained, medium-grained, and coarse-grained image blocks. The proportion of each type of image block is then determined. As a hyperparameter controlling the image segmentation method, it consequently affects the bitrate and the distortion and perceptual characteristics of the decoded image. Sorted from high to low information entropy, the top... % of the image patches were divided into fine-grained blocks, and then... % of the image patches were divided into medium-granularity blocks, and finally A certain percentage of image blocks are divided into coarse-grained image blocks. In practice, this can be pre-defined based on requirements or experience. The value of , + + =100.

[0032] Mask the image Setting the region to all zeros marks the area marked as a fine-grained block in the image mask. The corresponding position is set to 1, and the region marked as a medium-granularity block is in the image mask. The corresponding position is set to 1, and the region marked as a coarse-grained block in the image mask. The corresponding position is set to 1, resulting in the final image mask representing the image segmentation pattern. .

[0033] 2) Image to be encoded The data is fed into a hierarchical encoder to obtain features at different depths. Its spatial dimension corresponds to the spatial dimension of the image mask, and then the image mask is used to process the depth features. Spatial filtering is performed to obtain hierarchical features with non-repetitive representations. ;

[0034] In this embodiment, a vector quantization variational autoencoder (VQ-VAE) backbone network is preferably used as a hierarchical encoder, and hierarchical features with different network depths and spatial dimensions are obtained from its last stage, penultimate stage, and penultimate stage. The specific implementation of the Vector Quantization Variational Autoencoder (VQ-VAE) can be achieved using existing technologies, which will not be elaborated upon in this invention.

[0035] Subsequently, based on the image mask The process of filtering hierarchical features through element-wise multiplication can be represented as follows:

[0036]

[0037] in, This is an element-wise multiplication. The final result is a hierarchical feature representation with no repetition. .

[0038] 3) Based on the hierarchical vector quantization codebook For hierarchical features Perform vector quantization to obtain its discrete index representation, and encode it into a binary bitstream file;

[0039] In the embodiment, firstly, the hierarchical features are... Spatial dimensionality is expanded, and all-zero features filtered by the image mask are removed. Then, vector quantization is used to transform the high-dimensional continuous feature vector into a one-dimensional discrete index representation. A specific vector quantization codebook is used. For example, the entries in the codebook are The features to be vector quantized are ,in For codebook index, If the feature index is used, then the vector quantization process can be represented as:

[0040]

[0041]

[0042] in, These are features after vector quantization. It is in the codebook Discrete indexes in This is the Euclidean distance. (Based on...) Each feature in the dataset is vector-quantized to obtain its corresponding index representation. By performing the above operations on the features and codebook of each layer, a discrete index representation is obtained. Ultimately, through Huffman lossless coding, the... and Encoded as a binary file for storage or transmission.

[0043] Step 2, Decoder Model Construction, includes restoring the bitstream file to a discrete index representation, then restoring the discrete index representation to a feature representation through a hierarchical vector quantization codebook, and finally decoding the image through a hybrid conditional decoder;

[0044] The encoder model in this embodiment is implemented as follows:

[0045] 1) Decode the binary bitstream file into a discrete index representation, and then use the hierarchical vector quantization codebook. Reconstructing the hierarchical features after vector quantization ;

[0046] In this embodiment, the binary file is first obtained through Huffman decoding. and .pass The index value in the corresponding codebook The corresponding quantization features are queried in the database, and the index representation vector is decoded into feature representation.

[0047] Subsequently, through The spatial location information of the features stored in the database is used to place the decoded features in the correct spatial location, restoring the original feature dimensions, and thus obtaining... .

[0048] 2) Hierarchical features The image is fed into a mixed conditional decoder to reconstruct the decoded image. .

[0049] In this embodiment, a hybrid conditional decoder scheme is provided. First, a feature aggregation module fuses features from different levels; this process can be represented as:

[0050]

[0051] in, This indicates a downsampling module with convolutional layers. The fused information features are obtained by feature aggregation.

[0052] Subsequently, the aggregated features undergo hierarchical decoding and feature updates to reconstruct the decoded image. This process can be represented as:

[0053]

[0054]

[0055]

[0056] in, This represents a sub-network representing a different decoding stage. For the inverse operation, These are the first layer decoding features, the second layer decoding features, and the decoded image, respectively.

[0057] Step 3, model training, includes training the encoder model built in Step 1 and the decoder model built in Step 2, involving joint training of the hierarchical encoder in the encoder model, the hybrid conditional decoder in the decoder model, and the shared hierarchical vector quantization codebook.

[0058] In this embodiment, the general training method of Vector Quantization Variational Autoencoder (VQ-VAE) is preferably used to optimize distortion and perceptual bitrate adaptation performance. Specifically, the training platform can be based on the PyTorch open-source framework and the Python language. It is also recommended to use a high-performance computer equipped with a high-performance graphics card for graphics computation.

[0059] Step 4, output the results, including encoding the input image into a bitstream file based on the encoder model trained in Step 3, and decoding the bitstream file into a reconstructed image based on the decoder model trained in Step 3.

[0060] In this embodiment, the training results in step 3 are demonstrated. Image encoding is achieved by inputting the image to be encoded and outputting the corresponding bitstream file. Image decoding is achieved by inputting the bitstream file and outputting the corresponding reconstructed image.

[0061] To facilitate understanding of the technical effects of this invention, a performance comparison between this invention and mainstream image compression algorithms is provided. The compared algorithms include distortion-oriented methods (traditional algorithm VTM, deep learning algorithms M&S, CTC) and perception-oriented methods (deep learning algorithms HiFiC, MS-ILLM, CDC, CGIC). All experimental data were obtained on the Kodak dataset, and the results are shown in Figure 2. It can be seen that under low bitrate conditions (bpp < 0.2 on the horizontal axis), this invention achieves performance comparable to the perception-oriented single-bitrate algorithms MS-ILLM and CDC in terms of perceptual quality metrics FID and LPIPS. Under high bitrate conditions (bpp > 0.2 on the horizontal axis), this invention approaches the distortion-oriented algorithms in terms of PSNR and outperforms the perception-oriented methods. Furthermore, this invention achieves finer-grained bitrate control, simultaneously considering both distortion quality at high bitrates and perceptual quality at low bitrates, adaptively balancing distortion and perceptual tradeoffs within finely varied bitrates.

[0062] The present invention provides an electronic device for adaptive variable bitrate and depth image compression with distortion and perceptual bitrate. The electronic device for adaptive variable bitrate and depth image compression with distortion and perceptual bitrate described below can be referred to in correspondence with the image compression method for adaptive variable bitrate and depth image compression with distortion and perceptual bitrate described above.

[0063] The electronic device may include a processor, a communications interface, memory, and a communication bus, wherein the processor, communications interface, and memory communicate with each other via the communication bus. The processor can call logical instructions in the memory to execute a variable bitrate depth image compression method that adapts to distortion and perceptual bitrate, mainly including the software processing part mentioned above.

[0064] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0065] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program that can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the software processing part of the distortion and perceptual bit rate adaptive variable bit rate depth image compression method provided by the above methods.

[0066] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the software processing portion of the distortion and perceptual bit rate adaptive variable bit rate depth image compression method provided by the above methods.

[0067] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0068] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A variable bitrate depth image compression method that adapts to both distortion and perceptual bitrate, characterized in that, The following process is performed: Encoder model construction, including dividing the image to be encoded into three different sizes of non-overlapping image patches—fine-grained, medium-grained, and coarse-grained—based on the image's information density, and then using image masking. The segmentation results are identified, and the hierarchical features are extracted by the hierarchical encoder. The hierarchical features are then filtered using an image mask to obtain non-repetitive hierarchical features. The layered features are vector-quantized based on a layered vector quantization codebook to generate discrete index representations and encode them into a bitstream file. The layered features are filtered using an image mask, which involves element-wise multiplication of the image mask with the layered features to obtain non-repeating layered features. The bitrate control of the layered vector quantization codebook is achieved by adjusting the granularity distribution ratio of image blocks; fine-grained image blocks are allocated to high-information-density regions to preserve texture details, while coarse-grained image blocks are allocated to low-information-density regions to reduce the bitrate. The decoder model is constructed by decoding the bitstream file into a discrete index representation and recovering the layered features using the layered vector quantization codebook. A reconstructed image is generated by fusing and decoding layered features using a hybrid conditional decoder. The fusing and decoding of layered features using the hybrid conditional decoder is implemented as follows: [The text abruptly ends here, so the translation stops.] The features are fed into the hybrid conditional decoder, where they first pass through a feature aggregation module to fuse features from different levels, as shown below: in, This indicates a downsampling module with convolutional layers. The fused information features are obtained through feature aggregation; subsequently, the aggregated features undergo hierarchical decoding and feature updating to reconstruct the decoded image. , is represented as: in, This represents a sub-network representing a different decoding stage. For the inverse operation, For element-wise multiplication, These are the first layer of decoding features, the second layer of decoding features, and the decoded image, respectively; the model training process includes joint training of the encoder model, the decoder model, and the hierarchical vector quantization codebook; the output results include encoding the input image into a bitstream file using the trained encoder model, and decoding the bitstream file into a reconstructed image using the trained decoder model.

2. The variable bitrate depth image compression method with distortion and perceptual bitrate adaptation according to claim 1, characterized in that: The image block division is based on information density, dividing the image into three different sizes of non-overlapping image blocks: fine-grained, medium-grained, and coarse-grained. All three sizes of image blocks are square, and their pixel size is a power of 2, and they can form a geometric sequence with a ratio of 2.

3. The variable bitrate depth image compression method with distortion and perceptual bitrate adaptation according to claim 2, characterized in that: The proportion of fine-grained, medium-grained, and coarse-grained image blocks is controlled by hyperparameters.

4. The variable bitrate depth image compression method with distortion and perceptual bitrate adaptation according to claim 1, characterized in that: The hierarchical encoder uses a vector quantization variational autoencoder backbone network to extract hierarchical features from different network depths.

5. The variable bitrate depth image compression method with distortion and perceptual bitrate adaptation according to claim 1, characterized in that: The model training is achieved using a training method based on vector quantization variational autoencoder, which realizes an adaptive balance between distortion and perception tradeoffs.

6. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the variable bitrate depth image compression method with distortion and perceptual bitrate adaptation as described in any one of claims 1 to 5.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the variable bitrate depth image compression method with distortion and perceptual bitrate adaptation as described in any one of claims 1 to 5.

8. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by the processor, it implements the variable bitrate depth image compression method with distortion and perceptual bitrate adaptation as described in any one of claims 1 to 5.

Images