Ultra-low-bitrate high-quality learned image compression with hybrid learned sparse and continuous visual representations with high fidelity guidance

Abstract

A method for performing Learned Image Compression (LIC) using a hybrid Learned Sparse Image Representation (LSVR) and Learned Continuous Visual Representation (LCVR) encoding is described. The method, implemented by a video encoding device, includes encoding an image into a bitstream as a set of code indices representing a set of basis elements according to LSVR; encoding the image into the bitstream as a series of continuous real numbers according to LCVR; determining, via a neural network (NN), a degradation-aware latent, which approximates degradation that occurs to the image during LCVR encoding; encoding the degradation-aware latent into the bitstream; and storing the bitstream.

Description

Atty. Docket No.4502-83401 (6000689PCT02) ULTRA-LOW-BITRATE HIGH-QUALITY LEARNED IMAGE COMPRESSION WITH HYBRID LEARNED SPARSE AND CONTINUOUS VISUAL REPRESENTATIONS WITH HIGH FIDELITY GUIDANCE CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No.63 / 574,341 filed on April 4, 2024. The disclosure of the aforementioned application is hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0002] The present disclosure relates to Learned Image Compression (LIC), and in particular, to hybrid learning based image compression systems that employ both Learned Continuous Visual Representation (LCVR) and Learned Sparse Image Representation (LSVR). BACKGROUND

[0003] Learned Image Compression (LIC) is a video coding approach that employs neural networks (NN) to compress video for transmission between a sender and a receiver. In some cases, LIC may result in superior performance over coding methods like Joint Photographic Experts Group (JPEG), Versatile Video Coding (VVC), and High Efficiency Video Coding (HEVC). However, different LIC approaches have different benefits and drawbacks. SUMMARY

[0004] A first aspect relates to a method implemented in a video encoding device, the method comprising: encoding an image into a bitstream as a set of code indices ^^^representing a set of basis elements ^^^according to Learned Sparse Image Representation (LSVR); encoding the image into the bitstream as a series of continuous real numbers ^^^according to Learned Continuous Visual Representation (LCVR); determining, via a neural network (NN), a degradation-aware latent ^^^,Atty. Docket No.4502-83401 (6000689PCT02) which approximates degradation that occurs to the image during LCVR encoding; encoding the degradation-aware latent ^^^into the bitstream; and storing the bitstream.

[0005] Optionally, in an implementation according to the first aspect, the degradation-aware latent ^^^is encoded into the bitstream via entropy coding.

[0006] Optionally, in an implementation according to the first aspect or any implementation thereof, the degradation-aware latent ^^^is determined by decoding the series of continuous real numbers ^^^to obtain a LCVR-based decoded image x^^and using the NN to estimate degradation that occurred to the LCVR-based decoded image x^^based on prior training on degradation that occurs to images during LCVR.

[0007] Optionally, in an implementation according to the first aspect or any implementation thereof, the degradation-aware latent ^^^is determined by using the NN to estimate degradation that occurred to the series of continuous real numbers ^^^based on prior training on degradation that occurs to images during LCVR.

[0008] Optionally, in an implementation according to the first aspect or any implementation thereof, the NN is trained with a target reconstruction loss including pixel-level distortion and perceptual loss between an output image and an input image.

[0009] Optionally, in an implementation according to the first aspect or any implementation thereof, the degradation-aware latent ^^^is determined for use as a condition when decoding the image from the set of basis elements ^^^.

[0010] Optionally, in an implementation according to the first aspect or any implementation thereof, the degradation-aware latent ^^^is determined for use as a condition when decoding the image from a fusion of the set of basis elements ^^^and the series of continuous real numbers ^^^.

[0011] A second aspect relates to a method implemented in a video decoding device, the method comprising: receiving a bitstream comprising an image coded as a set of code indices ^^^representing a set of basis elements Ŷ^^according to Learned Sparse Image Representation (LSVR) and as a series of continuous real numbers ^^^^according to Learned Continuous Visual Representation (LCVR); obtaining a degradation-aware latent, which approximates degradation thatAtty. Docket No.4502-83401 (6000689PCT02) occurred to the image during LCVR encoding; and decoding the image based on the set of basis elements Ŷ^^, the series of continuous real numbers ^^^^, and the degradation-aware latent.

[0012] Optionally, in an implementation according to the second aspect, the degradation-aware latent is denoted as degradation-aware latent ^^^and is obtained from the bitstream.

[0013] Optionally, in an implementation according to the second aspect or any implementation thereof, the degradation-aware latent ^^^is determined by a first neural network (NN) based on an LCVR-based decoded image x^^.

[0014] Optionally, in an implementation according to the second aspect or any implementation thereof, the degradation-aware latent ^^^is determined by a first NN based on the set of basis elements Ŷ^^and the series of continuous real numbers ^^^^.

[0015] Optionally, in an implementation according to the second aspect or any implementation thereof, applying, by a second NN, reverse diffusion to the degradation-aware latent ^^^to obtain a de-noised degradation-aware latent ^^^^.

[0016] Optionally, in an implementation according to the second aspect or any implementation thereof, applying reverse diffusion comprises determining a diffusion latent ^^^^from a LCVR-based decoded image ^^^and iteratively applying a noise to the degradation-aware latent ^^^based on the diffusion latent ^^^^to obtain a higher quality degradation-aware latent ^^^^.

[0017] Optionally, in an implementation according to the second aspect or any implementation thereof, applying reverse diffusion comprises determining a diffusion latent ^^^^from the series of continuous real numbers ^^^^and iteratively applying a noise to the degradation-aware latent ^^^based on the diffusion latent ^^^^to obtain a higher quality degradation-aware latent ^^^^.

[0018] Optionally, in an implementation according to the second aspect or any implementation thereof, the noise is a Gaussian noise.

[0019] Optionally, in an implementation according to the second aspect or any implementation thereof, decoding the image comprises applying the degradation aware latent to the series of continuous real numbers ^^^^to obtain a high quality series of continuous real numbers ^^ ^^^^ ^ andAtty. Docket No.4502-83401 (6000689PCT02) using the high quality series of continuous real numbers ^^ ^^^^ ^ as a condition for decoding the set of basis elements Ŷ^^.

[0020] Optionally, in an implementation according to the second aspect or any implementation thereof, decoding the image comprises fusing the series of continuous real numbers ^^^^and the basis elements Ŷ^^to determine a combined set of coded features Ŷ^^^and using the degradation aware latent as a condition for decoding the combined set of coded features Ŷ^^^.

[0021] A third aspect relates to an apparatus comprising: a memory or storage means configured to store instructions; and one or more processors or processing means coupled to the memory or the storage means and configured to execute the instructions to cause the apparatus to perform a method according to the first aspect or any implementation thereof.

[0022] A fourth aspect relates to an apparatus comprising: a memory or storage means configured to store instructions; and one or more processors or processing means coupled to the memory or the storage means and configured to execute the instructions to cause the apparatus to perform a method according the second aspect or any implementation thereof.

[0023] A fifth aspect relates to a computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computer-executable instructions when executed by one or more processors of an apparatus, cause the apparatus to perform a method according the first aspect, the second aspect, or any implementation thereof.

[0024] For clarity, any one of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create a new embodiment within the scope of the present disclosure.

[0025] These and other features, and the advantages thereof, will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.Atty. Docket No.4502-83401 (6000689PCT02) BRIEF DESCRIPTION OF DRAWINGS

[0026] For a more complete understanding of this disclosure, reference is now made to the following brief description, taken in connection with the accompanying drawings and detailed description, wherein like reference numerals between drawings represent like parts.

[0027] FIG.1 is a diagram illustrating an example framework for Learned Image Compression (LIC) using Learned Continuous Visual Representation (LCVR).

[0028] FIG. 2 is a diagram illustrating an example framework for LIC using Learned Sparse Image Representation (LSVR).

[0029] FIG.3 is a diagram illustrating an example framework for LIC using a hybrid LCVR and LSVR process.

[0030] FIG.4 is a diagram illustrating a first example of an improved framework for LIC using a hybrid LCVR and LSVR process.

[0031] FIG. 5 is a diagram illustrating a second example of an improved framework for LIC using a hybrid LCVR and LSVR process.

[0032] FIG.6 is a diagram illustrating a third example of an improved framework for LIC using a hybrid LCVR and LSVR process.

[0033] FIG.7 is a diagram illustrating a fourth example of an improved framework for LIC using a hybrid LCVR and LSVR process.

[0034] FIG. 8 is a diagram illustrating an example implementation of a codebook index and codebook retriever k modules.

[0035] FIG.9 is a diagram illustrating an example implementation of a decoder.

[0036] FIG.10 is a diagram illustrating an example implementation of a decoder.

[0037] FIG.11 is a diagram illustrating an example implementation of a reverse diffusion unit.

[0038] FIG.12 is a flowchart illustrating a processing workflow of an encoding device according to an embodiment of the present disclosure.

[0039] FIG.13 is a flowchart illustrating a processing workflow of a decoding device according to an embodiment of the present disclosure.Atty. Docket No.4502-83401 (6000689PCT02)

[0040] FIG.14 is a diagram illustrating an apparatus according to an embodiment of the present disclosure. DESCRIPTION OF EMBODIMENTS

[0041] It should be understood at the outset that, although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0042] LCVR and LSVR are example LIC methods that use a NN to code video data. LCVR and LSVR can be used alone or in combination. Example LCVR-based, LSVR-based, or hybrid LCVR and LSVR-based LIC methods all have difficulty in generating high-quality reconstruction with ultra-low bitrates. For example, when the bitrate is very low, LSVR-based approaches have poor fidelity with various artifacts, while LCVR-based approaches have poor perceptual quality and often poor fidelity too. Hybrid LCVR and LSVR-based LIC approaches cannot overcome both issues by simply trading off perceptual quality and fidelity, since the performance is bounded by the LCVR-based and LSVR-based LIC frameworks that have difficulty in the first place with ultra-low bitrates.

[0043] Disclosed herein are various systems and methods for encoding and decoding an image. For example, the disclose examples employ a hybrid LSVR and LCVR coding. The LSVR encoding can be decoded into an image high perception quality, but is often inaccurate (e.g., low fidelity). The LCVR can be decoded into an image that is generally accurate, but has low perception quality. Further, at very low bit rates, the LCVR encoding may result in an image that may also be inaccurate when decoded. The disclosed examples include a degradation-aware embedding unit that computes a degradation-aware latent ^^^. The degradation-aware embedding unit is an artificial intelligence (AI) based NN that is trained to determine the types of degradation that occur during the LCVR encoding process. The degradation-aware embedding unit can then use a decoded version of anAtty. Docket No.4502-83401 (6000689PCT02) encoded LCVR image to determine the degradation-aware latent ^^^, which approximates the degradation that occurs during the LCVR encoding process. A decoder can then use both the degradation-aware latent ^^^and the decoded LCVR image to guide the LSVR decoding process. In this way, the LSVR process can decode an LSVR image with high perception quality, while making decoding decisions based on both the degradation-aware latent ^^^and the decoded LCVR image to select decoding options that result in an LSVR image that is also accurate (e.g., high fidelity). In some examples, the degradation-aware embedding unit is implemented in the encoder and the degradation-aware latent ^^^is sent to the decoder. In other examples, the degradation-aware embedding unit is implemented in the decoder and the degradation-aware latent ^^^is calculated at the decoder to avoid transmitting the degradation-aware latent ^^^. In such examples, a reverse diffusion unit may also be implemented at the decoder. The reverse diffusion unit may use an AI NN that is trained on noise that is generally created during the LCVR encoding process, such as complicated Gaussian type noise. The reverse diffusion unit may apply an inverse of such noise to the degradation-aware latent ^^^calculated at the decoder in order to determine a more accurate and / or higher quality degradation-aware latent ^^^^.

[0044] FIG.1 is a diagram illustrating an example framework 100 for LIC using LCVR. LIC is a video coding approach that employs deep learning artificial intelligence models, such as neural networks (NNs) to encode video into a bitstream at an encoder and then decode the bitstream to recover the video at a decoder. LIC may be used as an alternative to hand-designed encoding algorithms such as Joint Photographic Experts Group (JPEG), High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC). LCVR is an example NN based LIC mechanism that uses AI to code video data. To perform LCVR, a NN is trained to code video data. The training data is formatted to encourage the NN to maximize compression while minimizing restoration loss between an encoded image and a corresponding decoded image.

[0045] For example, framework 100 comprises a sender and a receiver. The sender comprises an LCVR encoder 101 and a quantization and entropy encoder 103. The corresponding receiver comprises a dequantization and entropy decoder 105 and a decoder 107. The LCVR encoder 101 isAtty. Docket No.4502-83401 (6000689PCT02) an AI model trained to perform LCVR, which is a method of compressing an image by representing the image as a series of continuous real numbers, also referred to herein as an LCVR embedding feature. In framework 100, the LCVR encoder 101 receives video, denoted as X, and outputs a set of continuous real numbers denoted as ^^^. The quantization and entropy encoder 103 is either an AI model or a crafted algorithm configured to compress a series of real numbers into a data string known as a bitstream in a manner that can be efficiently stored and transmitted. Quantization is a process of mapping continuous values into a smaller set of discrete values. Quantization is a lossy process that rounds or truncates data, which reduces data size at the cost of data loss known as quantization error. Entropy coding is a lossless process that represents data as space efficient codes. Accordingly, the quantization and entropy encoder 103 reduces video data size by using quantization to remove some data precision and then further reduces data size by converting the quantized data into space efficient codes.

[0046] The receiver is designed to reconstruct video data from a bitstream. The receiver is configured to obtain video data from the bitstream. Specifically, the obtained video data should be as close to the originally encoded video data as possible. The dequantization and entropy decoder 105 is configured to reverse the process applied by the quantization and entropy encoder 103 at the sender. For example, the dequantization and entropy decoder 105 is configured to use entropy decoding to convert the space efficient codes in the bitstream back into quantized numbers. The dequantization and entropy decoder 105 then uses dequantization algorithms to attempt to add back the precision lost by the quantization process. Because quantization is an inherently non-invertible process, dequantization may employ statics, filters, and other mathematical algorithms to provide an educated guess as to what data was lost and attempt to add such data back to the quantized numbers and convert them into an approximation of the series of continuous real numbers, denoted as ^^^^. The decoder 107 is generally an AI based NN configured to reverse the coding process performed by the LCVR encoder 101. For example, the decoder 107 may be trained on the same or similar training data to as the LCVR encoder 101 and may be trained as part of the same training process as the LCVR encoder 101. The decoder 107 is configured to convert the continuous real numbers ^^^^Atty. Docket No.4502-83401 (6000689PCT02) back into video data, denoted as reconstructed output ^^. ^^ should be a similar to X as possible. However, some data loss is inherent and is largely unavoidable.

[0047] Accordingly, framework 100 operates by passing an input image x through an LCVR encoder 101 to generate an image embedding feature ^^^which comprises continuous real numbers. Then ^^^is further compressed through a quantization & entropy encoder 103 into a data string that is for storage and transmission. On the receiver side, a decoded image embedding feature ^^^^is recovered from the received string through a dequantization & entropy decoder 105. Then a decoder 107 reconstructs an output image ^^ based on the decoded image embedding feature ^^^^. The training target is to obtain the LCVR by minimizing the restoration loss between the reconstructed output ^^ and the original input x, and by minimizing the bits used to represent the image embedding feature ^^^for storage and transmission. One problem with LCVR-based LIC is the reconstruction quality is poor at low bitrates. For example, the output of LCVR-based LIC is usually overly smooth with severe degradation in fine details.

[0048] FIG.2 is a diagram illustrating an example framework 200 for LIC using LSVR. LSVR uses sparse representation learning for modelling generic images. For example, framework 200 comprises a sender and a receiver. The sender comprises an LSVR encoder 221, a codebook index 223, and an integer entropy encoder 225. The receiver comprises an integer entropy decoder 227, a codebook retriever 229, and a decoder 207.

[0049] The LSVR encoder 221 is an AI based NN that is configured for LIC. The LSVR encoder 221 is a vector-quantized autoencoder configured to perform LSVR by being trained with adversarial and perceptual loss in conjunction with a highly compressed codebook index 223 in the image. For example, LSVR may be used for LIC to pursue high perceptual quality in reconstruction. LSVR may include finding a compact and efficient representation of images by learning a dictionary of basis elements, also referred to herein as an LSVR-based embedding feature, and using a sparse combination of them to represent the image. For example, instead of using all the pixels in an image, a sparse representation may represent the image using only a few chosen basis elements, which can be learned from a dataset of images. The codebook index 223 is a table that maps the basis elementsAtty. Docket No.4502-83401 (6000689PCT02) to codewords. The integer entropy encoder 225 is similar to the quantization and entropy encoder 103 in framework 100. However, the integer entropy encoder 225 may not apply quantization as the codewords from the codebook index 223 are discrete values. Accordingly, the integer entropy encoder 225 is configured to use entropy coding on the codewords from the codebook index 223 to create a coded bitstream.

[0050] The integer entropy decoder 227 is a decoder that is configured to reverse the process of the integer entropy encoder 225. Accordingly, the integer entropy decoder 227 uses entropy decoding to convert the entropy coded bitstream back into codewords. The codebook retriever 229 is a table that is configured to convert the codewords back into basis elements. The decoder 207 is generally an AI based NN configured to reverse the coding process performed by the LSVR encoder 221. For example, the decoder 207 may be trained on the same or similar training data to as the LSVR encoder 221 and may be trained as part of the same training process as the LSVR encoder 221. As such, the decoder 207 is configured to convert the basis elements back into video images.

[0051] In summary, in framework 200 an input image, denoted as x, can be encoded into an image embedding feature ^^^by the LSVR encoder 221. Then ^^^is mapped into a sequence of code indices ^^^using a learned codebook in the codebook indexing 223. The code indices are integers, which can be effectively stored or transferred. ^^^can then be further compressed by the integer entropy encoder 225 to further reduce the bitrate in a lossless way. The receiver side can then recover the code indices ^^^by employing the integer entropy decoder 227. Then the same codebook can be used to obtain a decoded image embedding feature ^^^^from ^^^in the codebook retriever 229, for example by using the codewords in the codebook corresponding to the received code indices. A decoder 207 can then reconstruct an output ^^ from this decoded image embedding feature ^^^^. The learned codebook in the codebook index 223 and the codebook retriever 229 can be optimized to balance codebook efficiency and reconstruction quality.

[0052] An example problem with LSVR-based LIC is poor reconstruction fidelity. Since the learned codebook may be trained with high-quality images targeting at high-perceptual quality reconstruction, the output may look good to human eyes but may not match the original input, forAtty. Docket No.4502-83401 (6000689PCT02) example due to lacking fidelity details. Also, sometimes there are various artifacts in the reconstructed image, especially when input x has large degradations. Accordingly, the LSVR process from framework 200 may result in inaccurate reconstructed images while the LCVR process from framework 100 result in poor quality images.

[0053] FIG.3 is a diagram illustrating an example framework 300 for LIC using a hybrid LCVR and LSVR process. Framework 300 is a combination of framework 100 and framework 200. Framework 300 comprises a sender and a receiver. The sender comprises an LSVR encoder 221, a codebook index 223, an integer entropy encoder 225, an LCVR encoder 101, and a quantization and entropy encoder 103. The receiver comprises an integer entropy decoder 227, a codebook retriever 229, a dequantization and entropy decoder 105, and a decoder 207. The decoder 207 contains the functionality of both decoder 107 and decoder 207. Framework 300 allows each image in a video to be encoded by both LSVR and LCVR and then decoded by LSVR and LCVR. The decoder 207 can then decode by using the best results from both LSVR and LCVR.

[0054] This approach combines LCVR and LSVR for improved LIC with balanced perceptual quality and fidelity. As an example, an input image x can be encoded into an LSVR-based embedding feature ^^^by an LSVR encoder 221 and encoded into an LCVR-based embedding feature ^^^by an LCVR encoder 101. Then ^^^is mapped into a sequence of code indices ^^^using a learned codebook in a codebook index 223. ^^^can then be further compressed by the integer entropy encoder 225 to further reduce the bitrate in a lossless way. The receiver side recovers the code indices ^^^by using the integer entropy decoder 227. Then the same codebook is used to obtain a decoded LSVR-based embedding feature ^^^^from ^^^in the codebook retriever 229. At the same time ^^^is further compressed through the quantization and entropy encoder 103 into a data string that is efficient for storage and transmission. On the receiver side, a decoded LCVR-based embedding feature ^^^^is recovered from the received string through the dequantization and entropy decoder 105. The decoder 207 can then reconstruct an output ^^ by combining the decoded LSVR- based embedding feature ^^^^and the decoded LCVR-based embedding feature ^^^^, for example by using ^^^^as a condition of ^^^^for conditioned reconstruction.Atty. Docket No.4502-83401 (6000689PCT02)

[0055] The problem with the example in framework 300 is that LIC using hybrid LSVR and LCVR may perform poorly at ultra-low-bitrates. The decoded LCVR-based embedding feature ^^^^aims to provide fidelity information drawn from the current input x (e.g. to increase accuracy) to compensate for the poor accuracy in the decoded LSVR-based embedding feature ^^^^. However, when the bitrate is very low, the quality of ^^^^is too bad to provide good information for high-quality reconstruction.

[0056] This disclosure includes example general frameworks for ultra-low-bitrate LIC with High-Quality (HQ) reconstruction using hybrid LSVR and LCVR with high-fidelity guidance. For example, by using a joint embedding LCVR between an original HQ input and an ultra-low-bitrate low-quality (LQ) input, a high-fidelity-preserving signal can be obtained to recover a high-fidelity LCVR guidance on the receiver side, which can be combined with the LSVR to reconstruct an HQ output with both high perceptual quality and high fidelity. Furthermore, a diffusion model can be used to further reduce the bits to transfer the high-fidelity-preserving signal. The example frameworks are quite flexible to accommodate various ultra-low-bitrate compression needs, and this disclosure describes different configurations with and without the additional diffusion model, to deal with input images with different resolutions and qualities.

[0057] FIGs. 4-7 describe example embodiments of different configurations of the disclosed framework. The system configurations in FIGs.4-5 transfer the high-fidelity LCVR guidance signal, denoted as a degradation-aware latent ^^^, from the sender to the receiver. In comparison, the system configurations in FIGs. 6-7 compute a degradation-aware diffusion latent ^^^^in the receiver as an approximated version of the degradation-aware latent ^^^, without transferring the high-fidelity LCVR guidance signal.

[0058] FIG.4 is a diagram illustrating a first example of an improved framework 400 for LIC using a hybrid LCVR and LSVR process. For ease of understanding, the LSVR process of framework 400 is discussed first, then the LCVR process is discussed send hereinbelow.

[0059] Framework 400 comprises a sender and a receiver. The sender comprises a group of K LSVR encoders 421 and a group of K codebook indices 423, that are similar to LSVR encoder 221Atty. Docket No.4502-83401 (6000689PCT02) and codebook index 223, respectively, from framework 300. By having multiple LSVR encoders 421 and codebook indices 423, the results of each pair can be weighted to obtain a more optimal set of encoded video data. The LSVR encoders 421 encode the input video x to produce basis elements ^^^^ -^^^^ , which are converted to codewords ^^^^ -^^^^ by codebook indices 423. The sender also comprises a weight prediction 431. The weight prediction 431 is a software unit configured to determine what weighted combination of ^^^^ -^^^^ provides the best output. The determined weights are denoted as ^^^. The weights ^^^are encoded into a bitstream. The framework 400 further comprises an integer entropy encoder 425, which is substantially similar to integer entropy encoder 225 in framework 300. The integer entropy encoder 425 is configured to perform entropy encoding on the codewords ^^^^ -^^^^ and store the encoded codewords ^^^^ -^^^^ in the bitstream. By reading the bitstream, the decoder is able to decode the encoded codewords ^^^^ -^^^^ and weight the results using weights ^^^to obtain the LSVR encoded portion of the video.

[0060] The receiver includes an integer entropy decoder 427, which is substantially similar to integer entropy decoder 227 from framework 300. The integer entropy decoder 427 is configured toemploy entropy decoding to obtain the encoded codewords ^^ ^^^ - ^^^ . The receiver furthercomprises K codebook retrievers 429, which are substantially similar to codebook retriever 229 in framework 300. The codebook retrievers 429 convert the encoded codewords ^^^^ -^^^^ into basis elements ^^^^ -^^^^ . The receiver further comprises a fusion 432, which is a unit configuredto fuse the basis elements ^^^^ -^^^^ into a single set of elements Ŷ^^by weighting each set of basis elements using weights ^^^. Framework 400 further comprises a decoder 407, which is substantially similar to decoder 207 from framework 300. The basis elements Ŷ^^are forwarded to decoder 407 to be decoded.

[0061] The LCVR process is now discussed. The sender a fidelity preserving LCVR encoder 401 and a fidelity preserving LCVR decoder 402, which are similar to LCVR encoder 101 and decoder 107, respectively, from framework 100. The fidelity preserving LCVR encoder 401 is configured to encode the input video x according to LCVR to output a set of continuous real numbers ^^^. The set of continuous real numbers ^^^is then forwarded to the fidelity preservingAtty. Docket No.4502-83401 (6000689PCT02) LCVR decoder 402, which decodes the set of continuous real^^^to obtain a reconstructed version of the input video x, denoted as LCVR-based decoded image x^^.

[0062] The sender also comprises a degradation-aware embedding 409, which is a software unit configured to compute a degradation-aware latent ^^^based on the LCVR-based decoded image x^^. For example, the degradation-aware embedding 409 may be a NN that is trained on degradation caused by the LCVR process. Accordingly, the degradation-aware embedding 409 is generally aware of the types of degradation that occurs when LCVR is applied to an image. The LCVR-based decoded image x^^approximates the image the receiver will decode as part of the LCVR decoding process upon receiving the bitstream. As such, the degradation-aware embedding 409 is configuredto review the LCVR-based decoded image x^ ^ , determine the kinds of degradation that likelyoccurred during the LCVR encoding process, and provide an estimate of the degradation that occurred to the input image (e.g., input video x ) when the input image was encoded by the fidelity preserving LCVR encoder 401. The estimate of the estimate of the degradation that occurred to the input image when the input image was encoded by the fidelity preserving LCVR encoder 401 is referred to herein as the degradation-aware latent ^^^.

[0063] The sender also comprises a quantization and entropy encoder 403 and an entropy encoder 411. The quantization and entropy encoder 403 is substantially similar to quantization and entropy encoder 103 in framework 300. The entropy encoder 411 is substantially similar to quantization and entropy encoder 403, but may not comprise a quantization function. The quantization and entropy encoder 403 is configured to receive the set of continuous real numbers ^^^from the fidelity preserving LCVR encoder 401 and encode them into the bitstream in a manner that is substantially similar to the quantization and entropy encoder 103 in framework 300. The entropy encoder 411 is configured to apply entropy encoding to the degradation-aware latent ^^^.

[0064] The receiver further comprises a dequantization and entropy decoder 408 and an entropy decoder 406. The dequantization and entropy decoder 408 is substantially similar to dequantization and entropy decoder 105 in framework 300. The dequantization and entropy decoder 408 is configured to decode the bitstream using entropy decoding and dequantize the result to convert theAtty. Docket No.4502-83401 (6000689PCT02) space efficient codes in the bitstream back into series of continuous real numbers ^^^^and send the series of continuous real numbers ^^^^to the decoder 407. The entropy decoder 406 is substantially similar to the dequantization and entropy decoder 408, but may not contain a dequantization function. The entropy decoder 406 is configured to decode the bitstream to obtain the degradation- aware latent ^^^, which is also forwarded to the decoder 407.

[0065] The decoder 407 is configured to receive the basis , the series of continuous l numbers ^^, and the d^rea^^egradation-aware latent ^^. The decoder is an AI based NN trained in both LSVR and LCVR. The decoder is configured to use the degradation-aware latent ^^^and the series of continuous real numbers ^^^^can then be used separate conditions when decoding thebasis elements Ŷ^^ to obtain a reconstructed output image ^^ . This results in a more accuratereconstructed output ^^. For example, the decoder 407 may decode a weighted fusion of the basis elements Ŷ^^from the LSVR process, which results in an image with high perception quality. The decoder can also use the series of continuous real numbers ^^^^and the degradation-aware latent ^^^when decoding the basis elements Ŷ^^to select a decoding of the basis elements Ŷ^^that is high fidelity, and therefore accurately represents the original input image x.

[0066] FIG.5 is a diagram illustrating a second example of an improved framework 500 for LIC using a hybrid LCVR and LSVR process. Framework 500 is similar to framework 400. The LSVR process for framework 500 is substantially similar to the LSVR process for framework 400. The LCVR process for framework 500 is also similar to the LCVR process for framework 400. The difference is that framework 500 does not comprise a fidelity preserving LCVR decoder 402. In framework 500, the set of continuous real numbers ^^^output by the fidelity preserving LCVR encoder 401 are forwarded directly to the degradation-aware embedding 409. The degradation- aware embedding 409 then determines the degradation-aware latent ^^^based on the set of continuous real numbers ^^^instead of based on the LCVR-based decoded image x^^as in framework 400. Also, the set of continuous real numbers ^^^is forwarded as input into the LSVR encoders 421. This allows the LSVR encoders 421 to produce more accurate basis elements ^^^^ -^^^^ during theAtty. Docket No.4502-83401 (6000689PCT02) LSVR encoding process. The receiver side of framework 500 is substantially similar to the receiver side of framework 400.

[0067] As such, for the system configurations FIGs.4-5, there are two example embodiments of the system framework, denoted as framework 400 and framework 500, respectively. For both embodiments, the system is provided with an input image x, where x is a general three dimensional(3D) tensor with shape ^ × ℎ × ^, where w, h, c are the width, height, and number of channels ofthe image. For example, c = 3 for color images, c = 1 for spectral images, or c = 4 for RGB-D(color and depth) images. The input x is first fed into A set of # (# ≥ 1) modules, LSVR Encoding1, …, LSVR encoding K to generate a set of K LSVR-based embedding feature ^^ ^^^ , ⋯ , ^^^ . Each^^^) is a tensor with shape ^) × ℎ) × *) where the width and height ^) and ℎ) depend on the inputwidth and height as well as the network structure of the LSVR Encoding k module, and *)is the number of feature channels. Various neural networks can be used as the LSVR encoding k module, including Visual Transformers (ViT) or Convolutional Neural Networks (CNN). Then each LSVR- based embedding feature ^^^) is further processed by a Codebook Indexing k module to compute a sparse codebook-based latent feature ^^^) .

[0068] After that, the decoded LSVR-based embedding features ^^^ ^^^ , ⋯ , ^^^^ are fed into afusion module to compute a comb decoded LSVR-based^^^. There are manyways to generate ^^^ ^ ^ ^ ^ ^^ from ^^^^ , ⋯ , ^^^^ . For example, when K=1, ^^^ = ^^^^ . When K>1, ^^^ can be aconcatenation of ^^^^^ , ⋯ , ^^^^^ if *^ = ⋯ = *), or can be concatenation of a transferred version of^^^^^ , ⋯ , ^^^^^ where each ^^^^^ is first transferred through an NN to have aligned feature dimension. Inone embodiment, an additional weight ^^^is computed in the sender in a weight prediction modulebased on the LSVR-based embedding feature ^^ , ⋯ , ^^ . T ^^^ ^^ he weight ^^ of shape ^) × ℎ) × # istransferred (often after further compression to reduce the amount of bits to transfer) to the receiver,and is used by the Fusion module to weighted combine ^^^ ^ ^^^ , ⋯ , ^^^^ into ^^^:^^^^,+ = ∑^ )-^ ^^ ^,+^^^^),+for each “super-pixel” ^^^^,+ (. = 1, … , ^) × ℎ))

[0069] The and data flows are marked as dotted lines in FIGs.4-5.Atty. Docket No.4502-83401 (6000689PCT02)

[0070] Parallel with the LSVR processing pipeline, the input x is also encoded by a fidelity- preserving LCVR encoding module to generate an LCVR-based embedding feature ^^^. ^^^is further processed by a quantization and entropy encoding module to compute a bitstream which is transferred to the receiver. The receiver computes a decoded LCVR-based embedding feature ^^^^through a dequantization and entropy decoding module based on the received bitstream. Any LCVR- based image compression methods can be used for the LCVR process.

[0071] For the system framework described in FIG.4, in the sender, a fidelity-preserving LCVR decoding module is used to compute an LCVR-based decoded image ^^^from ^^^. Then a degradation-aware embedding module is used to compute the degradation-aware latent ^^^based on the original input x and the LCVR-based decoded image ^^^. ^^^may be a feature vector with d (a few hundred) dimensions, which is further processed by an entropy encoding module to reduce the transferring bits in a lossless way and is transmitted to the receiver. The receiver then recovers ^^^through an entropy decoding module.

[0072] For the system framework described in FIG. 5, in the sender, the degradation-aware embedding module computes the degradation-aware latent ^^^based on the LCVR-basedembedding feature ^^ and the LSVR-based embedding f ^ ^ ^^ eature ^^^ , ⋯ , ^^^ . As in FIG. 4, ^^usually is a feature vector with d (a few hundred) dimensions, which is further processed by an entropy encoding module to reduce the transferring bits in a lossless way and is transmitted to the receiver. The receiver then recovers ^^^through an entropy decoding module.

[0073] Finally, in both FIGs. 4-5, a decoder reconstructs the output image ^^ based on thecombined decoded LSVR-based embedding feature ^^^^, the decoded LCVR-based embedding feature ^^^^, and the degradation-aware latent ^^^. There are multiple ways for the decoder to combine the LSVR-based and LCVR-based features as well as the fidelity-preserving guidance to reconstruct the output.

[0074] FIG.6 is a diagram illustrating a third example of an improved framework 600 for LIC using a hybrid LCVR and LSVR process. Framework 600 is similar to framework 400, but framework 600 is configured to calculate the degradation-aware latent ^^^at the receiver side toAtty. Docket No.4502-83401 (6000689PCT02) avoid having to send the degradation-aware latent ^^^in the bitstream. The LSVR process in framework 600 is substantially similar to the LSVR process in framework 400. Regarding the LCVR process, the sender in framework 600 includes only the fidelity preserving LCVR encoder 401 and the quantization and entropy encoder 403. The sender in framework 600 does not include the fidelity preserving LCVR decoder 402, the degradation-aware embedding 409, or the entropy encoder 411. In this way, the sender in framework 600 encodes the set of continuous real numbers ^^^using the fidelity preserving LCVR encoder 401 and then quantizes and encodes the result in the bitstream using the quantization and entropy encoder 403.

[0075] Instead, the degradation-aware embedding 409 is implemented at the receiver. Specifically, the receiver comprises a fidelity preserving LCVR decoder 402, a degradation-aware embedding 409, and a reverse diffusion 610. The receiver employs the dequantization and entropy decoder 408 to receive, dequantize, and entropy decode the bitstream to obtain the series of continuous real numbers ^^^^. The series of continuous real numbers ^^^^is then sent to the decoder 407. The series of continuous real numbers ^^^^is also sent to the fidelity preserving LCVR decoder 402, which decodes the series of continuous real numbers ^^^^to obtain a LCVR-based decoded image x^^.

[0076] The LCVR-based decoded image x^^is then forwarded to the degradation-aware embedding 409 and the reverse diffusion 610. The degradation-aware embedding 409 at the decoder then determines the degradation-aware latent ^^^based on the LCVR-based decoded image x^^as in framework 400. The degradation -aware latent ^^^is also forwarded to the reverse diffusion 610.

[0077] The reverse diffusion 610 is an AI NN that is trained on noise that is generally created during the LCVR encoding process. Such noise is generally a complicated form of Gaussian noise. The reverse diffusion 610 uses the LCVR-based decoded image x^^to determine an approximation of the noise added during the LCVR encoding process. The reverse diffusion 610 then determines an inverse of the noise and applies the inverse noise to the degradation-aware latent ^^^in order to determine a de-noised degradation-aware latent ^^^^. The de-noised degradation-aware latent ^^^^isAtty. Docket No.4502-83401 (6000689PCT02) then forwarded to the decoder 407 for use, along with the series of continuous real numbers ^^^^, to decode the basis elements in a more accurate manner.

[0078] FIG.7 is a diagram illustrating a fourth example of an improved framework 700 for LIC using a hybrid LCVR and LSVR process. Framework 700 is similar to framework 600. For example, framework 700 is another example that calculates the degradation-aware latent ^^^at the receiver instead of coding the degradation-aware latent ^^^in the bitstream. The LSVR process in framework 700 is substantially similar to the LSVR process in framework 600. Further, the sender side of the LCVR process is substantially similar in framework 600 and framework 700. On the receiver side of the LCVR process, the receiver includes a dequantization and entropy decoder 408, a degradation-aware embedding 409, and the reverse diffusion 610, but does not include a fidelity preserving LCVR decoder 402.

[0079] The dequantization and entropy decoder 408 receives, dequantizes, and entropy decodes the bitstream to obtain the series of continuous real numbers ^^^^. The series of continuous real numbers ^^^^are then forwarded directly to the degradation-aware embedding 409, to the reverse diffusion 610, and the decoder 407. The weighted fused set of basis elements Ŷ^^is also forwarded to the degradation-aware embedding 409. The degradation-aware embedding 409 then uses the weighted fused set of basis elements Ŷ^^and the series of continuous real numbers ^^^^to determine the degradation-aware latent ^^^. The degradation-aware latent ^^^is then to the reversediffusion 610 for calculation of the de-noised degradation-aware latent ^^^^in a manner that is substantially similar to framework 600.

[0080] Accordingly, for both embodiments in FIGs.6-7, the system is provided with an inputimage x, where x is a general 3D tensor with shape ^ × ℎ × ^, where w, h, c are the width, height,and number of channels of the image. For example, c = 3 for color images, c = 1 for spectralimages, or c = 4 for RGB-D (color and depth) images. The input x is first fed into A set of # (# ≥1) modules, LSVR encoding 1, …, LSVR encoding K to generate a set of K LSVR-basedembedding features ^^ , ⋯ , ^^ . ^^^ ^^ Each ^^) is a tensor with shape ^) × ℎ) × *) where the widthand height ^)and ℎ)depend on the input width and height as well as the network structure of theAtty. Docket No.4502-83401 (6000689PCT02) LSVR encoding k module, and *)is the number of feature channels. Various neural networks can be used as the LSVR encoding k module, including Visual Transformers (ViT) or Convolutional Neural Networks (CNN). Then each LSVR-based embedding feature ^^^) is further processed by a codebook indexing k module to compute a sparse codebook-based latent feature ^^^) . The detailed processing workflow of the codebook indexing k module is described in the sender part of FIG.8 below. A code generation k module computes the sparse codebook-based latent feature ^^^) based on the feature tensor ^^^) and a basis codebook 0). The basis codebook 0)comprises of 2)of codewords, each having *)dimensions. Each “super-pixel'' ^^^^),+in ^^) (. = 1, … , ^) × ℎ)) is thecodeword index of a codeword ^),+ ∈ 0) that is nearest to the corresponding latent feature ^^^),+: ^),+ = 456789^:,;∈ ^: <8=>(^),? , ^^^),+), where <8=>() is a distanceis, the entire ^^^) has ^) × ℎ)integers corresponding to the indices of ^) × ℎ) codewords, which can be efficiently transmitted tothe receiver with very little bit consumption.

[0081] Usually the sparse codebook-based latent features ^^ ^^^ , ⋯ , ^^^ are further processed byan integer entropy encoding module in a lossless way to further reduce the transmission bits, and the receiver process the received bits by an integer entropy decoding module to recover the sparsecodebook-based latent features ^^ ^^^ , ⋯ , ^^^ . Then on the receiver side, each sparse codebook-basedlatent features ^^^) is further processed by a codebook retrieval k module to generate a decoded LSVR-based embedding feature ^^^^) . The detailed processing workflow of the Codebook Retrieval k module is described in the receiver part of FIG. 8 below. The received ^^^) is fed into a feature retrieval k module to retrieve the decoded LSVR-based embedding feature ^^^^) of shape^) × ℎ) × *) based on the same basis codebook 0) as the sender. For example, each “super-pixel”^^^^),+ (. = 1, … , ^) × ℎ)) is the codeword with index ^^^),+. After the decoded LSVR-based embedding features ^^^ ^^^ , ⋯ , ^^^^ are fed into aa combined decoded LSVR-based embedding feature ^^^^. There are manyways to generate ^^^ f ^ ^ ^ ^ ^^ rom ^^^^ , ⋯ , ^^^^ . For example, when K=1, ^^^ = ^^^^ . When K>1, ^^^ can be aAtty. Docket No.4502-83401 (6000689PCT02)concatenation of ^^^ , ⋯ , ^^^^^ ^^ if *^ = ⋯ = *), or can be concatenation of a transferred version of^^^ , ^ ^^^ ⋯ , ^^^^ where each ^^^^ is first transferred through an NN to have aligned feature dimension. Inone embodiment, an additional weight ^^^is computed in the sender in a weight prediction modulebased on the LSVR-based embedding feature ^^ ^ ^^^ , ⋯ , ^^^ . The weight ^^ of shape ^) × ℎ) × # istransferred (often after further compression to reduce the amount of bits to transfer) to the receiver,and is used by the fusion module to weighted combine ^^^ , ⋯ ^ ^^^ , ^^^^ into ^^^:^^^ = ∑^ -^ ^^ ^^^for each “super-pixel” ^^^^,+ (. = 1, … , ^) × ℎ))

[0083] and data flows are marked as dotted lines FIGs.6-7.

[0084] Parallel with the LSVR processing pipeline, the input x is also encoded by a fidelity- preserving LCVR encoding module to generate an LCVR-based embedding feature ^^^. ^^^is further processed by a quantization and entropy encoding module to compute a bitstream which is transferred to the receiver. The receiver computes a decoded LCVR-based embedding feature ^^^^through a dequantization and entropy decoding module based on the received bitstream. Any LCVR-based image compression methods can be used for this purpose.

[0085] For the system framework described in FIG. 6, in the receiver, a fidelity-preserving LCVR decoding module is used to compute an LCVR-based decoded image ^^^from ^^^^. Then a degradation-aware embedding module is used to compute the degradation-aware latent ^^^based on the LCVR-based decoded image ^^^. Comparing with the corresponding degradation-aware latent ^^^in FIGs.4-5 that is computed in the sender and sent to the receiver, this degradation-aware latent ^^^is a degraded version computed from the LCVR-based decoded image ^^^instead of the original input x. Then a reverse diffusion process is used to recover a restored degradation-aware latent ^^^^based on the degradation-aware latent ^^^and the LCVR-based decoded image ^^^. ^^^^has high fidelity information to improve the final reconstruction.

[0086] For the system framework 700 described in FIG.7, in the receiver, the degradation-aware embedding module computes the degradation-aware latent ^^^based on the decoded LCVR-based embedding feature ^^^^and the combined decoded LSVR-based embedding feature ^^^^. Then, like inAtty. Docket No.4502-83401 (6000689PCT02) FIG.6, a reverse diffusion process is used to recover a restored degradation-aware latent ^^^^based on the degradation-aware latent ^^^and the decoded LCVR-based embedding feature ^^^^.

[0087] Finally, in both FIG.6 and FIG.7, a decoder reconstructs the output image ^^ based on the combined decoded LSVR-based embedding feature ^^^^, the decoded LCVR-based embedding feature ^^^^, and the restored degradation-aware latent ^^^^.

[0088] There are multiple ways for the decoder to combine the LSVR-based and LCVR-based features as well as the fidelity-preserving guidance to reconstruct the output as discussed with respect to FIGs.9-10 below.

[0089] FIG.8 is a diagram illustrating an example implementation 800 of a codebook index and codebook retriever k modules. Implementation 800 can be used in any of frameworks 400-700. Implementation 800 contains a sender with a codebook index k 823 and an integer entropy encoder 425 and a receiver with an integer entropy decoder 427 and a codebook retriever k 829. For example, the codebook index k 823 and codebook retriever k 829 may be used to implement the codebook index k 423 and codebook retriever k 429, respectively, in any of FIGs. 4-7. Further, the integer entropy encoder 425 and the integer entropy decoder 427 are substantially the same as similar components used in FIGs.4-7.

[0090] The code generation k unit 832 is configured to receive a series of basis elements from an LSVR encoder, where the basis elements represent an input image. The code generation k unit 832 then uses the codebook index k 823 to determine indices of codewords that compactly represent the basis elements. The indices of codewords may be referred to as a sparse codebook-based latent feature ^^^) . For example, the codebook index k 823 comprises a basis codebook 0)831 and a code generation k unit 832. The code generation k unit 832 is configured to compute a sparse codebook- based latent feature ^^^) based on a feature tensor ^^^) from an LSVR encoder and the basis codebook 0)831. The basis codebook 0)831 comprises 2)of codewords, each having *)dimensions. Each super-pixel ^^^^),+in ^^) (. = 1, … , ^) × ℎ)) is a codeword index of a codeword ^),+ ∈ 0) that isnearest to the corresponding latent feature ^^^),+: ^),+ = 456789^:,;∈ ^: <8=>(^),? , ^^^),+),Atty. Docket No.4502-83401 (6000689PCT02) where <8=>() is a distance metric, such as L1 or L2 norm. That is, the entire ^^^) has ^) × ℎ)integers corresponding to the indices of ^) × ℎ) codewords, which can be efficiently transmitted tothe receiver with very little bit consumption.

[0091] The sparse codebook-based latent features ^^ ^^^ , ⋯ , ^^^ can then be further processed byan integer entropy encoder 425 in a lossless way to further reduce the transmission bits included in the bitstream. The receiver can process the received bits by an integer entropy decoder 427 to recoverthe sparse codebook-based latent features ^^^^ , ⋯ , ^^^^ . Then on the receiver side, each sparsecodebook-based latent features ^^^) is further processed by the codebook retriever k 829 to generate a decoded LSVR-based embedding feature ^^^^) . For example, codebook retriever k 829 may comprise a basis codebook 0)833 that is substantially similar to basis codebook 0)831. The codebook retriever k 829 may also comprise a feature retriever k 384 that is configured to reverse the process of the code generation k unit 832 by using the basis codebook 0)833 to look up the basis elements based on the received codeword indices. Accordingly, the received codewords ^^^) are fed into the feature retriever k 829 to retrieve the decoded LSVR-based embedding feature ^^^^) of shape^) × ℎ) × *) based on the same basis codebook 0) 833 as the sender. For example, each super-pixel ^^^^),+ (. = 1, … , ^) × ℎ)) is the codeword with index ^^^),+.

[0092] FIG. 9 is a diagram illustrating an example implementation of a decoder 900. For example, decoder 900 may be employed as a decoder 407. The decoder 900 comprises an LCVR- based restoration 941 and a conditioned generation 943. The LCVR-based restoration 941 is configured to receive a signaled degradation-aware latent ^^^or a computed degradation-aware latent ^^^^with an accuracy that has been increased via reverse diffusion. The LCVR-based restoration 941 also receives the series of continuous real numbers ^^^^encoded by the LCVR process. The LCVR-based restoration 941 is configured to use the degradation-aware latent ^^^or the computed degradation-aware latent ^^^^to reverse the degradation applied to the series of continuous real numbers ^^^^during the LCVR encoding process. The result of this process can be denoted as ^^^^^^ ^ . The conditioned generation 943 is configured to receive and decode the basis elements Ŷ^^based on a series of conditions. For example, the conditioned generation 943 isAtty. Docket No.4502-83401 (6000689PCT02) configured to use the continuous real numbers corrected by the degradation aware latent ^^^^^^ ^ as a condition to decode the basis elements. This results in causing the basis elements Ŷ^^to be decoded in a manner that more accurately represents the originally encoded image.

[0093] Accordingly, the decoded LCVR-based embedding feature ^^^^and the signaled degradation-aware latent ^^^or the restored degradation-aware latent ^^^^are first fed into an LCVR- based restoration 941 to generate an high quality (HQ) LCVR-based embedding feature ^^ ^^^^ ^ .Compared with the decoded LCVR-based embedding feature ^^^^, ^^ ^^^^ ^ has high fidelity information recovered from the restored degradation-aware latent ^^^^. Then the HQ LCVR-based embedding feature ^^ ^^^^ ^ and the combined decoded LSVR-based embedding feature ^^^^are used by the conditioned generation 943 to compute the output ^^. The LCVR-based restoration 941 is an NN that can employ multiple types of network structures. For example, a dynamic image resolution (IR) transformer can be used. The conditioned generation 943 is also an NN that can employ multiple types of network structures. For example, conditioned generation 943 can be the decoder part of a Variational AutoEncoder (VAE), which takes as input the combined feature of ^^ ^^^^ ^ and ^^^^. There are multiple ways to combine ^^ ^^^^ and ^^^, for example by ^^^^ ^^concatenation or by using ^^^to modulate ^^^^and tune ^^^^through an affine transformation.

[0094] FIG. 10 is a diagram illustrating an example implementation of a decoder 1000. For example, decoder 1000 may be employed as a decoder 407. The decoder 1000 comprises an LSVR- LCVR fusion 1042 is configured to receive and fuse the series of continuous real numbers ^^^^and the basis elements Ŷ^^to determine a combined set of coded features Ŷ^^^. The decoder 1000 also comprises conditioned generation 1043, which is substantially similar to conditioned generation 1043. The conditioned generation 1043 is configured to receive the combined set of coded features Ŷ^^^and decode the combined set of coded features Ŷ^^^by using the signaled degradation-aware latent ^^^or the computed degradation-aware latent ^^^^as an condition. This results in causing theAtty. Docket No.4502-83401 (6000689PCT02) combined set of coded features Ŷ^^^to be decoded in a manner that more accurately represents the originally encoded image.

[0095] Accordingly, the decoded LCVR-based embedding feature ^^^^and the decoded LSVR- based embedding feature ^^^^are first combined in an LSVR-LCVR fusion 1042 to generate an LSVR-LCVR combined embedding feature ^^^^^. Then ^^^^^and the restored degradation-aware latent ^^^^or the signaled degradation-aware latent ^^^are used by the conditioned generation 1043 to compute the output ^^. The LSVR-LCVR fusion 1042 is an NN that can employ multiple types of network structures. For example, the LSVR-LCVR fusion 1042 can be a modulation network that uses ^^^^to modulate ^^^^and tune ^^^^through an affine transformation into ^^^^^. The LSVR-LCVR fusion 1042 can also be a transformer with cross attention modules that bring ^^^^and ^^^^into a joint attention space. As with FIG.9, the conditioned generation 1043 is an NN that can employ multiple network structures.

[0096] FIG.11 is a diagram illustrating an example implementation of a reverse diffusion unit 1100, which may be used to implement a reverse diffusion 610 in FIG.6 or FIG.7. As noted above, the reverse diffusion unit 1100 is configured determine noise applied to image data during the LCVR encoding process and apply a reverse of such noise to create more accurate image data for decoding. The reverse diffusion unit 1100 comprises a diffusion embedding 1145 and a reverse prediction 1147. Depending on the example, the diffusion embedding 1145 is configured to receive either a series of continuous real numbers ^^^^or a LCVR-based decoded image x^^. The diffusion embedding 1145 is then configured to determine a diffusion latent ^^^^the represents noise applied to the image during the LCVR encoding process. The reverse prediction 1147 is configured to receive the diffusion latent ^^^^and the computed degradation-aware latent ^^^, and use an iterative process to apply a reverse of the noise applied to the image in order to generate a higher quality degradation-aware latent ^^^^.

[0097] For example, given the degradation-aware latent ^^^, the reverse diffusion unit 1100 uses a diffusion process such as the Latent Diffusion Model to compute the restored degradation-aware latent ^^^^. The diffusion embedding 1145 first transforms the decoded LCVR-based embeddingAtty. Docket No.4502-83401 (6000689PCT02) feature ^^^^or the LCVR-based decoded image ^^^into a diffusion latent ^^^^. Then the reverseprediction 1147 computes a reverse diffusion step ^^^F) . A total of Titerations are taken, and t=1,…,T, and the first ^^^^,Gis initialized as a random noise like Gaussian noise. T can be preset, or can be determined for each input x. T can be determined on the receiver side or can be determined on the sender side and sent to the receiver side. Then after H iterations, ^^^^,Qis output as the restored degradation-aware latent ^^^^. The reverse prediction 1147 is usually an NN that can employ any network structures, such as convolutional neural networks or Transformers.

[0098] The different components in the frameworks discussed herein can be trained altogether or piece by piece. This disclosure does not put any restriction on the network architectures of various components or the training methods of the components. An example training process for use in training the components discussed herein is included below.

[0099] Stage 1.1: In the example embodiments of FIGs. 4-8, the LSVR encoder k, the basiscodebook 0), the code generation k, the feature retriever k, for all R = 1, … , #, the integer entropyencoder, the integer entropy decoder, the fusion, the decoder, and theunits can be trained first based on a large set of images with high qualities and high resolutions. In this stage, the decoder uses unconditioned reconstruction without using ^^^^and ^^^^or ^^^. The training target is to minimize the reconstruction loss such as pixel-level distortion and perceptual loss between the output ^^ and the original input x, and other codebook loss, semantic loss, etc.

[0100] Stage 1.2: For the system configuration described in FIG. 4 and FIG. 6, the fidelity- preserving LCVR encoder, the fidelity-preserving LCVR decoder, the quantization and entropy encoder, and the dequantization and entropy decoder are trained using a large set of images with various qualities and resolutions. The training target is to minimize the reconstruction loss such as pixel-level distortion and perceptual loss between the output ^^^and the original input x, as well as bitrate loss. For the system described in FIG. 5 and FIG.7, the trained parameters of the fidelity- preserving LCVR encoder, the quantization and entropy encoder, and the dequantization and entropy decoder are copied to the corresponding units.Atty. Docket No.4502-83401 (6000689PCT02)

[0101] Stage 2: The units trained from Stage 1 are kept frozen, except for the decoder. Then the degradation-aware embedding, the entropy encoder, the entropy decoder, and the decoder are trained by using a large set of images with various qualities and resolutions. In this stage, the decoder uses the final conditioned generation process with ^^^^and ^^^^or ^^^. The training target is to minimize the reconstruction loss such as pixel-level distortion and perceptual loss between output ^^ and input x.

[0102] Stage 3: The units trained from Stage 2 are kept frozen, except for the degradation-aware embedding. Then the degradation-aware embedding and the reverse diffusion are trained by using a large set of images with various qualities and resolutions. The training target is to minimize the reconstruction loss such as pixel-level distortion and perceptual loss between output ^^ and input x, as well as diffusion loss.

[0103] The disclosed frameworks achieve ultra-low-bitrate image compression with high- quality reconstruction at the same time. By using a joint embedding LCVR between the original HQ input and the ultra-low-bitrate low-quality (LQ) input, a high-fidelity-preserving signal can be obtained to recover a high-fidelity LCVR guidance on the receiver side, which can be combined with the LSVR to reconstruct an HQ output with both high perceptual quality and high fidelity. The disclosed includes two different configurations of the system, where the diffusion model can be used for further bit reduction. Depending on the compression needs of each individual data, the system can be flexibly configured to accommodate input images with different qualities and resolutions.

[0104] FIG. 12 is a flowchart illustrating a processing workflow 1200 of an encoding device according to an embodiment of the present disclosure. The processing workflow 1200 is an example of a method performed by a video encoder or a video encoding device according to an embodiment of the present disclosure. The encoder, at step 1202, encodes an image into a bitstream as a set of code indices ^^^representing a set of basis elements ^^^according to LSVR. The encoder, at step 1204, encodes the image into the bitstream as a series of continuous real numbers ^^^according to LCVR. At low bitrates, the LSVR encoding can be decoded with a high perception quality and low accuracy / fidelity and the LCVR encoding can be decoded with low perception quality and lowAtty. Docket No.4502-83401 (6000689PCT02) accuracy / fidelity. Accordingly, the encoder, at step 1206, determines, via a NN, a degradation-aware latent ^^^, which approximates degradation that occurs to the image during LCVR encoding. The degradation-aware latent ^^^can be used to increase the accuracy / fidelity of the LCVR encoding. In this way, the degradation-aware latent ^^^and the LCVR encoding can be used as conditions when decoding the LSVR encoding, which results in a decoded image with both high perception quality and high accuracy / fidelity. The encoder, at step 1208, encodes the degradation-aware latent ^^^into the bitstream. The encoder, at step 1210, then stores the bitstream. Additional details of the particular steps of the processing workflow 1200 are described above in the present disclosure.

[0105] FIG. 13 is a flowchart illustrating a processing workflow 1300 of a decoding device according to an embodiment of the present disclosure. The processing workflow 1300 is an example of a method performed by a video decoder or a video decoding device according to an embodiment of the present disclosure. The decoder, at step 1302, receives a bitstream comprising an image coded as a set of code indices ^^^representing a set of basis elements Ŷ^^according to LSVR and as a series of continuous real numbers ^^^^according to LCVR. The decoder, at step 1304, obtains a degradation-aware latent, which approximates degradation that occurred to the image during LCVR encoding. In some examples, the degradation-aware latent may be obtained from the bitstream by entropy decoding, in which case the degradation-aware latent is denoted as degradation-aware latent ^^^. In other examples, the degradation-aware latent may be computed at the decoder. In such cases, the encoder may optionally, at step 1306, apply, by a second NN, reverse diffusion to the degradation-aware latent ^^^to obtain a de-noised degradation-aware latent ^^^^. The encoder may then, at step 1308, decode the image based on the set of basis elements Ŷ^^, the series of continuous real numbers ^^^^, and the degradation-aware latent. In some examples, decoding the image may include applying the degradation aware latent to the series of continuous real numbers ^^^^to obtain a high quality series of continuous real numbers ^^ ^^^^ ^ and using the high quality series of continuous real numbers ^^ ^^^^ ^ as a condition for decoding the set of basis elements Ŷ^^. In some examples, decoding the image may include fusing the series of continuous real numbers ^^^^and the basis elements Ŷ^^to determine a combined set of coded features Ŷ^^^and using the degradationAtty. Docket No.4502-83401 (6000689PCT02) aware latent as a condition for decoding the combined set of coded Additional detailsof the particular steps of the processing workflow 1300 are described above in the present disclosure.

[0106] FIG.14 is a diagram illustrating an apparatus 1400 according to an embodiment of the present disclosure. The apparatus 1400 can be used to implement embodiments of the present disclosure. For example, the apparatus 1400 may be configured to perform the functions of a video encoding device or a video decoding device according to any of the embodiments of the present disclosure. The apparatus 1400 includes receiver units (RX) 1420 or receiving means for receiving data via ingress ports 1410. The apparatus 1400 also includes transmitter units (TX) 1440 or transmitting means for transmitting via data egress ports 1450. For example, on a sending side, the video encoding device may use the RX 1420 or receiving means to obtain an original image and then use the TX 1440 or transmitting means for transmitting an encoded bitstream. On the receiver side, the video decoding device may use the RX 1420 or receiving means to receive the bitstream an then use the TX 1440 or transmitting means for transmitting the decoded images of the original images (e.g., the reconstructed output image ^^) to a display device or to another computing device.

[0107] The apparatus 1400 includes a memory 1460 or data storing means for storing the instructions and various data. The memory 1460 can be any type of, or combination of, memory components capable of storing data and / or instructions. For example, the memory 1460 can include volatile and / or non-volatile memory such as read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM). The memory 1460 can also include one or more disks, tape drives, and solid-state drives. In some embodiments, the memory 1460 can be used as an over-flow data storage device to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution. In some embodiments, the memory 1460 can be memory that is integrated with the processor 1430.

[0108] The apparatus 1400 has one or more processors 1430 or other processing means (e.g., central processing unit (CPU)) to process instructions. The one or more processors 1430 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field-programmableAtty. Docket No.4502-83401 (6000689PCT02) gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The one or more processors 1430 are communicatively coupled via a system bus with the ingress ports 1410, RX 1420, TX 1440, egress ports 1450, and memory 1460. The one or more processors 1430 can be configured to execute instructions stored in the memory 1460. As an example, in one embodiment, the memory 1460 stores a hybrid LCVR and LSVR module 1470. The hybrid LCVR and LSVR module 1470 includes data, executable instructions, and / or one more sub-modules for implementing the disclosed embodiments. Thus, the one or more processors 1430 provide a means for performing any computational, comparison, determination, initiation, configuration, or any other action corresponding to the claims when the appropriate instruction is executed by the processor 1430. Thus, the inclusion of the hybrid LCVR and LSVR module 1470 substantially improves the functionality of the apparatus 1400.

[0109] While several embodiments have been provided in the present disclosure, it may be understood that the disclosed systems and methods might be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the disclosure is not to be limited to the details given herein. For example, the various elements or components may be combined or integrated in another system or certain features may be omitted, or not implemented.

[0110] In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items shown or discussed as coupled or directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediate component whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and may be made without departing from the spirit and scope disclosed herein.

Claims

Atty. Docket No.4502-83401 (6000689PCT02) CLAIMS What is claimed is:

1. A method implemented by a video encoding device, the method comprising: encoding an image into a bitstream as a set of code indices ^^^representing a set of basis elements ^^^according to Learned Sparse Image Representation (LSVR); encoding the image into the bitstream as a series of continuous real numbers ^^^according to Learned Continuous Visual Representation (LCVR); determining, via a neural network (NN), a degradation-aware latent ^^^, which approximates degradation that occurs to the image during LCVR encoding; encoding the degradation-aware latent ^^^into the bitstream; and storing the bitstream.

2. The method according to claim 1, wherein the degradation-aware latent ^^^is encoded into the bitstream via entropy coding.

3. The method according to any of claims 1-2, wherein the degradation-aware latent ^^^is determined by decoding the series of continuous real numbers ^^^to obtain a LCVR-based decoded image x^^and using the NN to estimate degradation that occurred to the LCVR-based decoded image x^^based on prior training on degradation that occurs to images during LCVR.

4. The method according to any of claims 1-3, wherein the degradation-aware latent ^^^is determined by using the NN to estimate degradation that occurred to the series of continuous real numbers ^^^based on prior training on degradation that occurs to images during LCVR.

5. The method according to any of claims 1-4, wherein the NN is trained with a target reconstruction loss including pixel-level distortion and perceptual loss between an output image and an input image.Atty. Docket No.4502-83401 (6000689PCT02) 6. The method according to any of claims 1-5, wherein the degradation-aware latent ^^^is determined for use as a condition when decoding the image from the set of basis elements ^^^.

7. The method according to any of claims 1-6, wherein the degradation-aware latent ^^^is determined for use as a condition when decoding the image from a fusion of the set of basis elements ^^^and the series of continuous real numbers ^^^.

8. A method implemented by a video decoding device, the method comprising: receiving a bitstream comprising an image coded as a set of code indices ^^^representing a set of basis elements Ŷ^^according to Learned Sparse Image Representation (LSVR) and as a series of continuous real numbers ^^^^according to Learned Continuous Visual Representation (LCVR); obtaining a degradation-aware latent, which approximates degradation that occurred to the image during LCVR encoding; and decoding the image based on the set of basis elements Ŷ^^, the series of continuous real numbers ^^^^, and the degradation-aware latent.

9. The method according to claim 8, wherein the degradation-aware latent is denoted as degradation-aware latent ^^^and is obtained from the bitstream.

10. The method according to any of claims 8-9, wherein the degradation-aware latent ^^^is determined by a first neural network (NN) based on an LCVR-based decoded image x^^.

11. The method according to any of claims 8-10, wherein the degradation-aware latent ^^^is determined by a first NN based on the set of basis elements Ŷ^^and the series of continuous real numbers ^^^^.Atty. Docket No.4502-83401 (6000689PCT02) 12. The method according to any of claims 8-11, further comprising applying, by a second NN, reverse diffusion to the degradation-aware latent ^^^to obtain a de-noised degradation-aware latent ^^^^.

13. The method according to any of claims 8-12, wherein applying reverse diffusion comprises determining a diffusion latent ^^^^from a LCVR-based decoded image ^^^and iteratively applying a noise to the degradation-aware latent ^^^based on the diffusion latent ^^^^to obtain a higher quality degradation-aware latent ^^^^.

14. The method according to any of claims 8-13, wherein applying reverse diffusion comprises determining a diffusion latent ^^^^from the series of continuous real numbers ^^^^and iteratively applying a noise to the degradation-aware latent ^^^based on the diffusion latent ^^^^to obtain a higher quality degradation-aware latent ^^^^.

15. The method according to any of claims 9-14, wherein the noise is a Gaussian noise.

16. The method according to any of claims 9-15, wherein decoding the image comprises applying the degradation aware latent to the series of continuous real numbers ^^^^to obtain a high quality series of continuous real numbers ^^ ^^^^ ^ and using the high quality series of continuous real numbers ^^ ^^^^ ^ as a condition for decoding the set of basis elements Ŷ^^.

17. The method according to any of claims 9-16, wherein decoding the image comprises fusing the series of continuous real numbers ^^^^and the basis elements Ŷ^^to determine a combined set of coded features Ŷ^^^and using the degradation aware latent as a condition for decoding the combined set of coded features Ŷ^^^.Atty. Docket No.4502-83401 (6000689PCT02) 18. An apparatus comprising: a memory or storage means configured to store instructions; and one or more processors or processing means coupled to the memory or the storage means and configured to execute the instructions to cause the apparatus to perform a method according to any of claims 1-7.

19. An apparatus comprising: a memory or storage means configured to store instructions; and one or more processors or processing means coupled to the memory or the storage means and configured to execute the instructions to cause the apparatus to perform a method according to any of claims 8-17.

20. A computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computer-executable instructions when executed by one or more processors of an apparatus, cause the apparatus to perform a method according to any of claims 1-17.

Images