Learned image compression with masked vision-language modelling
Masked vision-language modelling in Learned Image Compression addresses the limitations of existing LIC methods by enabling flexible bitrate and quality adjustments, achieving high compression efficiency and quality through multi-modal encoding and decoding techniques.
Patent Information
- Application Number
- PCT/US2024/040846
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-08-29
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-28
AI Technical Summary
Current Learned Image Compression (LIC) methods face limitations in compression performance due to model capacity constraints, difficulty in balancing rate-distortion loss, and lack of flexibility in controlling compression results, which affects both compression efficiency and perceptual quality.
The proposed solution employs masked vision-language modelling using multi-modal vision-language models, diffusion models, and transformers to encode and decode images, allowing for flexible bitrate and quality adjustments by computing adjusted masked features and diffusion latent features, enabling efficient transmission and high-quality reconstruction.
This approach achieves high compression rates with maintained reconstruction quality, offering flexible bitrate adjustments and improved performance over traditional methods, suitable for various applications including telecommunications, healthcare, and consumer electronics.
Smart Images

Figure US2024040846_28112024_PF_FP_ABST
Abstract
Description
Atty. Docket No.4502-81901 (6000641PCT02) LEARNED IMAGE COMPRESSION WITH MASKED VISION-LANGUAGE MODELLING CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims priority to U.S. Provisional Application No. 63 / 579,465 filed on August 29, 2023. The disclosures of the aforementioned applications are hereby incorporated by reference in their entireties. TECHNICAL FIELD
[0002] The present disclosure relates to Learned Image Compression (LIC), and in particular, to LIC with masked vision-language modelling for image compression. BACKGROUND
[0003] Artificial intelligence generated content (AIGC) uses a wide range of image generative models, including generative adversarial networks (GAN), diffusion models, and auto-regressive (AR) models. The goal is to enable fast and accessible high-quality content creation. Various methods have been developed to allow for efficient manipulation of the generated content using different types of inputs, such as using text descriptions and / or spatial / spatiotemporal compositions, like sketches or segmentations.
[0004] Large-scale pretrained Vision-Language Models (VLM) have reached a milestone in text-to-image generation for AIGC. By training a very large model using very large datasets of captioned images from the internet, a multi-modal language-image pre-training representation, like Contrastive Language-Image Pre-training (CLIP) or Bootstrapping Language-Image Pre-training (BLIP), can be successfully learned through self-supervised contrastive learning. The joint embedding space of text and image is robust to image distribution shift, which enables language- guided zero-shot image generation. SUMMARY
[0005] A first aspect relates to a method implemented by an encoder. The method includes encoding an original image into a sparse visual feature; generating a sparse text feature indicating content of the original image; computing, based on a control signal and the original image, aAtty. Docket No.4502-81901 (6000641PCT02) control latent feature; computing, based on the sparse visual feature, the sparse text feature, and the control latent feature, an adjusted masked sparse visual feature, an adjusted masked sparse text feature, and an adjusted masked control latent feature, respectively; computing a diffusion latent feature; and transmitting the adjusted masked sparse visual feature, the adjusted masked sparse text feature, the adjusted masked control latent, and the diffusion latent feature to a decoder.
[0006] Optionally, in a first implementation according to any of the first aspect or any implementation thereof, the method further includes computing, based on the original image, the adjusted masked sparse visual feature, and the adjusted masked control latent, the diffusion latent feature, wherein the diffusion latent feature captures fidelity and expressiveness details of the original image.
[0007] Optionally, in a second implementation according to any of the first aspect or any implementation thereof, wherein the control latent feature indicates an encoded control requirement.
[0008] Optionally, in a third implementation according to any of the first aspect or any implementation thereof, wherein computing the control latent feature comprises: computing, based on the control signal, a text instruction, wherein the text instruction is a text description describing control requirements of the control signal; computing, based on the text instruction, the original image, the control signal, and a prompt VLM, an input-oriented text instruction and additional input-oriented prompt instruction; and computing, based on the input-oriented text instruction and the additional input-oriented prompt instruction, the control latent feature.
[0009] Optionally, in a fourth implementation according to any of the first aspect or any implementation thereof, wherein the original image is a general three-dimensional (3D) tensor with shape ^ × ℎ × ^, where w, h, c are a width, a height, and a number of channels of an image, and wherein encoding the original image into the sparse visual feature comprises: encoding the original image into a vision feature tensor with shape ^^× ℎ^× ^, wherein width ^^and height ℎ^depend on the width and the height of the original image, and wherein d is a number of feature channels; and computing, based on the vision feature tensor and a vision codebook, a sparse visualAtty. Docket No.4502-81901 (6000641PCT02) feature, wherein the vision codebook comprises a plurality of codewords, and wherein each codeword has d dimension.
[0010] Optionally, in a fifth implementation according to any of the first aspect or any implementation thereof, wherein encoding the original image into the vision feature tensor comprises dividing, using a visual-language (VL) transformer, the original image into patches and encoding the patches as a sequence.
[0011] Optionally, in a sixth implementation according to any of the first aspect or any implementation thereof, wherein encoding the original image into the vision feature tensor comprises encoding in a parallel manner, using a convolutional neural network (CNN), the original image as an entire image.
[0012] Optionally, in a seventh implementation according to any of the first aspect or any implementation thereof, wherein generating the sparse text feature comprises: computing, based on the original image, a text latent feature comprising text description, wherein the text description describes the content of the original image; and generating, based on the text latent feature, the sparse text feature.
[0013] Optionally, in an eighth implementation according to any of the first aspect or any implementation thereof, wherein computing the text latent feature comprises: computing an enriched text latent feature; and computing, based on the enriched text latent feature and / or a text codebook, the text latent feature, wherein the text codebook comprises a plurality of codewords, wherein each codeword has natural language elements.
[0014] Optionally, in a ninth implementation according to any of the first aspect or any implementation thereof, wherein computing, based on the sparse text feature, the adjusted masked sparse text feature comprises: computing, based on the sparse visual feature, a masked sparse text feature by applying a text mask to the sparse text feature to change corresponding tokens in the sparse text feature according to the text mask, wherein the text mask is a binary mask having the same shape as the sparse text feature, and wherein the tokens having value 0 are set to a designated value indicating being removed; and computing, based on the masked sparse text feature and usingAtty. Docket No.4502-81901 (6000641PCT02) a visual-language (VL) transformer, the adjusted masked sparse text feature
[0015] Optionally, in a tenth implementation according to any of the first aspect or any implementation thereof, wherein computing, based on the sparse text feature, the adjusted masked sparse text feature comprises: computing, based on the sparse visual feature, a masked sparse text feature by applying a text mask to the sparse text feature to change corresponding tokens in the sparse text feature according to the text mask, wherein the text mask is a binary mask having the same shape as the sparse text feature, and wherein the tokens having value 0 are set to a designated value indicating being removed; and computing, based on the masked sparse text feature and using a visual-language (VL) transformer, the adjusted masked sparse text feature.
[0016] Optionally, in an eleventh implementation according to any of the first aspect or any implementation thereof, the method further includes computing, based on the control latent feature, the adjusted masked control latent by applying a visual mask to the control latent.
[0017] Optionally, in a twelfth implementation according to any of the first aspect or any implementation thereof, wherein the adjusted masked control latent comprises an adjusted masked input-oriented text instruction and an adjusted masked additional input-oriented prompt instruction.
[0018] Optionally, in a thirteen implementation according to any of the first aspect or any implementation thereof, wherein computing the diffusion latent feature comprises: downsampling the original image to smaller resolution images; and encoding the smaller resolution images to obtain the diffusion latent feature using information from the adjusted masked sparse visual feature.
[0019] A second aspect relates to a method implemented by a decoder. The method includes receiving a vision-language latent feature of an original image and a diffusion latent feature of the original image, wherein the vision-language latent feature comprises text and integers; computing, based on the vision-language latent feature, a decoded vision-language feature; reconstructing, based on the decoded vision-language feature, a baseline image output; computing, based on the diffusion latent feature and the baseline image output, a supplementary output; and constructing, based on the supplementary output and the baseline image output, a final decoded image output.
[0020] A third aspect relates to a method implemented by a decoder. The method includesAtty. Docket No.4502-81901 (6000641PCT02) receiving an adjusted masked sparse visual feature, an adjusted masked sparse text feature, an adjusted masked control latent, and a diffusion latent feature of an original image; generating, based on the received adjusted masked sparse visual feature, a recovered masked visual feature; computing, based on the received adjusted masked sparse text feature, an encoded masked text feature; computing, based on the adjusted masked control latent, the recovered sparse visual feature, and the encoded masked sparse text feature, an encoded masked control feature; reconstructing, based on the recovered masked visual feature, the encoded masked text feature, and the encoded masked control feature, a baseline image output; computing, based on the diffusion latent feature, the masked sparse visual feature, and the masked sparse text feature, a supplementary output; and constructing, based on the supplementary output and the baseline image output, a final decoded image output.
[0021] Optionally, in a first implementation according to any of the preceding aspects or any implementation thereof, wherein the recovered masked visual feature is based on a vision codebook, and wherein the encoded masked sparse text feature is based on a text codebook.
[0022] Optionally, in a first implementation according to any of the preceding aspects or any implementation thereof, wherein computing the supplementary output comprises: recovering, based on the diffusion latent feature, a reconstructed image; computing an embedded latent feature based on the reconstructed image; and computing, based on the embedded latent feature and using the baseline image output as a diffusion condition, the supplementary output.
[0023] Optionally, in a first implementation according to any of the preceding aspects or any implementation thereof, wherein computing the supplementary output comprises: recovering, based on the diffusion latent feature, a reconstructed image; computing an embedded latent feature based on the reconstructed image; and computing, based on the embedded latent feature and using the recovered masked visual feature and the encoded masked text feature as a diffusion condition, the supplementary output.
[0024] Optionally, in a first implementation according to any of the preceding aspects or any implementation thereof, wherein computing the supplementary output comprises: computing anAtty. Docket No.4502-81901 (6000641PCT02) embedded latent feature based on the diffusion latent feature; and computing, based on the embedded latent feature and using the baseline image output as a diffusion condition, the supplementary output.
[0025] Optionally, in a first implementation according to any of the preceding aspects or any implementation thereof, wherein computing the supplementary output comprises: computing an embedded latent feature based on the diffusion latent feature; and computing, based on the embedded latent feature and using the recovered masked visual feature and the encoded masked text feature as a diffusion condition, the supplementary output.
[0026] Optionally, in a first implementation according to any of the preceding aspects or any implementation thereof, the method further includes computing the supplementary output using a Denoising Diffusion Probabilistic Model (DDPM) or a Denoising Diffusion Implicit Model (DDIM).
[0027] Optionally, in a first implementation according to any of the preceding aspects or any implementation thereof, wherein computing the supplementary output comprises encoding, using an embedding network, the baseline image output from a pixel domain to a latent domain when the baseline image output is used as the diffusion condition.
[0028] Optionally, in a first implementation according to any of the preceding aspects or any implementation thereof, wherein computing the supplementary output comprises transforming, using a transformation network, a combined result to a dimension corresponding to the embedded latent feature when the recovered masked visual feature and the encoded masked text feature is used as a diffusion condition.
[0029] A fourth aspect relates to an encoder comprising a memory or storage means configured to store instructions; and one or more processors or processing means coupled to the memory or the storage means and configured to execute the instructions to cause the encoder to perform the method according to any of the preceding aspect or any implementation thereof.
[0030] A fifth aspect relates to a decoder comprising a memory or storage means configured to store instructions; and one or more processors or processing means coupled to the memory orAtty. Docket No.4502-81901 (6000641PCT02) the storage means and configured to execute the instructions to cause the decoder to perform the method according to any of the preceding aspect or any implementation thereof.
[0031] A sixth aspect relates to a computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computer- executable instructions when executed by a processor of an apparatus, cause the apparatus to perform the method according to any of the preceding aspect or any implementation thereof.
[0032] For clarity, any one of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create a new embodiment within the scope of the present disclosure.
[0033] These and other features, and the advantages thereof, will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF DRAWINGS
[0034] For a more complete understanding of this disclosure, reference is now made to the following brief description, taken in connection with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.
[0035] FIG.1 is a diagram illustrating a general processing pipeline for AIGC.
[0036] FIG.2 is a diagram illustrating a general framework for LIC.
[0037] FIG. 3 is a diagram illustrating a general framework for compression for machines (CfM).
[0038] FIG. 4 is a diagram illustrating a general framework for Learned Sparse Image Representation (LSIR).
[0039] FIG. 5A illustrates an encoding / decoding framework according to an embodiment of the present disclosure.
[0040] FIG. 5B illustrates an encoding / decoding framework according to an embodiment of the present disclosure.
[0041] FIG.6 illustrates a detailed workflow of a main branch according to an embodiment ofAtty. Docket No.4502-81901 (6000641PCT02) the present disclosure.
[0042] FIGS.7A-7C illustrate a detailed workflow of a text code generation module according to an embodiment of the present disclosure.
[0043] FIG. 8 illustrates a detailed workflow of a VL transformer module according to an embodiment of the present disclosure.
[0044] FIG. 9 illustrates a detailed workflow of a reconstruction module according to an embodiment of the present disclosure.
[0045] FIG. 10 illustrates a processing workflow of a control branch according to an embodiment of the present disclosure.
[0046] FIG. 11 illustrates a processing workflow of a control branch according to an embodiment of the present disclosure.
[0047] FIG.12 illustrates a processing workflow of a control adjustment module according to an embodiment of the present disclosure.
[0048] FIG. 13A illustrates a processing workflow of a diffusion branch according to an embodiment of the present disclosure.
[0049] FIG. 13B illustrates a processing workflow of a diffusion branch according to an embodiment of the present disclosure.
[0050] FIG. 14A and FIG. 14B illustrate a reverse diffusion module according to two embodiments of the present disclosure.
[0051] FIG. 15 is a diagram illustrating an apparatus according to an embodiment of the present disclosure. DESCRIPTION OF EMBODIMENTS
[0052] It should be understood at the outset that, although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated andAtty. Docket No.4502-81901 (6000641PCT02) described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.
[0053] Disclosed herein are various systems and methods for encoding and decoding an image. The present disclosure proposes a general framework that uses masked vision-language modelling for image compression. Based on several neural network models, including the multi-modal vision-language models, the large language models, the diffusion models, the masked encoders, and the querying transformers, the embodiments of the present disclosure provide a high compression rate with high reconstruction quality, where the bitrate and the reconstruction quality can be flexibly adjusted according to different compression demand.
[0054] As a practical application, the disclosed techniques enhance practical applications by significantly improving the functions of an encoder. For example, in telecommunications and digital media, it optimizes data transmission and streaming by compressing high-resolution images efficiently, thereby reducing bandwidth usage while maintaining quality. For example, in healthcare, it facilitates faster and more efficient storage and transmission of medical images, which is critical for remote consultations and diagnostics. Furthermore, in consumer electronics, it increases storage capacity in devices like smartphones and cameras, enabling users to save more high-quality images. Additionally, for cloud storage and autonomous vehicles, it lowers storage costs and enhances real-time image processing capabilities. Overall, the improved encoder functions provide high compression rates, maintain reconstruction quality, and offer flexible bitrate adjustments, making it a valuable tool for efficient data management and enhanced user experiences in various applications.
[0055] FIG. 1 is a diagram 100 illustrating a general processing pipeline for AIGC. A prompt input y is passed through a prompt encoder 102. The prompt input y may be a text input that provides a description of content to be generated using AI. In some embodiments, the prompt input y may also include images related to the to-be-generated AI content. The prompt encoder 102 is a component configured to encode the input prompts y into a format that a multi-modal embedding network 104 can understand and process. In an embodiment, the prompt encoder 102 is configuredAtty. Docket No.4502-81901 (6000641PCT02) to generate a prompt embedding feature ^^from the prompt input y, which represents the prompt input y encoded into an input format for the multi-modal embedding network 104. The prompt embedding feature ^^captures the semantic meaning and contextual information of the prompt input y, enabling the multi-modal embedding network 104 to understand and process the input effectively. The multi-modal embedding network 104 is a type of neural network architecture designed to merge information from multiple modalities, such as text, images, audio, or other types of data. In an embodiment, the multi-modal embedding network 104 is configured to compute an image embedding feature ^^that models the prior of ^(^^|^^) . In an embodiment, the image embedding feature ^^is a numerical representation of an image in a high-dimensional vector space. The image embedding feature ^^is passed to a decoder 106. The decoder 106 is a decoding neural network configured to compute an output image ^^ based on the image embedding feature ^^and the prompt embedding feature ^^. The target is to achieve high visual perceptual quality (e.g., natural and photo realistic, low level of visible artifacts) of the generated image ^^, and the semantic alignment of ^^ to the requirement described by the prompt input y.
[0056] FIG. 2 is a diagram 200 illustrating a general framework for LIC. LIC is a modern approach to image compression that utilizes deep learning techniques to learn efficient representations of images. Traditional image compression techniques rely on handcrafted algorithms that transform the image data into a compressed format. However, LIC aims to improve compression efficiency by training neural networks to automatically learn the most effective compression strategies directly from the data. LIC based on neural networks (NN) has been largely studied in recent years and has shown superior performance over traditional coding methods like Joint Photographic Experts Group (JPEG), Versatile Video Coding (VVC), and High Efficiency Video Coding (HEVC). In the depicted embodiment, on a sender side, an input image x is passed through an input encoder 202 to generate an image embedding feature ^^, which is a representation of the input image x in a numerical format. In an embodiment, the input encoder 202 is a neural network configured to convert the raw pixel values of the input image x into a compressed and semantically meaningful numerical representation in a high-dimensional vectorAtty. Docket No.4502-81901 (6000641PCT02) space. In some embodiments, the image embedding feature ^^is further compressed through quantization and arithmetic coding into a data string that is efficient for storage and transmission from the sender to a receiver.
[0057] In an embodiment, on the receiver side, a decoded image embedding feature ^̂^is recovered from the received data string sent by the sender using arithmetic decoding and dequantization. The decoded image embedding feature ^̂^is used as input for a decoder 204. The decoder 204 is configured to reconstruct an output image ^^ based on the decoded image embedding feature ^̂^. The target is to minimize the restoration loss between the reconstructed output ^^ and the original input x, and to minimize the bits to represent the image embedding feature ^^for storage and transmission.
[0058] Traditionally, compression methods are developed for human consumption. That is, the reconstructed output ^^ is targeted to be viewed by human, and the goal is to preserve high visual quality. The compression induced artifacts in ^^ can largely degrade the performance of some machine analytic tasks, such as detection and recognition tasks, since the information needed for such tasks may be altered or lost during compression. To facilitate machine analytics, standard activities such as the Moving Picture Experts Group (MPEG) Video Coding for Machines (VCM) and JPEG-AI have been launched to investigate compression methods that are suitable for machine analytic tasks.
[0059] FIG.3 is a diagram 300 illustrating a general framework for CfM. In FIG.3, a machine- oriented pre-processing module 302 and / or machine-oriented post-processing module 304 are used before and / or after a video compression method (e.g., VVC, HEVC, LIC, etc.) to pre-process the input image x before the input encoder 202 (as described in FIG. 2) and / or post-process the reconstructed output ^^ after the decoder 204 (as described in FIG. 2) for a connected target machine analytic task model 306. In an embodiment, the machine-oriented pre-processing module 302 and / or the machine oriented post-processing module 304 are optimized with the machine analytic task model in the end-to-end fashion (while keeping the video compression method and the machine analytic task model unchanged) by using the task performance loss. In an embodiment,Atty. Docket No.4502-81901 (6000641PCT02) one set of the machine-oriented pre-processing module 302 and / or the machine oriented post- processing module 304 are used for each specific task model of each machine analytic task.
[0060] FIG. 4 is a diagram 400 illustrating a general framework for Learned Sparse Image Representation (LSIR). In LSIR, a vector-quantized autoencoder in the image domain is trained based on adversarial and perceptual loss (e.g., using the Vector Quantized Generative Adversarial Network (VQGAN) method) to learn a highly compressed codebook 402. The learned codebook 402 comprises a collection of codewords used in a compression algorithm. The goal is to represent an image using a set of codewords from the codebook 402 more efficiently than directly encoding each vector or group of pixels within the image. The learned codebook 402 is optimized end-to- end to balance codebook efficiency and reconstruction quality. As shown in FIG.4, an input image x is passed through the input encoder 202 to generate the image embedding feature ^^as described in FIG. 2. For LSIR, on the sender side, the image embedding feature ^^is mapped into a sequence of code indices ^^^using the learned codebook 402. The code indices ^^^are integers that can be effectively stored or transmitted from the sender to a receiver. On the receiver side, the same learned codebook 402 is used to recover the decoded image embedding feature ^̂^(e.g., by using the codewords in the codebook 402 corresponding to the received code indices ^^^). The decoder 204 is then configured to reconstruct an output image ^^ based on the decoded image embedding feature ^̂^.
[0061] Image captioning aims at generating text keywords, sentences, and / or paragraphs to describe the content of the image. Pretrained Large Language Models (LLM) like GPT-3 has shown strong capability in natural language understanding and natural language generation. The LLM is trained over large-scale text data like spreadsheets, fictional novels, etc., and can perform a variety of linguistic tasks. Previous methods have combined VLM with LLM to improve the image captioning performance by exploiting the distinct knowledge learned from the text domain and the image-text joint domain.
[0062] Although text-driven image synthesis has been successfully explored in AIGC, text- driven image editing remains challenging for generative models. The innate requirement of anAtty. Docket No.4502-81901 (6000641PCT02) editing method is to preserve most of the original image content while changing only the desired content directed by text description. It is non-trivial for AIGC to fulfil such a requirement due to the lack of control over the generated content based on text descriptions. To circumvent this issue, most text-driven image editing methods require the user to explicitly point out the region to be edited, e.g., using masks. The recent Prompt-to-Prompt Editing (P2PE) edits images by injecting the cross-attention maps during the diffusion process to control the generation of pixels corresponding to text prompts during the diffusion steps.
[0063] Masked Image Modelling (MIM) has been shown effective to boost the quality of the learned visual latent representations in a self-supervised fashion, for both downstream tasks like classification and recognition and for reconstruction tasks like image synthesis. A vision transformer is usually used to predict the masked missing tokens from the unmasked ones. The recent Masked Generative Encoder (MAGE) uses a unified training framework, where a pre- trained VQGAN generates tokens from images and then a masked encoder and an image generator are trained together based on the tokens.
[0064] The current LIC framework, as described in FIG. 2, relies on learning a general compact image representation (i.e., a latent space where the image embedding feature ^^can capture the gist of the input x to reconstruct ^^). This framework has several severe limitations. First, the compression performance is innately bounded by the model capacity (e.g., the network structure and number of parameters of the input encoder 202 and the decoder 204) in learning the general prior ^(^|^^) in the image domain. Due to the limited model capacity, limited training data, and limited computation resources in both training and test stage, it is hard to further improve the compression performance beyond a good baseline. Second, the LIC models are taught to balance the competing goals in the rate-distortion (RD) loss, where reducing reconstruction distortion and reducing bitrate contradict with each other. It is hard to improve the compression performance and the perceptual quality at the same time, due to the difficulty in balancing different loss terms in end-to-end training.
[0065] The current CfM framework, as described in FIG. 3, has little flexibility or generalityAtty. Docket No.4502-81901 (6000641PCT02) since one set of machine-oriented pre-processing module 302 and / or machine-oriented post- processing module 304 are customized for a specific task model of each task. When multiple tasks (e.g., multiple levels of recognition) are needed, the CfM framework needs to compute and transmit multiple encoded streams using multiple sets of model parameters.
[0066] Comparing with LIC, the current LSIR framework, as described in FIG. 4, provides inferior performance in image compression. This is because when used for compression, LSIR has an aggressive goal of high compression rate by using a compact codebook (i.e., the learned codebook 402 in FIG.4) to model the complicated generic image prior ^(^|^^). The reconstructed image usually lacks expressive and fidelity details. In addition, it is quite challenging to flexibly control the compression result to fit different compression targets, such as to preserve the fidelity, to improve perceptual quality, or to fulfil other needs of using the image. A compression method with flexibility, scalability, and generality that can suit various compression needs is highly desired for practical usage.
[0067] The large-scale VLMs provides an opportunity to overcome the limitations in representation learning solely in the image domain. The joint embedding space of text and image largely expand the learning capacity by exploring the joint prior ^(^, ^|^^, ^^) in the language- image joint domain. The text-driven image editing like P2PE also provides an opportunity to flexibly control the compression process by using text prompts that are innately highly compressed to be easily transmitted. However, it is non-trivial to use AIGC for LIC. For instance, as described in FIG. 1 and FIG. 2, AIGC and LIC have different goals. LIC, as shown in FIG. 2, requires reconstruction of the original input x, and the current AIGC framework, as shown in FIG.1, is not designed to guarantee such a requirement. That is, from the prompt input y, the generated ^^ is drawn from the joint distribution ^(^, ^|^^, ^^), which is usually not a reconstructed version of the original input x. Also, the prompt-based control in P2PE through diffusion models still cannot guarantee the authenticity of the unedited content.
[0068] The success of MIM in visual representations learning is inspiring. However, it has not been extended to learn vision-language representations. It can be expected that the maskingAtty. Docket No.4502-81901 (6000641PCT02) strategy can help to improve the performance of the learned VLM representation. Furthermore, in terms of compression, the masking strategy further reduces bit counts, since the masked pixels can be skipped without being transmitted.
[0069] The present disclosure proposes a general framework that uses the powerful multi- modal representation learning in AIGC for LIC, based on a masked Vision-Language Modelling (MAVLM) method to achieve high compression efficiency with high reconstruction quality, where the bitrate and the reconstruction quality can be flexibly adjusted to according to different compression demand.
[0070] FIG. 5A illustrates an encoding / decoding framework 500A according to an embodiment of the present disclosure. The encoding / decoding framework 500A has three processing branches: a main branch 502, a control branch 504, and a diffusion branch 506. As shown in FIG. 5A, on the sender sider, the main branch 502 uses a sparse image encoder 508 to encode, by using a Learned Sparse Vision-Language Representation (LSVLR), the original input x into a sparse visual feature ^^^^, and uses an image-grounded text generator 510 to generate a sparse text feature ^^^^to describe the content of the input x. The control branch 504 takes as inputs a control signal ctl and the original input x and generates a control latent ^^^^^through a control generation module 512. In an embodiment, the sparse visual feature ^^^^, the sparse text feature ^^^^, and the control latent ^^^^^are fed into an MAVLM module 514. The MAVLM module 514 is configured to compute an adjusted masked sparse visual feature ^̃^^^^^, an adjusted masked sparse text feature ^̃^^^^^, and an adjusted masked control latent ^̃^^^^^^. The adjusted masked sparse visual feature ^̃^^^^^, the adjusted masked sparse text feature ^̃^^^^^, and the adjusted masked control latent ^̃^^^^^^usually consist of integers, texts, and a few numbers, which can be efficiently transmitted to a decoder.
[0071] Additionally, the diffusion branch 506, computes, using a compression module 516, a diffusion latent feature ^^^^^^^ ^!"based on the original input x and the adjusted masked sparseAtty. Docket No.4502-81901 (6000641PCT02) visual feature ^̃^^^^^and the adjusted masked control latent ^̃^^^^^^. The diffusion latent feature ^^^^^^^ ^!"captures the gist of the fidelity and expressiveness details of the original image x supplementary to the mainThe diffusion latent feature ^^^^^^^ ^!"consumes very low bitrate to transmit and may be further compressed by quantization and arithmetic coding. The adjusted masked sparse visual feature ^̃^^^^^, the adjusted masked sparse text feature ^̃^^^^^, the adjusted masked control latent ^̃^^^^^^, and the diffusion latent feature ^^^^^^^ ^!"are transmitted from the encoder on the sender side to the decoder on the receiver side.
[0072] In FIG.5A, on the receiver side, in the main branch, a visual feature recovery module 518 is configured to generate a recovered masked visual feature ^̂^^^^^based on the received adjusted masked sparse visual feature ^̃^^^^^. A text encoding module 520 is configured to compute an encoded masked text feature ^̂^^^^based on received adjusted masked sparse text feature ^̃^^^^^. Furthermore, in the control branch 504, a control encoding module 522 is configured to compute an encoded masked control feature ^̂^^^^^^based on the received adjusted masked control latent ^̃^^^^^^, the recovered masked visual feature ^̂^^^^^, and the encoded masked text featurean embodiment, a reconstruction module 524 is configured to reconstruct a baseline output ^^#$^"based on the recovered masked visual feature ^̂^^^^^, the encoded masked text feature ^̂^^^^, and the encoded masked control feature ^̂^^^^^^.
[0073] As shown in FIG.5A, the baseline output ^^#$^"will be combined with supplementary information from the diffusion branch to reconstruct the final output ^^. In an embodiment, the decoder uses a diffused decompression module 526 to compute a supplementary output ^^^%based on the received diffusion latent feature ^^^^^^and the baseline output ^^#$^". The supplementary output ^^^%and the baseline image output ^^#$^"are passed to a fusion module 528. The fusion module 528 is configured to combine the supplementary output ^^^%and the baseline image output ^^#$^"to reconstruct the final output ^^, which represents a decoded imageAtty. Docket No.4502-81901 (6000641PCT02) of the original input x. In all of the disclosed embodiments, the decoder may then transmit the final output ^^ to a display device for displaying of the decoded image, or transmit the final output ^^ to another computing device such as, but not limited to, a client device that requested the image. In some embodiments, the decoder may pass the decoded image to another application (e.g., an image editing application) for further processing.
[0074] FIG. 5B illustrates an encoding / decoding framework 500B according to an embodiment of the present disclosure. The encoding / decoding framework 500B is similar to the encoding / decoding framework 500A in FIG. 5A, except that in the decoder, the diffused decompression module 530 is configured to compute the supplementary output ^^^%based on the received diffusion latent feature ^^^^^^, the recovered masked visual feature ^̂^^^^^, and the encoded masked text feature ^̂^^^^.
[0075] In the disclosed embodiments, the term “feature” generally refers to a set of values that together provide a detailed representation of specific information or characteristics relevant to the original image or its content, and these features are manipulated and processed through various stages of the methods described. In an example, the sparse visual feature represents the original image in a reduced or compressed form that retains essential visual information while being sparse. In another example, the sparse text feature indicates the content of the original image in a textual or descriptive format and is also sparse.
[0076] FIG. 6 illustrates a detailed workflow 600 of a main branch according to an embodiment of the present disclosure. As discussed above in FIGS.5A-5B, the main branch 502 uses a sparse image encoder 508 to encode the original input x into a sparse visual feature ^^^^. In an embodiment, the original image x is a general three-dimensional (3D) tensor with shape ^ × ℎ × ^, where w, h, c are the width, height, and number of channels of the image. For example, c = 3 for color images, c = 1 for spectral images, or c = 4 for RGB-D (color and depth) images or for video segments where c is the number of frames. As shown in FIG. 6, the sparse image encoder 508 includes a vision embedding module 602 configured to encode the original image x into a vision feature tensor ^^^with shape ^^× ℎ^× ^, where the width ^^and heightAtty. Docket No.4502-81901 (6000641PCT02) ℎ^depend on the input width and height as well as the network structure of the vision embedding module 602, and where d is the number of feature channels. Various neural networks can be used as the vision embedding module 602. For example, in one embodiment, a visual transformer (ViT) may be used. The ViT is configured to divide the original image x into patches and encode the patches as a sequence. In another embodiment, a convolutional neural network (CNN) structure is used where the entire original image x is encoded in a parallel manner.
[0077] The vision feature tensor ^^^is then passed to a vision code generation module 604. The vision code generation module 604 is configured to compute a sparse codebook-based latent feature ^^^^based on the vision feature tensor ^^^and a vision codebook +^606. In an embodiment, the vision codebook +^606 comprises of ,^of codewords, each having d dimensions. Each pixel ^^^^,^in ^^^^(- = 1, … , ^^× ℎ^) corresponds to a codeword ^^^∈ +^that is nearest to the corresponding latent feature ^^^,^: ^^^= 012345^6,7∈ 8694:;(^^,<, ^^^,^),
[0078] where 94:;() is a distance metric, such as L1 or L2 norm. The L1 norm is the sum of the absolute value of the entries in the vector. The L2 norm is the square root of the sum of the entries of the vector. That is, the entire sparse codebook-based latent feature ^^^^has ^^× ℎ^integers corresponding to the indices of ^^× ℎ^codewords. The sparse codebook-based latent feature ^^^^can be efficiently transmitted to the decoder in a lossless way with very little bit consumption.
[0079] Additionally, as discussed above in FIGS.5A-5B, the main branch 502 uses an image- grounded text generator 510 to generate a sparse text feature ^^^^to describe the content of the input x. As shown in FIG.6, on the sender side, the image-grounded text generator 510 includes a a text generation module 608 that takes the original image as input and computes a language latent feature ^^^. In an embodiment, the language latent feature ^^^contains text words and / orAtty. Docket No.4502-81901 (6000641PCT02) sentences that can be efficiently transmitted to the decoder. The text generation module 608 generates text describing the content of the original image x. In an embodiment, the text generation module 608 uses a pre-trained multi-modal vision-language representation such as CLIP or BLIP that learns the joint prior ^(^, ^|^^, ^^) of the original image x and the associated text descriptions y and computes the conditional ^(^|^) based on the pre-trained multi-modal vision- language representation. The image-grounded text generator 510 further includes a text code generation module 610 and a large language model (LLM) 612 to compute the sparse text feature ^^^^based on the text latent feature ^^^.
[0080] FIGS. 7A-7C illustrate exemplar embodiments of a text code generation module according to an embodiment of the present disclosure. As shown in FIG. 7A, the text code generation module 702 comprises an LLM 704 to compute an enriched text latent feature ^^^^="by generating natural language descriptions. In an embodiment, the text code generation module 702 comprises a text code mapping module 706 configure to compute the sparse text feature ^^^^based on a text codebook 708. The text codebook +^708 contains ,>codewords where each codewords can be natural language elements like words, phrases, sentences or combinations of these elements. The text codebook +^708 can also be learned, where each codeword is a dT- dimension feature vector. The sparse text feature ^^^^consists of codeword indices (e.g., integers) that can be easily transmitted to the decoder.
[0081] As shown in FIG.7B, the text code generation module 710 directly use the text latent feature ^^^to compute the sparse text feature ^^^^by the text code mapping module 712 with the text codebook 714, without using the LLM. As shown in FIG. 7C, in the text code generation module 716, the enriched text latent feature ^^^^="generated from the LLM 718 is directly used as the sparse text feature ^^^^, which is also efficient to transmit, without using the text code mapping module. In all of the disclosed embodiments, the text codebook can be rule defined or pre-trained, wherein the LLM is a pretrained large language model such as Generative Pre-trained Transformer (GPT-3).Atty. Docket No.4502-81901 (6000641PCT02)
[0082] Referring back to FIG.6, the sparse visual feature ^^^^, the sparse text feature ^^^^, and the control latent ^^^^^are fed into an MAVLM module 614. The MAVLM module 614 comprises a visual masking module 616 configured to compute the masked sparse visual feature ^^^^^^based on the sparse visual feature ^^^^, and a text masking module 618 configured to compute the masked sparse text feature ^^^^^^based on the sparse text feature ^^^^. The visual masking module 616 applies a visual mask to the sparse visual feature ^^^^to change the corresponding items (a.k.a. tokens) in ^^^^according to the visual mask. For example, the visual mask can be a binary mask having the same shape as ^^^^, where tokens in ^^^^^^with the corresponding items in the visual mask having value 0 are set to a designated value indicating being removed. The Text Masking module 618 applies a text mask to the sparse text feature ^^^^to change the corresponding tokens in ^^^^according to the text mask. For example, the text mask can be a binary mask having the same shape as ^^^^, where tokens in ^^^^^^with the corresponding items in the text mask having value 0 are set to a designated value indicating being removed.
[0083] Additionally, the control latent ^^^^^is fed into the visual masking module 616 to compute the adjusted masked control latent ^̃^^^^^^. For example, the items in the control latent ^^^^^corresponding to the removed tokens in ^^^^^^are also removed in ^̃^^^^^^. The MAVLM module 614 comprises further comprises a vision-language (VL) transformer 620 configured to compute the adjusted masked sparse visual feature ^̃^^^^^and the adjusted masked sparse text feature ^̃^^^^^based on the masked sparse visual feature ^^^^^^and the masked sparse text feature ^^^^^^. The VL transformer 620 aims to connect the vision and language features through cross-modality prediction, and at the same time improve the robustness of the learned visual and text feature representations through masked prediction.
[0084] FIG. 8 illustrates a detailed workflow of a VL transformer module according to anAtty. Docket No.4502-81901 (6000641PCT02) embodiment of the present disclosure. The VL transformer 800 is a detailed example of the VL transformer described in FIG. 6. In an embodiment, the VL Transformer 800 uses a querying transformer structure such as the Q-Former. The VL transformer 800 includes a visual embedding module 802 configured to compute a masked visual embedding feature ^́^^^^^based on the masked sparse visual feature ^^^^^^, and a text embedding module 804 configured to compute a masked text embedding feature ^́^^^^^based on the masked sparse text feature ^^^^^^. In an embodiment, the visual embedding module 802 can simply retrieve the codewords from the vision codebook according to the codeword indices in ^^^^^^. For example, each “pixel” of unmasked ^́^^^^^,^(- = 1, … , ^^× ℎ^) is the codeword with index ^^^^^^,^. For the masked (removed) tokens(predefined or learned in training) d-dimension feature vector can be used. In an embodiment, the text embedding module 804, the structure depends on the format of ^^^^^^. In one example, when ^^^^^^consists of texts, the text embedding module 804 usually takes an encoder network structure to compute a dTdimensional feature vector for each token in ^^^^^^. In one example, when ^^^^^^consists of codeword indices of a learned text codebook, the text embedding module 804 can simply retrieve the codewords from the text codebook according to the codeword indices in ^^^^^^. In an embodiment, as for the masked tokens in ^^^^^^, a predetermined (predefined or learned in training) dTdimensional feature vector can be used.
[0085] The VL transformer 800 further includes a visual self-attention blocks module 806 configured to pass the masked visual embedding feature ^́^^^^^. In an embodiment, the visual self-attention blocks module 806 has the self-attention transformer layers where the query, key, and value are all generated from the masked visual embedding feature ^́^^^^^. The VL transformer 800 further includes a text self-attention blocks module 808 configured to pass the masked textAtty. Docket No.4502-81901 (6000641PCT02) embedding feature ^́^^^^^. In an embodiment, the text self-attention blocks module 808 has the self-attention transformer layers where the query, key, and value are all generated from the masked text embedding feature ^́^^^^^.
[0086] In an embodiment, the output of the visual self-attention blocks module 806 and the text self-attention blocks module 808 are passed through a visual cross-attention blocks module 810 and a text cross-attention blocks module 812. In an embodiment, visual cross-attention blocks module 810 has the cross-attention transformer layers where the key and value are generated from the masked visual embedding feature ^́^^^^^while the query is generated from the masked text embedding feature ^́^^^^^. The text self-attention blocks module 812 has the cross-attention transformer layers where the key and value are generated from the masked text embedding feature ^́^^^^^while the query is generated from the masked visual embedding feature ^́^^^^^.
[0087] The visual cross-attention blocks module 810 followed by a visual predictor module 814 to compute the adjusted masked sparse visual feature ^̃^^^^^. The visual predictor module 814 aims to generate an unmasked visual embedding feature ^́^^^by using a prediction network that are trained to predict the codeword indices of the masked visual tokens in ^^^^^^. The text cross-attention blocks module 812 followed by a text predictor module 816 to compute the adjusted masked sparse text feature ^̃^^^^^. The text predictor module 816 aims to generate an unmasked text embedding feature ^́^^^by using a prediction network that are trained to predict the codeword indices of the masked text tokens in ^^^^^^.
[0088] The VL transformer 800 further includes a visual mask adjustment module 818 and a text mask adjustment module 820. The visual mask adjustment module 818 is configured to compute the adjusted masked sparse visual feature ^̃^^^^^by using a network that are trained to compute a mask that can maintain the most important information to reconstruct the input fromAtty. Docket No.4502-81901 (6000641PCT02) the adjusted masked output. The text mask adjustment module 820 is configured to compute the adjusted masked sparse text feature ^̃^^^^^by using a network that is trained to compute a mask that can maintain the most important information to reconstruct the input from the adjusted masked output. In general, the VL transformer 800 tunes the original ^^^^^^and ^^^^^^by considering the cross-modal and uni-modal relations of vision and language tokens.
[0089] In an embodiment, in the test stage, the visual masking module 616 and the text masking module 618 can be seen as additional bit reduction method where the masked-out tokens can be further indicated with reduced bit consumption. This masking ratio in the test stage can be flexibly tuned and can be skipped (i.e., without applying masks to remove any elements) according to the actual compression needs. During the training process, a relatively high non-zero masking ratio is used to train the VL transformer 800.
[0090] Referring back to FIG. 6, the adjusted masked sparse visual feature ^̃^^^^^and the adjusted masked sparse text feature ^̃^^^^^from the VT transformer 620 and the adjusted masked control latent ^̃^^^^^^are transmitted from the encoder on the sender side to the decoder on the receiver side.
[0091] On the receiver side, the decoder includes a visual feature recovery module 622. In this embodiment, the visual feature recovery module 622 is configured to generate, based upon the received adjusted masked sparse visual feature ^̃^^^^^, the recovered masked visual feature ^̂^^^^^of shape ^^× ℎ^× ^ based on the same vision codebook +^624 as the sender. In an embodiment, each “pixel” of ^̂^^^^^(- = 1, … , ^^× ℎ^) is the codeword with index ^̂^^^^^,^.
[0092] In an embodiment, the decoder further includes a text encoding module 626 configured to compute the encoded masked text feature z ̂_x^(L-M) based upon the received adjusted masked sparse text feature ^̃^^^^^. The text encoding module 626 depends on the format of ^̃^^^^^. In one example, when ^̃^^^^^consists of texts, the text encoding module 626 usually takes an encoderAtty. Docket No.4502-81901 (6000641PCT02) network structure to compute a dT dimensional feature vector for each token in ^̃^^^^^. In one example, when ^̃^^^^^consists of codeword indices of a learned text codebook, the text encoding module 626, can simply retrieve the codewords from the text codebook according to the codeword indices in ^̃^^^^^. The masked tokens in ^̃^^^^^can be set to a predetermined (pretrained or learned in training) d-dimension feature vector in ^̂^^^^^. The masked tokens in ^̃^^^^^can be set to a predetermined (pretrained or learned in training) dT-dimension feature vector in ^̂^^^^.
[0093] In an embodiment, the decoder further includes a control encoding module 628 configured to compute the encoded masked control feature ^̂^^^^^^based on the received adjusted masked control latent ^̃^^^^^^as well as the recovered masked visual feature ^̂^^^^^. Details of control encoding module 628 are described below in FIG.10. As shown in FIG.6, the recovered masked visual feature ^̂^^^^^and the recovered masked text feature ^̂^^^^are fed into the reconstruction module 630 together with the encoded masked control feature ^̂^^^^^^to reconstruct the baseline output ^^#$^". There are multiple ways to combine the recovered masked visual feature ^̂^^^^^, the recovered masked text feature ^̂^^^^, and the encoded masked control feature ^̂^^^^^^in the reconstruction module 630.
[0094] FIG.9 illustrates a detailed workflow of a reconstruction module 900 according to an embodiment of the present disclosure. As show in FIG. 6, the recovered masked visual feature ^̂^^^^^and the recovered masked text feature ^̂^^^^are fed into the reconstruction module 630 together with the encoded masked control feature ^̂^^^^^^to reconstruct the baseline output ^^#$^"In an embodiment, same as the VL Transformer in encoder on the sender side, the recovered masked visual feature ^̂^^^^^is fed into the visual self-attention blocks module 902 followed by the visual cross-attention blocks 904 and the visual predictor 906 to compute the recovered visual feature ^̂^^^. Tin an embodiment, the recovered masked text feature ^̂^^^^is fed into the text self- attention blocks 908 followed by the text cross-attention blocks 910 and the text predictor 912 toAtty. Docket No.4502-81901 (6000641PCT02) compute the recovered text feature ^̂^^. In an embodiment, a generator 914 computes the baseline output ^^#$^"based upon the recovered visual feature ^̂^^^, recovered text feature ^̂^^, and the encoded masked control feature ^̂^^^^^^.
[0095] The generator 914 usually has a network structure of multiple CNN layers like the decoding network of a variational autoencoder (VAE). In some embodiments, ^̂^^and ^̂^^^^^^are weighted combined with ^̂^^^by tuning ^̂^^^through an affine transformation and generate a new combined feature ^̂^^^8= ^^^^8(A^^^8^̂^^^+ C^^^8) with a weight ^^^^8and affine parameters A^^^8, C^^^8. A^^^8,H(^̂^^, ^̂^^^^^^)IJ, where ^F5() is the concatenation operation and H(^̂^^,information from ^̂^^and ^̂^^^^^^, e.g., through convolution. In other embodiments, the reconstruction module 900 can be a decoder diffusion model such as the text-conditioned image generation model or a guided language to image diffusion for generation and editing (GLIDE) model, or another prompt-conditioned image generation model. The recovered text feature ^̂^^and the encoded masked control feature ^̂^^^^provide guidance to the image diffusion process.
[0096] FIG. 10 illustrates a processing workflow of a control branch 1000 according to an embodiment of the present disclosure. In an embodiment, given the input control signal ctl, an instruction generation module 1002 first generates a text instruction ^^^^based on the control signal ctl using a prompt LLM 1004. The prompt LLM 1004 can be the same or different from the LLM used by the main branch in FIG.7. The text instruction ^^^^is a text description describing the control requirements of ctl. In an embodiment, a prompt generation module 1006 generates an input-oriented text instruction ^^^^^and additional input-oriented prompt instruction :^^^^based on the text instruction ^^^^, the input image x, the control signal ctl, and a prompt1008.
[0097] In an embodiment, the control signal ctl can take many different forms, such as one or a combination of the following control mechanisms: a text description, a sketch drawing, a bounding box, a color panel, etc. In an embodiment, the instruction generation module 1002 uses the prompt VLM 1008 such as Generative Pre-trained Transformer (GPT-3) to compute the textAtty. Docket No.4502-81901 (6000641PCT02) instruction ^^^^based on the text description in the control signal ctl. The text instruction ^^^^is usually an enriched text description elaborating the instruction of the text description in the control signal ctl. The other forms of the control signal ctl and the original input, together with the enriched text instruction ^^^^are processed in the prompt generation module 1006 to compute the input- oriented text instruction ^^^^^and additional input-oriented prompt instruction :^^^^using the prompt VLM 1008. The prompt VLM 1008 usually models the multimodal embedded representation between the text descriptions, various forms of prompts, and the images.
[0098] In an embodiment, the multimodal embedding of images and text descriptions learned with guidance from the image segmentation masks can be used as the prompt VLM 1008, which generates the additional input-oriented prompt instruction :^^^^as bounding boxes locating the regions that the control signal ctl focuses on. One concrete example is the control signal ctl aims at ensuring the reconstruction quality of a specific object in the scene (e.g., ctl is text description “HQ bear”). The enriched text instruction ^^^^elaborate such requirements (e.g., ^^^^is “ensure high quality and high resolution of the animal bear”). The input-oriented text instruction ^^^^^is computed to reflect the actual content of the input x (e.g., ^^^^^is “ensure high quality and high resolution of the brown bear catching fish in the river”). The additional input-oriented prompt instruction :^^^^can be the bonding box of the bear to focus on in the image.
[0099] In an embodiment, the control signal ctl can vary based on different compression requirements. For example, the control signal can be “Fidel bear” instead of “HQ bear” to emphasize the reconstruction fidelity of the specific content instead of perceptual quality. Such requirements can be useful for successive detection and recognition tasks for machine analysis. Besides text descriptions, other types of prompts can be used as control signal ctl, such as selecting a style of texture or a style of color. In another example, the control signal ctl can include a text description “foliage background” and a warm foliage color panel. The enriched text instruction ^^^^can be “tune image to have foliage color”. The above two examples can be combined into one complex control signal ctl as “Fidel bear, foliage background”, and the enriched text instruction ^^^^can be “tune image to have foliage color while keeping the animal bear asAtty. Docket No.4502-81901 (6000641PCT02) original”. Accordingly, the input-oriented text instruction ^^^^^and the additional input-oriented prompt instruction :^^^^will change to reflect such control instructions in guiding the reconstructed image.
[0100] In an embodiment, the cross-attention mechanism that are trained to capture the attention responses across multiple modalities including image, text, and various prompts can be used to implement the prompt VLM module 1008 such as the cross-attention used in P2PE. One exemplar structure of the prompt VLM module 1008 is a multimodal encoder with cross-attention followed by a multimodal generator. In an embodiment, the network structure of adding conditional prompt control can be used to implement the prompt VLM module 1008, where a desired type of prompt controls such as sketches, masks, bounding boxes, etc. can be added to a basic VLM for text-image embedding.
[0101] In an embodiment, the input-oriented text instruction ^^^^^and the additional input- oriented prompt instruction :^^^^form the control latent ^^^^^, which is fed into a visual masking module 1010 to generate the adjusted masked control latent ^̃^^^^^^according to the masking operation. For example, the input-oriented text instruction ^^^^can be adjusted to an adjusted masked input-oriented text instruction ^^^^^^as to “tune image to have foliage color while keeping the animal bear as original, focusing on the less masked animal bear”. Also, the additional input-oriented prompt instruction :^^^^is adjusted to an adjusted masked additional input-oriented prompt instruction :^^^^^^. In an embodiment, the control latent ^^^^^does not need to be changed and ^̃^^^^^^is^^^^^. After that the adjusted masked control latent ^̃^^^^^^is sent to the receiver side.
[0102] On the receiver side, the received ^̃^^^^^^is fed into a prompt embedding module 1012, which computes the encoded masked control feature ^̂^^^^^^by using the prompt VLM module 1014. In an embodiment, since the promptmodels the multimodal embedded representation between text descriptions, various forms of prompts, and images, the received adjusted masked input-oriented text instruction ^^^^^^^and the adjusted masked additional input- oriented prompt instruction :^^^^^^can be fed into this multimodal representation space to obtainAtty. Docket No.4502-81901 (6000641PCT02) the masked multimodal encoded feature ^̂^^^^^^through a multimodal encoder. In such case, the prompt embedding module 1012 can be the encoder with the cross-attention mechanism from the prompt VLM 1014.
[0103] FIG. 11 illustrates a processing workflow 1100 of a control branch according to an embodiment of the present disclosure. In an embodiment, the input control signal ctl, same as the process in FIG. 10. The control generation module 1102 and the visual masking module 1104 generate the adjusted masked control latent ^̃^^^^^^comprising of adjusted masked input-oriented text instruction ^^^^^^^and the additional input-oriented prompt instruction:^^^^^^. In an embodiment, the visual feature recovery module 1106 same as the receiver side of the main branch is used to compute the recovered masked visual feature ^̂^^^^^. The Text encoding module 1108 same as the receiver side is used to compute the recovered masked text feature ^̂^^^^. All ^̂^^^^^, ^̂^^^^, and the adjusted masked control latent ^^^^^^^are fed into the control encoding1112 same as the receiver side of the control Branch, which computes the encoded masked control feature ^̂^^^^^^. Then, the reconstruction module 1110 same as the receiver side of the main Branch computes the baseline output image ^^#$^"using the encoded masked control feature ^̂^^^^^^, the recovered masked visual feature ^̂^^^^^, and the recovered masked text feature ^̂^^^^. In an embodiment, a control adjustment module 1114 adjusts the adjusted masked control latent ^̃^^^^^^based on the reconstructed baseline output image ^^#$^"and the original input image x. Finally, after the control adjustment, the adjusted masked control latent ^̃^^^^^^finally is transmitted to the control encoding module 1116 of the receiver side.
[0104] FIG. 12 illustrates a processing workflow 1200 of a control adjustment module according to an embodiment of the present disclosure. As shown in FIG.12, the distortion between the original input x and the baseline output image ^^#$^"is used by a compute loss & perform update module 1202 to compute a distortion loss (e.g., MSE), which further updates the text description and prompts of the adjusted masked control latent ^̃^^^^^^, e.g., through backpropagating the gradient of the loss automatically. In somemanualAtty. Docket No.4502-81901 (6000641PCT02) adjustments can be performed to change the text description and prompts of the adjusted masked control latent ^̃^^^^^^, by observing the original input x and the baseline output image ^^#$^". In some other direct manipulation can be performed over the encoded image embedding, text embedding, and / or prompt embedding to change the adjusted masked control latent ^̃^^^^^^, e.g., through random noise injection. the control adjustment module 1200 can take various strategies to update the adjusted masked control latent ^̃^^^^^^. One exemplary structure of the control adjustment module 1200 is a multimodal encoder 1204 followed by a multimodal generator 1206. The multimodal encoder 1204 takes as input the image, the text description, and the other prompts, and computes the encoded image embedding, text embedding, and prompt embedding in the multimodal vision- language space. In an embodiment, the multimodal generator 1206 uses the encoded image embedding, text embedding, and prompt embedding to generate the image, text description, and prompts.
[0106] FIG. 13A illustrates a processing workflow of a diffusion branch according to an embodiment of the present disclosure. In general, the diffusion branch 1300A uses a conditional diffusion model (CDM) to provide, with little transmission bit costs, fidelity and expressiveness details from the original input image x to supplement the reconstructed output from the main branch such that the final output ^^ is authentic to the original input x. Further, in the depicted embodiment, the degradation module 1302 includes a down sampling module 1304 and an encoding module 1306. The original input x is first down sampled by the down sampling module 1304 to produce smaller resolutions that are then encoded by the encoding module 1306 into a diffusion latent feature ^^^^^^using information from the adjusted masked sparse visual feature ^̃^^^^^. In some embodiments, the down sampling module 1304 can use a learned down sampling network or a preset method like bicubic filter. In some embodiments, the encoding module 1306 uses a compression method (e.g., traditional coding tools like VVC / HEVC / JPEG, or an LIC method). In some embodiments, the adjusted masked sparse visual feature ^̃^^^^^guides theAtty. Docket No.4502-81901 (6000641PCT02) down sampling module 1304 and the encoding module 1306. For example, masked-out regions can be down sampled less and more bits can be assigned there, while unmasked regions can be down sampled more with less bits assigned such that bits can be better allocated to preserve the content of the input x. The down sampling module 1304 and the encoding module 1306 can also choose to down sample or encode all regions equally. Overall, a high compression rate is often used in the encoding module 1306 and the diffusion latent feature ^^^^^^has a small bitrate for transmission (further compressed by quantization and arithmetic coding).
[0107] On the receiver side, the received decoded diffusion latent feature ^̂^^^^^(usually after arithmetic decoding and dequantization) is used to compute theoutput ^^^%through a diffused decompression module 1308. In one embodiment, ^̂^^^^^is fed into a pixel recovery module 1310. The pixel recovery module 1310 is configured to recover a reconstructed image ^^^^. The pixel recovery module 1310 is the image decoding process of the corresponding image encoding process used by the encoding module 1306 on the sender side. A latent embedding module 1312 is configured to compute an embedded latent feature ^̂^^^based on the reconstructed image ^^^^. Then, using either the baseline output image ^^#$^"(corresponding to the workflow of FIG. 5A) or the recovered masked visual feature ^̂^^^^^and the recovered masked text feature ^̂^^^^(corresponding to the workflow of FIG.5B) as the diffusion condition, a reverse diffusion module 1314 generates the supplementary output ^^^%based on the embedded latent feature ^̂^^^. The latent embedding module 1312 is an encoder network such as the encoder part of a VAE.
[0108] FIG. 13B illustrates a processing workflow of a diffusion branch according to an embodiment of the present disclosure. On the sender side, the diffusion branch 1300B is the same as the diffusion branch 1300A in FIG.13A. On the receiver side, as shown in FIG.13B, ^̂^^^^^is fed into a latent transform module 1316. The latent transform module 1316 is configured to compute the embedded latent feature ^̂^^^. In general, the latent transform module 1316 perform enhancement over ^̂^^^^^by increasing the resolution and feature channel and obtain ^̂^^^. InAtty. Docket No.4502-81901 (6000641PCT02) some embodiments, the latent transform module 1316 can be eliminated and ^̂^^^is the same as ^̂^^^^^. Then, similar to FIG. 13A, the reverse diffusion module 1314 uses the embedded latent ^̂^^^and either, the baseline output image ^^#$^"(corresponding to the workflow of FIG. 5A) or the recovered masked visual feature ^̂^^^^^and the recovered masked text feature ^̂^^^^(corresponding to the workflow of FIG.5B) to generate the supplement output ^^^%.
[0109] The reverse diffusion module in FIG. 13A and FIG. 13B can use any diffusion processes, including a denoising diffusion probabilistic model (DDPM) or the updated denoising diffusion implicit model (DDIM). The reverse diffusion module can operate in the pixel domain as the DDPM or in the latent domain as latent diffusion model (LDM). In the case of the DDPM, the embodiment of FIG.10A is used where the latent embedding module is skipped. In such a case the reconstructed image ^^^^is directly fed into the reverse diffusion module to compute the supplement output ^^^%.
[0110] In an embodiment, when the encoding module 1306 is a traditional compression method like VVC / HEVC / JPEG, the framework in FIG.13A will be used where the pixel recovery module 1310 is the corresponding decoding process of the compression method to compute the reconstructed image ^^^^. When the encoding module 1306 is an LIC method, the pixel recovery module 1310 can be the corresponding decoding process of the LIC method in framework of FIG. 13A, or the framework of FIG.13B can be used where the intermediate decoded feature from the LIC can be directly transformed into ^̂^^^. The present disclosure does not place any restrictions on what compression method to use or what intermediate decoded feature to use.
[0111] FIG. 14A and FIG. 14B illustrate a reverse diffusion module according to two embodiments of the present disclosure. The reverse diffusion module of FIG. 14A and FIG. 14B are examples of the reverse diffusion module implemented in FIG. 13A and FIG. 13B that use a CDM for generating supplement image detail. The reverse diffusion module 1400A of FIG. 14A and 1400B of FIG.14B include a conditioning module 1402 and a reverse prediction module 1104.
[0112] The conditioning module 1402, given the baseline output image ^^#$^"(corresponding to the workflow of FIG. 5A) or the recovered masked visual feature ^̂^^^^^and the recoveredAtty. Docket No.4502-81901 (6000641PCT02) masked text feature ^̂^^^^(corresponding to the workflow of FIG.5B), is configured to compute a diffusion condition ^^. In some embodiments, when ^^#$^"is used, the conditioning module 1402 is an embedding network to encode the baseline image output ^^#$^"from the pixel domain into a latent (e.g., with the same dimensionality as the embedded latent feature ^̂^^^). In an embodiment, when ^̂^^^^^and ^̂^^^^is used as a diffusion condition, the conditioning module 1402 is a transformation network configured to combine ^̂^^^^^and ^̂^^^^and transform the combined result to the desired dimension (e.g., same as ^̂^^^). Then, a reverse prediction module 1404 is configured to compute the reverse diffusion step KL(^^^^M|^^^, NL(^^^, ^^)) for conditional DDPM or the reverse diffusion step KL(^̂^^M|^̂^, NL(^̂^, ^^)a total of T iterations is taken, and t=1, …, T. T can be pre-set, or can be determined for each input x. Alternatively, T can be determined on the receiver side or can be determined on the sender side and sent to the receiver side together with ^^^^^^^ ^!". For DDPM, as shown in FIG. 14A, after T iterations, ^^Ycan directly be used as output ^^^%. For LDM as shown in FIG.14B, after T iterations, the final ^̂Yis further processed by a decoding network 1406 (e.g., the up sampling part of a UNet) to generate the final output ^^^%. The output ^^#$^"from the main branch can be seen as an initial estimate of ^^Y^"^^= ^^#$^", which is combined with the output ^^^%from the diffusion branch to generate the final output ^^. In some embodiments, the fusion module 1408 can be simply an addition, i.e., ^^ = ^^^%+ ^^#$^". Other interpolation network can be used in the fusion model 1408 to further enhance the combination result.
[0113] Without loss of generalization, the CDM can be seen as using main branch to compute a deterministic initial estimate ^^Y^"^^= ^^#$^", and use this initial estimate (or the latent feature ^̂^^^^^and ^̂^^^^that generates ^^#$^") as conditions to guide the diffusion process to generate the residual details to supplement the initial estimate ^^#$^". The CDM method reduces the complexity of the diffusion task by switching the target from generating a whole natural image to generating the residual of an image, and provides robustness in controlling the generation processAtty. Docket No.4502-81901 (6000641PCT02) to recover the content of the original image. For the purpose of reducing the generation complexity, a similar CDM has been used for text-to-speech generation.
[0114] In some embodiments, the reverse prediction module can take the original score-based diffusion models using ordinary differential equation (ODE) or the consistency diffusion models based on probability-flow ordinary differential equation (PF-ODE). The number of iterations T can vary between a single step to many steps, i.e., T ≥ 1.
[0115] FIG.15 is a diagram illustrating an apparatus 1500 according to an embodiment of the present disclosure. The apparatus 1500 can be used to implement embodiments of the present disclosure such as, but not limited to, an encoder or a decoder. For example, the apparatus 1500 may be configured to perform the functions of an encoder or a decoder according to any of the embodiments shown in FIG.5A - FIG.14B. The apparatus 1500 includes receiver units (RX) 1520 or receiving means for receiving data via ingress ports 1510. The apparatus 1500 also includes transmitter units (TX) 1540 or transmitting means for transmitting via data egress ports 1550. For example, on the sender side, the encoder may use the RX 1520 or receiving means to obtain an original image and / or control instruction, and then use the TX 1540 or transmitting means for transmitting encoded image information (e.g., the vision-language latent feature, diffusion latent feature, and control latent requirement) to the receiver sider. On the receiver side, the decoder may use the RX 1520 or receiving means to obtain the encoded image information, and then use the TX or transmitting means for transmitting the decoded image of the original image (e.g., the final output ^^) to a display device or to another computing device.
[0116] The apparatus 1500 includes a memory 1560 or data storing means for storing the instructions and various data. The memory 1560 can be any type of, or combination of, memory components capable of storing data and / or instructions. For example, the memory 1560 can include volatile and / or non-volatile memory such as read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM). The memory 1560 can also include one or more disks, tape drives, and solid-state drives. In some embodiments, the memory 1560 can be used as an over-flow data storage device to storeAtty. Docket No.4502-81901 (6000641PCT02) programs when such programs are selected for execution, and to store instructions and data that are read during program execution.
[0117] The apparatus 1500 has one or more processors 1530 or other processing means (e.g., central processing unit (CPU)) to process instructions. The one or more processors 1530 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field- programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The one or more processors 1530 are communicatively coupled via a system bus with the ingress ports 1510, RX 1520, TX 1540, egress ports 1550, and memory 1560. The one or more processors 1530 can be configured to execute instructions stored in the memory 1560. Thus, the one or more processors 1530 provide a means for performing any computational, comparison, determination, initiation, configuration, or any other action corresponding to the claims when the appropriate instruction is executed by the processor. In some embodiments, the memory 1560 can be memory that is integrated with the processor 1530.
[0118] In one embodiment, the memory 1560 stores a LIC by MAVLM module 1570. The LIC by MAVLM module 1570 includes data, executable instructions, and / or one more sub-modules for implementing the disclosed embodiments. Thus, the inclusion of the LIC by MAVLM module 1570 substantially improves the functionality of the apparatus 1500.
[0119] Embodiments of the present disclosure provide at least the following technical advantages.
[0120] High compression rate with high-quality image generation at the same time by using the powerful multi-modal VLM representation through AIGC. The main branch employs a learned sparse vision-language representation that consists of integers and texts. Such a representation is highly efficient for transmission. The VLM models the joint distribution of the sparse codebook- based image representation and the corresponding text descriptions. Trained over a large scale of image-text data pairs, the VLM enables more abundant features from both image and text domains to better describe the input image than using image domain alone. The compression performance can be largely improved comparing to previous LIC methods that learn models in the imageAtty. Docket No.4502-81901 (6000641PCT02) domain solely.
[0121] MAVLM for learning generic and robust sparse vision-language representation. The present disclosure extends masked image modeling to masked image and text modeling, where an VL Transformer connects the sparse vision and language features through cross-modality prediction, and at the same time improves the robustness of the learned visual and text feature representations through masked prediction.
[0122] Flexible quality control: The diffusion branch provides supplement fidelity and expressive details extracted from the current input image to improve the reconstruction fidelity to the original input. Such details can be selectively added, and the quality of such details can be flexibly adjusted according to practical conditions like computation power, time requirements, quality requirements, and so on. For example, for low computation power with strict time constraint, such details may be skipped to deliver a decompressed output through main branch alone with one inference pass, and the output can be less authentic to the original input. When the target is to deliver a high-quality high-fidelity output and the computation power or time is not a concern, many diffusion iterations can be taken to add rich details to the output.
[0123] Flexible task-oriented control through prompts: The control branch uses the Prompt VLM that models the multimodal embedded representation between text, images and various forms of prompts to enable guided compression using prompt commands. The control signal can take a default form (e.g., to ensure fidelity or ensure perceptual quality), or can be set to accommodate a specific compression target (e.g., to emphasize on a specific object so that the object can be reconstructed in a certain way). The transmitted control latent can be automatically or manually adjusted to reduce an online loss.
[0124] In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items shown or discussed as coupled or directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediateAtty. Docket No.4502-81901 (6000641PCT02) component whether electrically, mechanically, or otherwise. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and may be made without departing from the spirit and scope disclosed herein.
Claims
Atty. Docket No.4502-81901 (6000641PCT02) CLAIMS What is claimed is:
1. A method implemented by an encoder, comprising: encoding an original image into a sparse visual feature; generating a sparse text feature indicating content of the original image; computing, based on a control signal and the original image, a control latent feature; computing, based on the sparse visual feature, the sparse text feature, and the control latent feature, an adjusted masked sparse visual feature, an adjusted masked sparse text feature, and an adjusted masked control latent feature, respectively; computing a diffusion latent feature; and transmitting the adjusted masked sparse visual feature, the adjusted masked sparse text feature, the adjusted masked control latent, and the diffusion latent feature to a decoder.
2. The method of claim 1, further comprising computing, based on the original image, the adjusted masked sparse visual feature, and the adjusted masked control latent, the diffusion latent feature, wherein the diffusion latent feature captures fidelity and expressiveness details of the original image.
3. The method of any of claims 1-2, wherein the control latent feature indicates an encoded control requirement.
4. The method of any of claims 1-3, wherein computing the control latent feature comprises: computing, based on the control signal, a text instruction, wherein the text instruction is a text description describing control requirements of the control signal; computing, based on the text instruction, the original image, the control signal, and a prompt visual-language model (VLM), an input-oriented text instruction and additional input- oriented prompt instruction; andAtty. Docket No.4502-81901 (6000641PCT02) computing, based on the input-oriented text instruction and the additional input-oriented prompt instruction, the control latent feature.
5. The method of any of claims 1-4, wherein the original image is a general three-dimensional (3D) tensor with shape ^ × ℎ × ^, where w, h, c are a width, a height, and a number of channels of an image, and wherein encoding the original image into the sparse visual feature comprises: encoding the original image into a vision feature tensor with shape ^^× ℎ^× ^, wherein width ^^and height ℎ^depend on the width and the height of the original image, and wherein d is a number of feature channels; and computing, based on the vision feature tensor and a vision codebook, a sparse visual feature, wherein the vision codebook comprises a plurality of codewords, and wherein each codeword has d dimension.
6. The method of any of claims 1-5, wherein encoding the original image into the vision feature tensor comprises dividing, using a visual transformer, the original image into patches and encoding the patches as a sequence.
7. The method of any of claims 1-5, wherein encoding the original image into the vision feature tensor comprises encoding in a parallel manner, using a convolutional neural network (CNN), the original image as an entire image.
8. The method of any of claims 1-7, wherein generating the sparse text feature comprises: computing, based on the original image, a text latent feature comprising text description, wherein the text description describes the content of the original image; and generating, based on the text latent feature, the sparse text feature.
9. The method of any of claims 1-8, wherein computing the text latent feature comprises:Atty. Docket No.4502-81901 (6000641PCT02) computing an enriched text latent feature; and computing, based on the enriched text latent feature and / or a text codebook, the text latent feature, wherein the text codebook comprises a plurality of codewords, wherein each codeword has natural language elements.
10. The method of any of claims 1-9, wherein computing, based on the sparse visual feature, the adjusted masked sparse visual feature comprises computing, based on the sparse visual feature, a masked sparse visual feature by applying a visual mask to the sparse visual feature to change corresponding tokens in the sparse visual feature according to the visual mask, wherein the visual mask is a binary mask having the same shape as the sparse visual feature, and wherein the tokens having value 0 are set to a designated value indicating being removed; and computing, based on the masked sparse visual feature and using a visual-language (VL) transformer, the adjusted masked sparse visual feature.
11. The method of any of claims 1-10, wherein computing, based on the sparse text feature, the adjusted masked sparse text feature comprises: computing, based on the sparse visual feature, a masked sparse text feature by applying a text mask to the sparse text feature to change corresponding tokens in the sparse text feature according to the text mask, wherein the text mask is a binary mask having the same shape as the sparse text feature, and wherein the tokens having value 0 are set to a designated value indicating being removed; and computing, based on the masked sparse text feature and using a visual-language (VL) transformer, the adjusted masked sparse text feature.
12. The method of any of claims 1-11, further comprising computing, based on the control latent feature, the adjusted masked control latent by applying a visual mask to the control latent.Atty. Docket No.4502-81901 (6000641PCT02) 13. The method of any of claims 1-12, wherein the adjusted masked control latent comprises an adjusted masked input-oriented text instruction and an adjusted masked additional input- oriented prompt instruction.
14. The method according to any of claims 1-13, wherein computing the diffusion latent feature comprises: downsampling the original image to smaller resolution images; and encoding the smaller resolution images to obtain the diffusion latent feature using information from the adjusted masked sparse visual feature.
15. A method implemented by a decoder, comprising: receiving an adjusted masked sparse visual feature, an adjusted masked sparse text feature, an adjusted masked control latent, and a diffusion latent feature of an original image; generating, based on the received adjusted masked sparse visual feature, a recovered masked visual feature; computing, based on the received adjusted masked sparse text feature, an encoded masked text feature; computing, based on the adjusted masked control latent, the recovered sparse visual feature, and the encoded masked sparse text feature, an encoded masked control feature; reconstructing, based on the recovered masked visual feature, the encoded masked text feature, and the encoded masked control feature, a baseline image output; computing, based on the diffusion latent feature and the baseline image output, a supplementary output; and constructing, based on the supplementary output and the baseline image output, a final decoded image output.Atty. Docket No.4502-81901 (6000641PCT02) 16. A method implemented by a decoder, comprising: receiving an adjusted masked sparse visual feature, an adjusted masked sparse text feature, an adjusted masked control latent, and a diffusion latent feature of an original image; generating, based on the received adjusted masked sparse visual feature, a recovered masked visual feature; computing, based on the received adjusted masked sparse text feature, an encoded masked text feature; computing, based on the adjusted masked control latent, the recovered sparse visual feature, and the encoded masked sparse text feature, an encoded masked control feature; reconstructing, based on the recovered masked visual feature, the encoded masked text feature, and the encoded masked control feature, a baseline image output; computing, based on the diffusion latent feature, the masked sparse visual feature, and the masked sparse text feature, a supplementary output; and constructing, based on the supplementary output and the baseline image output, a final decoded image output.
17. The method according to any of claims 15-16, wherein the recovered masked visual feature is based on a vision codebook, and wherein the encoded masked sparse text feature is based on a text codebook.
18. The method according to any of claims 15-17, wherein computing the supplementary output comprises: recovering, based on the diffusion latent feature, a reconstructed image; computing an embedded latent feature based on the reconstructed image; and computing, based on the embedded latent feature and using the baseline image output as a diffusion condition, the supplementary output.Atty. Docket No.4502-81901 (6000641PCT02) 19. The method according to any of claims 15-17, wherein computing the supplementary output comprises: recovering, based on the diffusion latent feature, a reconstructed image; computing an embedded latent feature based on the reconstructed image; and computing, based on the embedded latent feature and using the recovered masked visual feature and the encoded masked text feature as a diffusion condition, the supplementary output.
20. The method according to any of claims 15-17, wherein computing the supplementary output comprises: computing an embedded latent feature based on the diffusion latent feature; and computing, based on the embedded latent feature and using the baseline image output as a diffusion condition, the supplementary output.
21. The method according to any of claims 15-17, wherein computing the supplementary output comprises: computing an embedded latent feature based on the diffusion latent feature; and computing, based on the embedded latent feature and using the recovered masked visual feature and the encoded masked text feature as a diffusion condition, the supplementary output.
22. The method according to any of claims 15-17, further comprising computing the supplementary output using a Denoising Diffusion Probabilistic Model (DDPM) or a Denoising Diffusion Implicit Model (DDIM).
23. The method according to any of claims 15-22, wherein computing the supplementary output comprises encoding, using an embedding network, the baseline image output from a pixel domain to a latent domain when the baseline image output is used as the diffusion condition.Atty. Docket No.4502-81901 (6000641PCT02) 24. The method according to any of claims 15-23, wherein computing the supplementary output comprises transforming, using a transformation network, a combined result to a dimension corresponding to the embedded latent feature when the recovered masked visual feature and the encoded masked text feature is used as a diffusion condition.
25. An encoder, comprising: a memory or storage means configured to store instructions; and one or more processors or processing means coupled to the memory or the storage means and configured to execute the instructions to cause the encoder to perform a method according to any of claims 1-14.
26. A decoder, comprising: a memory or storage means configured to store instructions; and one or more processors or processing means coupled to the memory or the storage means and configured to execute the instructions to cause the decoder to perform a method according to any of claims 15-24.
27. A computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computer-executable instructions when executed by one or more processor of an apparatus, cause the apparatus to perform a method according to any of claims 1-27.
Citation Information
Patent Citations
Learned image compression by ai generated content
WO2024124261A2
Cited By
Video extension method and device
CN119364149A
Model training method, storage medium, electronic equipment and program product
CN121121346A
Cross-modal image-text analysis method for machine vision
CN121210958A
Virtual sonar image generation method and system
CN121414871A
Virtual sonar image generation method and system
CN121414871B
