Encoding and / or decoding apparatus, method and system for processing images

Abstract

The invention discloses an encoding and / or decoding device and method for processing an image. The encoding method includes extracting image encoding features from an image, quantizing the image encoding features based on a potential embedding space to convert the image encoding features into discrete potential variables, sampling the discrete potential variables using a sampling mask associated with a sampling factor to obtain sampling potential variables, and compressing the sampled potential variables in order to generate a bitstream related to the image. The decoding method includes decoding a bitstream associated with an image to obtain a sampling potential variable, and processing the sampling potential variable at least through a back diffusion process using a vector quantization-based diffusion model to reconstruct the image.

Description

Technical Field

[0001] The present invention relates to a coding and / or decoding device, method and system for processing images. Background Art

[0002] Image compression is a signal processing operation that facilitates efficient image transmission and storage. Known image compression standards include Joint Photographic Experts Group (JPEG), Joint Photographic Experts Group 2000 (JPEG2000), HEVC / H.265, Versatile Video Coding (VVC), etc. Most of these standards rely on encoder / decoder frameworks customized with block-based coding. Although these standards have been widely used, the demand for further improvement in coding performance is growing.

[0003] In recent years, deep image compression (image compression based on deep learning) has begun to gain attention. Deep image compression mainly includes image compression based on autoencoders and generative image compression. Generally speaking, in image compression based on autoencoders, the encoder learns to generate a compact representation of the input image, and the decoder reconstructs an image close to the input image based on the compact representation. In some examples, the variational autoencoder (VAE) related to the autoencoder (AE) model focuses on learning representations with continuous features. In some examples, the vector quantized variational autoencoder (VQ-VAE) quantizes potential features into integer labels to obtain a compact representation of the image. On the other hand, generative image compression generally compresses images using a generative model. In some examples, it can exhibit high compression efficiency and obtain visually good results even at lower bit rates.

[0004] More emerging methods include semantic compression with cross-modal representations, which aim to improve the compression efficiency of different methods. As an example, Li et al. proposed a cross-modal compression framework in the article entitled "Cross modal compression: Towards human-comprehensible semantic compression" (2021), which combines image-to-text (I2T) translation and text-to-image (T2I) generative models to achieve human-understandable results. However, this approach mainly emphasizes preserving semantic consistency rather than signal-level reconstruction. To address this limitation, as an example, Zhang et al. proposed a scalable cross-modal compression (SCMC) paradigm in the article entitled "Rethinking semantic image compression: Scalable representation with cross-modality transfer" (2023), which uses different modalities to hierarchically outline images to achieve semantic, structural, and signal-level reconstruction at different stages. Although these methods have achieved good results, most of these methods rely on using another modality (such as text) to directly represent visual scenes, so that fine-grained scalability and interactivity are not well supported. Summary of the invention

[0005] In a first aspect, the present invention provides a coding device. The coding device includes an image encoder, a quantization module, a sampling module and a compression module. The image encoder is used to extract image coding features from an image. The quantization module is used to quantize the image coding features based on a potential embedding space to convert the image coding features into discrete latent variables. The sampling module is used to sample the discrete latent variables using a sampling mask associated with a sampling factor to obtain sampled latent variables. The compression module is used to compress the sampled latent variables. The compressed sampled latent variables can be used to generate a bit stream associated with the image. Preferably, the sampling factor is adjustable.

[0006] In some embodiments of the first aspect, the sampling module is also used to generate a sampling mask, which includes: extracting a mask from the image to separate foreground information and background information related to the image, processing the extracted mask to obtain an anchor mask, generating an interval mask according to the sampling factor, and combining the anchor mask and the interval mask to generate a sampling mask.

[0007] In some implementations of the first aspect, the sampling module adjusts the size of the extracted mask and binarizes the adjusted mask to process the extracted mask.

[0008] In some embodiments of the first aspect, at least the image encoder and the latent embedding space are trained by a loss function.

[0009] In some embodiments of the first aspect, the training performed by the loss function is diffusion model training based on vector quantization.

[0010] In some embodiments of the first aspect, when training the vector quantization based diffusion model, parameters of the image encoder and the latent embedding space are fixed.

[0011] In some embodiments of the first aspect, the sampling module uses a saliency detection algorithm to separate foreground information and background information associated with discrete latent variables.

[0012] In some embodiments of the first aspect, the mask extracted from the image is a saliency mask, which is used to localize semantic information in the feature space, thereby separating foreground elements from background elements.

[0013] In some embodiments of the first aspect, the interval mask is an interval mask that is controlled by a sampling factor, wherein a larger sampling factor represents a larger interval and fewer latent variables are sampled.

[0014] In a second aspect, the present invention provides a decoding device. The decoding device includes an image decoder and a diffusion module based on vector quantization. The image decoder is used to decode a bit stream associated with an image to obtain sampled latent variables. The diffusion module based on vector quantization is used to process the sampled latent variables at least through a back-diffusion process to reconstruct the image.

[0015] In some embodiments of the second aspect, the vector quantization based diffusion module includes a semantic information input for editing the background of the image to be reconstructed.

[0016] In some embodiments of the second aspect, the vector quantization-based diffusion module is used to process the sampled latent variables through a forward diffusion process and a backward diffusion process to reconstruct the image (for example, when training the vector quantization-based diffusion module), and the vector quantization-based diffusion module uses a Markov chain in the forward diffusion process to sample the data in sequence and gradually introduce noise into the data.

[0017] In some implementations of the second aspect, the diffusion module based on vector quantization is a Gaussian transfer distribution at each time step in the forward diffusion process.

[0018] In some embodiments of the second aspect, after a fixed number T of time steps have passed, the vector quantization-based diffusion module generates a latent variable sequence x1, ..., x with a gradually increasing noise level in the forward diffusion process. T .

[0019] In some embodiments of the second aspect, the decoding device further includes a denoising network module for approximating the conditional transfer distribution in is the sampling latent variable.

[0020] In a third aspect, the present invention provides a coding and decoding system. The coding and decoding system comprises the coding device of the first aspect and the decoding device of the second aspect. Preferably, the potential embedding space is shared by the coding device and the decoding device.

[0021] In a fourth aspect, the present invention provides a coding method. The coding method comprises: extracting image coding features from an image, quantizing the image coding features based on a potential embedding space to convert the image coding features into discrete latent variables, sampling the discrete latent variables using a sampling mask associated with a sampling factor to obtain sampled latent variables, and compressing the sampled latent variables. The compressed sampled latent variables can be used to generate a bitstream associated with the image. Preferably, the sampling factor is adjustable.

[0022] In some embodiments of the fourth aspect, the encoding method also includes generating a sampling mask, which includes: extracting a mask from the image to separate foreground information and background information related to the image, processing the extracted mask to obtain an anchor mask, generating an interval mask based on the sampling factor, and combining the anchor mask and the interval mask to generate a sampling mask.

[0023] In some implementations of the fourth aspect, processing the extracted mask to obtain the anchor mask includes: adjusting the size of the extracted mask and binarizing the adjusted mask to obtain the anchor mask.

[0024] In some embodiments of the fourth aspect, the mask extracted from the image is a saliency mask, which is used to localize semantic information in the feature space, thereby separating foreground elements from background elements.

[0025] In some embodiments of the fourth aspect, the interval mask is an interval mask that is controlled by a sampling factor, where a larger sampling factor represents a larger interval and fewer latent variables are sampled.

[0026] In a fifth aspect, the present invention provides a decoding method, which comprises: decoding a bit stream associated with an image to obtain sampled latent variables, and processing the sampled latent variables by at least a back-diffusion process using a diffusion model based on vector quantization to reconstruct the image.

[0027] In some embodiments of the fifth aspect, the vector quantization based diffusion model includes a semantic information input for editing the background of the image to be reconstructed.

[0028] In some embodiments of the fifth aspect, reconstructing an image using a vector quantization-based diffusion model includes using a vector quantization-based diffusion model to process sampled latent variables through a forward diffusion process and a backward diffusion process to reconstruct the image (for example, when training a vector quantization-based diffusion model), and the forward diffusion process uses a Markov chain to sample data in sequence and gradually introduce noise into the data.

[0029] In some embodiments of the fifth aspect, each time step of the forward diffusion process is a Gaussian transfer distribution.

[0030] In some embodiments of the fifth aspect, after a fixed number T of time steps have passed, the forward diffusion process generates a latent variable sequence x1, ..., x with a gradually increasing noise level. T .

[0031] In some embodiments of the fifth aspect, the decoding method further comprises approximating the conditional transfer distribution in is the sampling latent variable.

[0032] In a sixth aspect, the present invention provides a coding device. The coding device includes: one or more processors and a memory. The memory stores one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for executing the coding method of the fourth aspect.

[0033] In a seventh aspect, the present invention provides a decoding device. The decoding device includes: one or more processors and a memory. The memory stores one or more programs configured to be executed by the one or more processors. The one or more programs include instructions for executing the decoding method of the fifth aspect.

[0034] In an eighth aspect, the present invention provides a coding and decoding system. The coding and decoding system comprises the coding device of the sixth aspect and the decoding device of the seventh aspect. Preferably, the potential embedding space is shared by the coding device and the decoding device.

[0035] In a ninth aspect, the present invention provides a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors, wherein the one or more programs include instructions for executing the encoding method of the fourth aspect.

[0036] In a tenth aspect, the present invention provides a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors, wherein the one or more programs include instructions for executing the decoding method of the fifth aspect.

[0037] By considering the detailed description and the accompanying drawings, other features and aspects of the present invention will become apparent. Where appropriate and applicable, any one or more features described herein for one aspect or embodiment may be combined with any one or more other features described herein for any one or more other aspects or embodiments.

[0038] Terms of degree or relative terms relating to quantities or conditions (e.g., "generally," "approximately," "substantially," etc.) are used to take into account (depending on the context) at least one of the following: manufacturing tolerances, degradation, assembly, use, trends, tendencies, actual imperfections, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Embodiments of the present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0040] Figure 1 is a conceptual diagram illustrating fine-grained encoding of some embodiments of the present invention;

[0041] Figure 2 is a schematic diagram illustrating a framework of fine-grained encoding of some embodiments of the present invention;

[0042] Figure 3 is a schematic diagram showing an example of stepwise sampling in an anchor point-based sampling strategy according to an embodiment of the present invention;

[0043] Figure 4 is a schematic diagram showing the overall framework of a vector quantization-based diffusion model for sampling latent variable reconstruction according to an embodiment of the present invention;

[0044] Figure 5 In one embodiment of the present invention, Figure 4 Schematic diagram of the structure of the transformer block in the vector quantization-based diffusion model;

[0045] Fig. 6A is a chart showing the rate-distortion performance (Deep Image Structure and Texture Similarity (DI STS)) of a model according to an embodiment of the present invention and some existing models in an example experiment of processing a CUB-200-2011 dataset;

[0046] Figure 6B is a chart showing the rate-distortion performance (learning-perceptual image patch similarity (LPI PS)) of a model according to an embodiment of the present invention and some existing models in an example experiment of processing a CUB-200-2011 dataset;

[0047] Figure 6Cis a chart showing the rate-distortion performance (accuracy) of a model according to an embodiment of the present invention and some existing models in an example experiment of processing a CUB-200-2011 dataset;

[0048] Fig.6D is a graph showing the rate-distortion performance (Deep Image Structure and Texture Similarity (DI STS)) of a model according to an embodiment of the present invention and some existing models in an example experiment of processing an Oxford flower dataset;

[0049] Fig. 6E is a graph showing the rate-distortion performance (Learning Perceptual Patch Similarity (LPI PS)) of a model according to an embodiment of the present invention and some existing models in an example experiment of processing an Oxford flower dataset;

[0050] Fig. 6F is a chart showing the rate-distortion performance (accuracy) of a model according to an embodiment of the present invention and some existing models in an example experiment of processing an Oxford flower dataset;

[0051] Figure 7 1 is a diagram showing a visual comparison of images obtained by a model according to an embodiment of the present invention and some existing models in an example experiment of processing a CUB-200-2011 dataset and an Oxford flower dataset (the value below each image is bpp / DISTS, where a lower DISTS value represents better perceived quality);

[0052] Figure 8 is a schematic diagram showing the structure of a transformer block for semantic information according to one embodiment of the present invention;

[0053] Fig. 9 1 is a diagram showing the results of semantic background generation in an example experiment of processing a CUB-200-2011 dataset and an Oxford flower dataset (the left column is the results obtained from the CUB-200-2011 dataset, and the right column is the results obtained from the Oxford flower dataset);

[0054] Fig. 10A is a chart showing the rate-distortion performance (Deep Image Structure and Texture Similarity (DIST)) of different example models of the present invention with different sampling strategies in an example ablation experiment processed on a CUB-200-2011 dataset;

[0055] Fig. 10B is a chart showing the rate-distortion performance (accuracy) of different example models with different sampling strategies of the present invention in an example ablation experiment processed on a CUB-200-2011 dataset;

[0056] Fig.111 is a diagram showing a visual comparison of images obtained by different example models of the present invention with different sampling strategies in an example ablation experiment processed on a CUB-200-2011 dataset (the value below each image is DISTS, where a lower DISTS value represents a better perceived quality);

[0057] Fig.12 1 is a diagram showing a visual comparison of images obtained by an example model of the present invention at different sampling factors (1-5) in an example ablation experiment processed on a CUB-200-2011 dataset;

[0058] Fig.13 is a data processing system showing some embodiments of the present invention, which can be used to perform the methods of some embodiments of the present invention;

[0059] Fig.14 is an encoding method illustrating some embodiments of the present invention; and

[0060] Fig.15 The following is a decoding method illustrating some embodiments of the present invention. DETAILED DESCRIPTION

[0061] The present disclosure shows how multimodal learning can benefit compression, particularly in terms of reconstruction quality and in terms of interactivity with artificial intelligence generated content (AIGC) models. Content about artificial intelligence generated content (AIGC) models can be found in an article by Liu et al. entitled "Semantics-to-signal scalable image compression with learned revertible representations" (2021). In some embodiments of the present invention, a fine-grained encoding method is provided by exploring the ability to use AIGC for multimodal learning. In some embodiments of the present invention, a diffusion model based on vector quantization is utilized, and a sampling strategy based on anchor points maintains semantically important feature vectors and samples background latent variables. In some embodiments of the present invention, dynamic adjustment of the sampling factor can enable the model to achieve fine-grained scalable encoding by compressing different latent variables. In some embodiments of the present invention, semantic information characterized by cross-modal input is further embedded to facilitate the generation of a background with controllable factors conveyed by natural language. In this way, the background can be manipulated using AIGC.

[0062] Figure 1 The concept 100 of fine-grained encoding of some embodiments of the present invention is shown. Figure 1As shown, the input image is encoded by the encoder. Then, the encoding is processed using an anchor-based sampling strategy. The anchor-based sampling strategy uses different sampling factors S to process the encoding. The encoding processed by the anchor-based sampling strategy can be converted into a bitstream. To reconstruct the image, the bitstream can be processed using an AIGC module and a decoder. The processing takes into account semantic information. Figure 1 Models with different sampling factors S are shown. Figure 1 In , the final decoded image is generated under the condition that the semantic information is blue sky and the latent variable is S = ∞.

[0063] Some contributions of this disclosure include: In some embodiments of the present invention, a fine-grained encoding method is provided, which combines semantically meaningful encoding with multimodal generation to form a new encoding paradigm that supports visual goodness, semantically meaningfulness, scalability and interactivity at the same time. In some embodiments of the present invention, an anchor-based sampling strategy is provided to preserve semantically important feature vectors. This method can facilitate fine-grained scalable coding, protect semantically important features, and ensure multimodal context generation and editing. In some embodiments of the present invention, a new generative model for a decoder based on a diffusion model of vector quantization is provided. The decoder can effectively reconstruct latent variables through cross-modal input. Therefore, good performance can be obtained in downstream tasks and image perceptual reconstruction.

[0064] The inventors of the present invention have a certain understanding of image compression through research and experiments. These understandings are provided below to help understand the background of some embodiments of the present invention. In general, image compression aims to compactly represent image signals for transmission and storage. Due to the excellent learning ability of deep neural networks, image compression based on deep learning has made a lot of progress in recent years. Image compression based on learning significantly improves compression performance, proving the ability of neural networks to perform nonlinear modeling of visual signals and improve compression efficiency. Scalable compression allows the input signal to be encoded as an embedded bit stream, which can then be partially decoded for subsequent reconstruction. Many scalable coding standards (e.g., JPEG2000, H.264 / AVC scalable extension (scalable extension) internal mode, HEVC scalable extension (scalableextension) internal mode) have been developed to better adapt to real-world applications. In addition, in addition to traditional scalable coding, scalable schemes based on learning-based compression have been explored, which enable representations to be constructed in a hierarchical manner. In one example, a typical self-encoder structure with a hierarchical design concept is used in image compression, which consists of convolutional and deconvolutional long short-term memory recursive neural networks, so that a model can produce multiple compression rates. In one example, the scalable coding framework decomposes image representation into a base layer and an enhancement layer, where the base layer contains deep learned features that form the basis of the framework, while the enhancement layer focuses on reconstructing textures to improve the visual quality of the output.

[0065] The inventors of the present invention have a certain understanding of multimodal generation through research. These understandings are provided below to help understand the background of some embodiments of the present invention. In general, multimodal generation benefits from the generation ability of neural networks and the progress of visual and language models. In recent years, different methods have been developed to lay this foundation. Examples of these methods include image-to-text (I2T) translation, text-to-image (T2I) generation, and structure-to-image (S2I) generation. Multimodal generation has been applied to compression scenarios. In one example, cross-modal redundancy is utilized and a multimodal compression framework for infrared and visible light image compression is designed. In one example, a cross-modal compression framework for semantic communication combines an I2T translation model and a T2I generation model. However, this method focuses mainly on maintaining semantic consistency rather than signal-level reconstruction. In one example, a hierarchical concept image compression method uses potential coding and reconstructed edge maps to synthesize images. In one example, a scalable cross-modal compression paradigm involves drawing images using the same modality hierarchy.

[0066] The inventors of the present invention have a certain understanding of diffusion models through research. These understandings are provided below to help understand the background of some embodiments of the present invention. In general, probability diffusion models show good performance in image generation tasks and exhibit desirable properties, such as stable training. However, probability diffusion models currently focus mainly on continuous diffusion models applied to image pixels. On the other hand, the applicability of discrete diffusion models in image generation has also been explored. In one example, D3PM uses discrete diffusion to generate images. In one example, regarding its application in image compression, a transform coding-based neural image compression technique is proposed, which uses a diffusion model. The method uses a denoising decoder that iteratively reconstructs a compressed image encoded by an ordinary neural encoder. In one example, a new lossy compression method (called DiffC) is based on an unconditional diffusion generation model and allows progressive encoding, allowing partial bitstreams to be decoded. In one example, a text-to-image architecture (called VQ Diffusion) models the VQ-VAE latent space in a non-autoregressive manner.

[0067] Fig.14 The encoding method 1400 of some embodiments of the present invention is described above. The encoding method 1400 can be implemented using software and / or hardware. The method 1400 includes extracting image coding features from an image at step 1402. The method 1400 also includes quantizing the image coding features based on the potential embedding space to convert the image coding features into discrete latent variables at step 1404. The method 1400 also includes sampling the discrete latent variables using a sampling mask to obtain sampled latent variables at step 1406. Preferably, the sampling factor is adjustable. The method 1400 also includes compressing the sampled latent variables at step 1408. The compressed sampled latent variables can be used to generate a bitstream associated with the image.

[0068] In some implementations, method 1400 further includes receiving an image before step 1402 .

[0069] In some embodiments, method 1400 further includes generating a sampling mask before step 1406. In some embodiments, generating the sampling mask before includes extracting a mask from the image to separate foreground information and background information associated with the image, processing the extracted mask to obtain an anchor mask, generating an interval mask according to a sampling factor, and combining the anchor mask and the interval mask to generate the sampling mask. In some embodiments, processing the extracted mask to obtain the anchor mask includes: binarizing the mask to obtain the anchor mask. In some embodiments, processing the extracted mask to obtain the anchor mask includes: adjusting the size of the extracted mask and binarizing the adjusted mask to obtain the anchor mask. In some embodiments, the mask extracted from the image is a salient mask, which is used to localize semantic information in the feature space, thereby separating foreground elements from background elements. In some embodiments, the interval mask is an interval mask, which is controlled by a sampling factor, wherein a larger sampling factor represents a larger interval and fewer potential variables are sampled.

[0070] Fig.15 The decoding method 1500 of some embodiments of the present invention can be implemented by software and / or hardware. The decoding method 1500 includes decoding a bit stream associated with an image to obtain sampled latent variables at step 1502. The method 1500 also includes processing the sampled latent variables by at least a back diffusion process using a diffusion model based on vector quantization to reconstruct the image at step 1504.

[0071] In some embodiments, the bitstream associated with the image in step 1502 is a bitstream obtained from the encoding method 1400 , and the image is an image encoded using the encoding method 1400 .

[0072] In some embodiments, the vector quantization based diffusion model includes a semantic information input, which is used to edit the background of the image to be reconstructed. In some embodiments, the forward diffusion process uses a Markov chain to sequentially sample data and gradually introduce noise into the data. In some embodiments, each time step of the forward diffusion process is a Gaussian transfer distribution. In some embodiments, after a fixed number of T time steps have passed, the forward diffusion process produces a latent variable sequence x1, ..., x with gradually increasing noise levels. T In some embodiments, the decoding method 1500 further includes approximating the conditional transition distribution in is the sampling latent variable.

[0073] Some more detailed embodiments of the present invention are provided below.

[0074] Some embodiments of the present invention propose a fine-grained encoding method that uses a vector quantization based diffusion model and an anchorbased sampling strategy. The anchorbased sampling strategy can effectively sample semantically important feature vectors while providing flexibility in sampling background feature vectors to achieve interactivity. The vector quantization based diffusion model as an AIGC model helps reconstruct the sampled latent variables.

[0075] Figure 2 The framework 200 of fine-grained coding of some embodiments of the present invention is shown, which includes the encoding process (from input image to bit stream generation) and the decoding process (from bit stream to reconstructed image). Figure 2 As shown, the main parts of the framework 200 include an encoder 202, a decoder 204, a shared embedding space 206, an anchor-based sampling strategy 208, and a vector quantization-based diffusion model 210. The anchor-based sampling strategy 208 is used to sample semantically important feature vectors while providing flexibility in sampling background feature vectors to achieve interactivity. The vector quantization-based diffusion model 210 can be used as an AIGC model to facilitate reconstruction of sampled latent variables.

[0076] exist Figure 2 In the embodiment of the present invention, the encoder 202 includes an image encoder, a quantization module (Q) and an entropy encoding module (AE) (for arithmetic encoding). The image encoder converts the image As input, where H, W, and C correspond to the height, width, and number of channels of the image, respectively. The image is processed by the image encoder to extract latent features Where h, w and c represent the height, width and number of channels of the encoded features respectively. Vector quantization is based on the latent embedding space where K is the size of the discrete latent space and D is the number of each latent embedding vector c i In this example, the shared embedding space 206 consists of K potential feature vectors, and the shared embedding space 206 is shared by the encoder 202 and the decoder 204. By performing a nearest neighbor search in the shared embedding space, the potential feature y can be converted into a discrete latent variable

[0077] At the same time, the anchor mask extraction module in the anchor-based sampling strategy 208 is used to extract the mask The anchor mask extraction module uses a salient detection algorithm to separate the foreground and background and help identify semantically important feature vectors. Subsequently, the size of the mask m is adjusted and binarized based on the anchor sampling strategy 208 to obtain the anchor mask m a In addition, the anchor-based sampling strategy 208 generates an interval mask m via an interval mask generator including a sampling factor i . Sampling mask is the anchor point mask m a and interval mask m i Combination of. Figure 1 Or as shown in 2, the sampling mask can be adjusted by the sampling factor Make adjustments. Sampling from latent variables to obtain sampled latent variables Then, z is compressed by lossless coding (arithmetic coding).

[0078] The decoder 204 mainly includes an image decoder, an arithmetic decoding module (AD), a dequantization module (DQ), and a diffusion model 210 based on vector quantization (although shown separately from the decoder 204, it can be regarded as a part of the decoder 204). In the decoding operation, the bit stream is decoded into sampled latent variables Then, all latent variables The reconstruction is performed via a vector quantization-based diffusion model. It should be noted that the semantic information is the input of the vector quantization-based diffusion model, which is used to edit the background. In order to reconstruct the latent features The corresponding latent embedding vector is selected in the latent embedding space during the dequantization operation. Then, the image decoder generates the reconstructed image

[0079] exist Figure 2 In the embodiment of the invention, encoder 202 is an important part of the process of compressing the image into a compact representation. While VAE is designed to learn representations with continuous latent features, VQ-VAE quantizes the latent features into indices (integer tokens), which produces better image reconstruction results. Figure 2 In the implementation, in order to obtain a compact image representation, the design philosophy of VQ-VAE is adopted. Figure 2 In the implementation scheme, the image encoder, image decoder and embedding space inherit the ideal properties of the VQ-VAE model (i.e., having high compression efficiency and being able to obtain compact image representation).

[0080] exist Figure 2 In the implementation method, the autoencoder model is first learned, which includes an image encoder and an image decoder. The image encoder is embedded in the latent space The image is compressed into discrete latent variables, and the image decoder reconstructs the image from the latent variables. Specifically, the input image I is applied to the image encoder E(·) to obtain y. Then, vector quantization Q(·) is used to map the latent embedding vector to its The closest entry in c k Then, the image is reconstructed through the image decoder, that is, in is the dequantized latent feature. Therefore, image compression can be viewed as a process of sampling latent variables from the shared embedding space 206 and compressing the quantized latent variables.

[0081] To train the image encoder E(·), image decoder D(·), and the shared latent embedding space, Figure 2 The loss function is used in the implementation This minimizes the reconstruction error and ensures that the encoded latent features are quantized to The closest entry in: Where sg[·] represents the stop gradient operation and β is the weight. When training the vector quantization based diffusion model 210, the parameters of the image encoder, image decoder and latent embedding space are fixed.

[0082] exist Figure 2 In the embodiment of , an anchor-based sampling strategy 208 is adopted. This method retains semantically important feature vectors when sampling background latent variables to adapt to the constraints of representation cost.

[0083] Specifically, in Figure 2 In the embodiment of , a salient mask is introduced, which localizes the semantic information in the feature space, thereby separating the foreground elements from the background elements and helping to identify semantically important feature vectors. As an example, the salient mask m can be obtained by the saliency detection method U 2 -Net and Alpha Matting technology are combined to extract. 2 A description of -Net can be found in the article titled "U2-net: Going deeper with nested u-structure for salient object detection" (2020) by Qin et al., and a description of "alpha matting technology" can be found in the article titled "Pymatting: A python library for alpha matting" (2020) by Germer et al. Figure 2In the embodiment of , the saliency mask m is adjusted to match the dimension of x and is represented as Then for m r Binarization is performed to obtain a binary mask (anchor mask) m a In addition, the interval mask m is generated based on the sampling factor via the interval mask generator. i , which is used to sample the background feature vector. The final mask is the anchor mask m a and interval mask m i The combination of Obtain, among which represents the “or” operation. Then, the sampled latent variable z can be obtained through the Hadamard product, that is, in The foreground is resolved and can be used as an anchor point to reconstruct the foreground content.

[0084] In the anchor-based sampling strategy 208, the anchor mask m a is maintained, and the interval mask m i The sampling factor S is controlled, where a larger sampling factor S means a larger interval and fewer latent variables are sampled. Therefore, the feature vector of the foreground can be protected and the background can be manipulated by adjusting the sampling factor S. In order to reconstruct all latent variables, the diffusion model 210 based on vector quantization can reconstruct all latent variables with the sampled latent variables as conditions. When the sampling interval is reduced, more latent variables are retained and compressed, which leads to better background reconstruction and higher image quality (such as Figure 1 By adjusting the sampling factor and compressing the sampling latent variables, the bit rate required for encoding can be achieved. Therefore, the anchor-based sampling strategy 208 not only helps to reconstruct visually good output but also benefits task performance. Figure 2 The model of the embodiment benefits from the anchor-based sampling strategy 208, thereby achieving scalability.

[0085] Figure 3 An example of stepwise sampling in an anchor-based sampling strategy is shown. The anchor-based sampling strategy can be Figure 2 Anchor-based sampling strategy in 208. Figure 3 In the example of , the sampling factor is 3. The first image (top left) shows the latent variables sampled using the anchor mask. Subsequent latent variables can then be extracted and compressed step by step from the first stage to the last stage, making the model scalable.

[0086] exist Figure 2In an embodiment, after decoding the sampled latent variables, a diffusion model 208 based on vector quantization is used to reconstruct the entire quantized vector. The diffusion model proposed in this embodiment takes advantage of the existing diffusion model for image generation, as described by Ramesh et al. in an article entitled "Zero-shot text-to-image generation" (2021). On the one hand, the latent variables are predicted in a unidirectional order, so the diffusion model can ignore the complex structural patterns of the image data. On the other hand, low-resolution latent variable reconstruction has a faster sampling speed compared to high-resolution image data generation.

[0087] The diffusion process is now further described. In this embodiment, the diffusion model 208 based on vector quantization includes a forward diffusion process and a reverse diffusion process. In some embodiments, during the training phase, the diffusion model 208 based on vector quantization may use the forward diffusion process and the reverse diffusion process. In some embodiments, during the testing phase or the inference phase, the diffusion model 208 based on vector quantization may only use the reverse diffusion process. In the forward process, the potential variable x is sampled sequentially using a Markov chain. t , where t = {1, ..., T}, and gradually introduces noise into the data. In this embodiment, each step in the forward process is a Gaussian transfer distribution q(x t ∣x t-1 ). Once a fixed number T of time steps have passed, the forward process produces a sequence of latent variables x1,…,x T , with a gradually increasing noise level. Eventually, x T Become a noise variable sampled from a normal distribution. However, since x0 is unknown at the inference stage, a transformer-based denoising network is trained in this embodiment to approximate the conditional transfer distribution in is the decoded sampled latent variable. Algorithms 1 and 2 show examples of the training and inference processes in pseudocode.

[0088]

[0089]

[0090] Figure 4 The overall framework of a diffusion model 400 based on vector quantization for sampling latent variable reconstruction according to one embodiment of the present invention is shown. Figure 4 The entire latent variable is shown, with x0 as the initial state. It should be noted that in this example, x0 is Figure 2The Markov diffusion process can be modeled as q(x t ∣x t-1 ). In this process, the transfer matrix Q is used t To specify from x t-1 to x t It should be understood that Q t The design of is important for discrete diffusion models because the signal can be more easily recovered from the noise through the inverse process. In this example, the mask and replacement transfer method published by Gu et al. in the article entitled "Vector quantized diffusion model for text-to-image synthesis" (2022) is adopted, and Q is given t The specific form is: where α t ∈[0,1],β t =(1-α t -β t ) / K, K is the number of categories, and γ t is the probability that the mark is replaced by the mask mark. Therefore, the forward Markov diffusion process can be expressed as: where v(x) is a one-hot column vector of length K. One of the important features of a Markov chain is that its intermediate steps can be marginalized, and at any time step x t The probability of can be derived directly from x0. This can be achieved by the following equation: in

[0091] To reverse the diffusion process, in this example, the denoising network is trained to obtain The conditional transfer distribution of is used to estimate the posterior of equation (3). The denoising network is trained to minimize the variational lower bound (This can be found in Sohl-Dickstein et al., “Deep unsupervised learning using non-equilibrium thermodynamics” (2015): Where p(x T ) is the prior distribution at time step T, which is predefined and fixed. Therefore, The value is constant and can be safely ignored / discarded during training. The posterior distribution q(x t-1 ∣x t ,x0) is traceable and can be calculated by the following equation:

[0092] In some embodiments, a fast inference strategy is used. In some embodiments, during the inference phase, a reparameterization technique is used to skip certain steps to achieve faster inference. Specifically, some embodiments do not use x T ,x T-1 ,x T-2 ,…,x0 to sample the image, but by reverse transfer at x T , …,x0 to sample the image: where Δ t Represents the time step. This method can efficiently sample images while minimizing the quality of the sampled image.

[0093] Now to Figure 4 The overall framework of the vector quantization-based diffusion model 400 for sampling latent variable reconstruction is further described. In some embodiments, a Transformer-based denoising model is provided to estimate the distribution like Figure 4 As shown, the model uses the latent variable at time step t Time step t and sampled latent variables As input, and output a noise-free variable distribution like Figure 4 As shown, the model includes a transformer block, a normalization layer, a linear layer, and a softmax layer.

[0094] Figure 5 It is shown that in one embodiment of the present invention, Figure 4 The structure of the converter block 500 in the vector quantization-based diffusion model 400 is shown in FIG. Figure 5 In the transformer block 500 of FIG. 5 , the adaptive layer normalization layer may be referred to as an AdaLN layer. A description of AdaLN may be found in the article entitled “Layernormalization” by Ba et al. (2016). In this embodiment, the AdaLN layer incorporates the current time step t into the network by using the following formula: AdaLN(h, t) = a t LayerNorm(h)+b t , where h is the intermediate activation and a t and b tThe value of is obtained by linear projection of the time step embedding. The transformer block 500 includes a self-attention module and a cross-attention module. z As the input of the cross attention and extracted from the sampled latent variables. After being processed by the transformer block 500, the feature can be obtained by Figure 4 The softmax layer in the vector quantization-based diffusion model 400 is projected into probabilities.

[0095] For the above reference Figures 1 to 5 The described embodiments were experimentally tested to verify them.

[0096] In the example experiment, two different data sets were used to train and test the example model of the present invention. The data sets include the CUB-200-2011 data set (reference Wah et al. in the disclosure entitled "The Caltech-UCSD birds-200-2011dataset" (2011)) and the Oxford flower data set (reference Nilsback et al. in the disclosure entitled "Automated flower classification over a large number of classes" (2008)). The CUB-200-2011 data set includes 8,855 and 2,933 images for training and testing, respectively, while the Oxford flower data set includes 2,040 and 6,149 images for training and testing, respectively. In the example experiment, all available training images are combined to optimize the model. In the example experiment, in order to prepare for the training process, all images are resized and centered to a resolution of 256×256.

[0097] In the example experiment, deep image structure and texture similarity (DISTS) and learning perceptual image patch similarity (LPIPS) were used as quality evaluation indicators to compare the perceptual quality. In order to evaluate the effectiveness of the example model of the present invention for downstream tasks, a classification model (FGVC-PIM) was used, which has been shown to have high accuracy for the original images in the test dataset. The results are evaluated with accuracy (%).

[0098] In the example experiment, the default settings of the vector quantization-based diffusion model of the example of the present invention were used, and the settings of VQ-Diffusion were used (details can be found in the article entitled "Vector quantized diffusion model for text-to-image synthesis (2022)" by Gu et al. In the example experiment, the time step was set to T = 100 and the loss weight was set to λ = 0.0005. To optimize the network of the example of the present invention, AdamW was used (some details can be found in the article entitled "Decoupled weight decay regularization (2017)" by Loshchilov et al.), where β1 = 0.9 and β2 = 0.96. After warming up for 5000 iterations, the learning rate was set to 0.00045.

[0099] In the example experiment, other existing models were used to compare with the example model of the present invention. These existing models include: The model proposed by Ballé et al. in the article titled “Variational image compression with a scalehyperprior” (2018) The model proposed by Minnen et al. in the article titled "Joint autoregressive and hierarchical priors for learned image compression" (2018) VTM model (VTM is the reference software for Versatile Video Coding (VVC). The version used in this example is VTM-15.2) The model proposed in the article titled “High-Fidelity Variable-Rate Image Compression via Invertible Activation Transformation” (2022) by Cai et al. In the example experiments, the parameters of the above three existing models are based on the parameters in the reference.

[0100] The following is described about the comparison of reconstructed data and downstream tasks. As mentioned above, in the example experiment, DISTS and LPIPS were used as quality evaluation indicators to compare the perceptual quality. In addition, in order to evaluate the effectiveness of the example model of the present invention for downstream tasks, a classification model (FGVC-PIM) was used, which has been shown to have high accuracy for the original images in the test dataset. The results are evaluated with accuracy (%).

[0101] 6A to 6F shows an example experiment processed on the CUB-200-2011 dataset ( FIG. 6A to FIG. 6C ) and an example experiment with the Oxford Flower Dataset ( FIG. 6D to FIG. 6F ), a comparison of the rate-distortion performance of the example model of the present invention and some of the above-mentioned existing models (DISTS( Fig. 6A and Fig.6D )、LPIPS( Figure 6B and Fig. 6E ), Accuracy( Figure 6C and Fig. 6F )).from 6A to 6F It can be seen that in both DISTS and LPIPS quality measures, the exemplary model of the present invention exhibits better overall reconstruction quality at low bit rates.

[0102] Figure 7 : shows a visual comparison of images obtained using the example model of the present invention and some of the above-mentioned existing models in an example experiment of processing the CUB-200-2011 dataset and the Oxford flower dataset (the value below each image is bpp / DISTS, where the lower the DISTS value, the better the perceived quality). Figure 7 As shown, the image reconstructed using the exemplary model of the present invention has good perceptual quality. In addition, due to the anchor-based sampling strategy and the vector quantization-based diffusion model in the exemplary model of the present invention, the exemplary model of the present invention achieves higher accuracy and can preserve the texture of the foreground object even at low bit rates.

[0103] Some embodiments of the present invention include semantic background generation. Some embodiments of the present invention develop a model that allows different backgrounds to be generated based on different semantic inputs. To train the model, in some embodiments, new training pairs are generated, which include a description of the desired background, its corresponding mask, and the original image. In addition, the foreground objects are removed using the mask and then the background is repaired using a professional artificial intelligence (AI) background remover (e.g., publicly disclosed at https: / / hotpot.ai / remove-background). The description of the background is generated using the content disclosed by Wang et al. in an article entitled "OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework" (2022). In order to achieve generation conditioned on semantics, the transformer block in the diffusion model (which is different from the transformer block 500) is adjusted.

[0104] Figure 8FIG. 8 is a diagram showing a transformer block 800 for semantic information according to an embodiment of the present invention, which is an example of the above-mentioned adjusted transformer. The transformer block 800 can be applied to Figure 4 In the vector quantization based diffusion model 400. In the transformer block 800, f x is from The extracted embedding vector,f z is from The extracted embedding vector, f s is the semantic feature extracted from the background description, and f x ' is the output of the converter block 800. Figure 8 As shown, unlike the transformer block 500, the input of the cross-attention module of the transformer block 800 is the semantic feature f s .

[0105] Fig. 9 The result of semantic background generation obtained by using the example model of the present invention is shown. It can be seen that the example model of the present invention can generate a semantically consistent background. Fig. 9 In the figure, the left column (related to birds) is the result obtained from the CUB-200-2011 dataset, and the right column (related to flowers) is the result obtained from the Oxford flower dataset. The generated images effectively retain the foreground objects, and the generated background is roughly consistent with the semantic information. Therefore, the example model of the present invention allows the background to be edited without affecting the foreground. This proves that it may have a wide range of applications.

[0106] In order to evaluate the effectiveness of the anchor-based sampling strategy, ablation studies were conducted on different example models (different sampling methods) of the present invention. These different example models of the present invention include: interval sampling without anchor points (abbreviated as "example (IS)"), random sampling without anchor points (abbreviated as "example (RS)"), random sampling with anchor points (abbreviated as "example (RS+anchor)"), and interval sampling with anchor points (abbreviated as "example (IS+anchor)").

[0107] Fig. 10A and Fig. 10B It is shown that in an example ablation experiment of a CUB-200-2011 dataset, the rate-distortion performance (DISTS( Fig. 10A ), Accuracy( Fig. 10B )). In addition, Fig.11 FIG. 4 shows a visual comparison of images obtained by different example models of the present invention in an example ablation experiment processed on a CUB-200-2011 dataset. FIG. 10A to FIG. 11As shown in , the anchor-based sampling strategy does not work well in downstream tasks because semantically important areas cannot be effectively protected. Interval sampling with anchors ("Example (IS+Anchor)") can capture important details in the image, while also reconstructing the background more effectively than random sampling with anchors ("Example (RS+Anchor)"). In contrast, random sampling without anchors ("Example (RS)") and interval sampling without anchors ("Example (IS)") show similar performance, among which interval sampling without anchors ("Example (IS)") produces better results in background reconstruction. As shown in Fig.11 As shown, in Fig.11 In the figure, the decoded image obtained using the example model of “Example (I S+Anchor Point)” can retain more details of the bird’s texture, and the foreground and background in the image are also more natural and realistic.

[0108] Fig.12 is a visual comparison of images obtained by the example model of the present invention at different sampling factors (1-5) in an example ablation experiment processed on a CUB-200-2011 dataset, where a larger sampling factor means fewer background feature vectors are referenced. Therefore, the smallest sampling factor shows the best background reconstruction ability. However, even when the sampling factor is equal to 5, the example model of the present invention can maintain good reconstruction quality for the decoded image.

[0109] In an example experiment, the encoding and decoding complexity of different compression schemes was comprehensively evaluated. The evaluated models include: VTM model, model of Cai et al., and example model of the present invention. Table 1 shows the average encoding time and decoding time of the models evaluated in the example experiment. Specifically, a data set consisting of 50 images was used to compare the average running time of the encoder and decoder of the three methods. The example model of the present invention has a moderate degree of complexity, and the encoding time and decoding time are slightly longer than the model of Cai et al., because it takes a larger amount of time to perform mask extraction in the encoding stage and sampling in the diffusion model in the decoding stage. In the example experiment, the QP in VTM is set to 47, and only the running time in the CPU can be counted for VTM. VTM has the lowest decoding complexity, but the encoding complexity is extremely high due to the use of a complex rate-distortion optimization process in the encoding process. Table 1 - Comparison of average encoding and decoding times

[0110] Existing image compression systems generally rely on compact representation of visual signals (by transforming the signal into a compact representation space). In this disclosure, we explore how multimodal learning can benefit compact image representation by developing a fine-grained encoding method. In this disclosure, some schemes are based on the paradigm shift that images can be perceived by both humans and machines (not just humans). In this way, the semantic information controlled by the multimodal representation can be further incorporated into the image coding process (via the AI ​​GC model).

[0111] In general, some embodiments of the present invention provide a fine-grained encoding method for compact image representation through multimodal learning. Some embodiments of the present invention incorporate semantic information controlled by multimodal representation through the AI ​​GC model, which can simultaneously achieve visually good reconstruction quality, semantically meaningful separation of background and foreground, scalability and interactive supported decoding. Some embodiments of the present invention utilize a diffusion model based on vector quantization and an anchor-based sampling strategy to retain semantically important feature vectors while adapting to the constraints of representation cost. The results of the example experiments show that the models of some embodiments of the present invention outperform existing image coding methods in terms of downstream analysis accuracy and perceptual reconstruction quality. Some embodiments of the present invention utilize a vector quantization variational autoencoder (VQ-VAE), which has high compression efficiency and can obtain compact image representation.

[0112] Some embodiments of the present invention provide a fine-grained encoding method, wherein the reconstructed image has good quality and can effectively protect semantically important areas, and the encoder is scalable and interactive. Some embodiments of the present invention provide an anchor-based sampling strategy that can retain semantically important feature vectors. Some embodiments of the present invention can achieve fine-grained scalable coding, protect semantically important features and facilitate multimodal background generation and editing. Some embodiments of the present invention provide a decoder based on a diffusion model of vector quantization. The decoder can effectively reconstruct latent variables through cross-modal input. As a result, better performance can be obtained in downstream tasks and image perception reconstruction.

[0113] Some embodiments of the present invention utilize multimodal learning to improve the quality of compression and interactivity with artificial intelligence generated content (AI GC) models. By exploring the capabilities of multimodal learning and AI GC, some embodiments of the present invention provide a fine-grained encoding method. Some embodiments of the present invention adopt a diffusion model based on vector quantization, and utilize an anchor-based sampling strategy to maintain semantically important feature vectors and sample background latent variables. In some embodiments of the present invention, the dynamic adjustment of the sampling factor enables the model to achieve fine-grained scalable encoding by compressing different latent variables. In some embodiments of the present invention, by further incorporating semantic information represented by cross-modal input, the background can be generated through semantic guidance, thereby achieving image editing.

[0114] Some embodiments of the present invention propose a method for implementing fine-grained scalable compression using multimodal learning and artificial intelligence to generate content models. Some embodiments of the present invention can have good rate-distortion performance at low bit rates, and the decoded images also have good results for downstream tasks.

[0115] Fig.13 An example data processing system 1300 of some embodiments of the present invention is shown. Data processing system 1300 may be used to perform the method (part or all) of one or more embodiments of the present invention. For example, data processing system 1300 may be used to implement Figure 1 The concept of Figure 2 The framework, Figure 4 Model Figure 5 The converter block, Figure 8 The converter block, Fig.14 The encoding method, Fig.15Decoding methods, etc. The data processing system 1300 generally includes appropriate components required to receive, store and execute appropriate computer instructions, commands and / or codes. The main components of the data processing system 1300 are the processor 1302 and the memory (storage) 1304. The processor 1302 may include one or more of the following: one or more CPUs, one or more MCUs, one or more logic circuits, one or more Raspberry Pi chips, one or more digital signal processors (DSPs), one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs) and / or any one or more digital or analog circuits / circuits configured to interpret and / or execute program instructions and / or process signals and / or information and / or data. The processor 1302 can be used to perform machine learning-based operations and non-machine learning-based operations. The processor 1302 can be used to train models (e.g., machine learning models). The memory 1304 may include: one or more volatile memories (e.g., RAM, DRAM, SRAM), one or more non-volatile memories (e.g., ROM, PROM, EPROM, EEPROM, FRAM, MRAM, FLASH, SSD, NAND, NVDIMM), or any combination thereof. Appropriate computer instructions, commands, codes, information and / or data may be stored in the memory 1304. For example, image data (input image, reconstructed image, etc.), mask data, sampling coefficient data, etc. may be stored in the memory 1304. For example, data for training a model may be stored in the memory 1304. For example, computer instructions for executing or facilitating the execution of a method (encoding method, decoding method) of one or more embodiments of the present invention may be stored in the memory 1304. The processor 1302 and the memory (storage) 1304 may be integrated or separated (and operably connected). Optionally, the data processing system 1300 further includes one or more input devices 1306. Examples of the input device 1306 include: a keyboard, a mouse, a stylus, an image scanner, a microphone, a tactile / touch input device (e.g., a touch-sensitive screen), an image / video input device (e.g., a camera), etc. In some embodiments, the input device 1306 may be used to receive an image (e.g., an input image), user input (e.g., semantic information, sampling coefficients, etc.). Optionally, the data processing system 1300 further includes one or more output devices 1308. Examples of output device 1308 include: a display (e.g., a monitor, a screen, a projector, etc.), a speaker, a headset, a headset, a printer, an additive manufacturing machine (e.g., a 3D printer), etc. The display may include an LCD display, an LED / OLED display, or other suitable display, which may or may not be touch-sensitive. In some embodiments, output device 1308 may be used to display an image (e.g., an input image, a reconstructed image, an image of a mask, etc.).The data processing system 1300 may also include one or more disk drives 1312, which may include one or more of the following: a solid-state drive, a hard disk drive, an optical drive, a flash drive, a tape drive, etc. A suitable operating system may be installed in the data processing system 1300, for example, in the disk drive 1312 or the memory 1304. The memory 1304 and the disk drive 1312 may be operated by the processor 1302. Optionally, the data processing system 1300 also includes a communication device 1310, which is used to communicate with a server, a personal computer, a terminal, a tablet computer, a phone, a watch, an Internet of Things (IoT) device, or other wireless computing device. The communication device 1310 may include one or more of the following: a modem, a network interface card (NIC), an integrated network interface, an NFC transceiver, a ZigBee transceiver, a Wi-Fi transceiver, a Bluetooth transceiver, a radio frequency transceiver, a cellular (2G, 3G, 4G, 5G, 5G above, etc.) transceiver, an optical port, an infrared port, a USB connection, or other wired or wireless communication interface. The transceiver can be implemented by one or more devices (integrated transmitter and receiver, separated transmitter and receiver, etc.). The communication link can be wired or wireless, which is used to transmit commands, instructions, information and / or data. In one example, the processor 1302, the memory 1304 (optionally, the input device 1306, the output device 1308, the communication device 1310 and the disk drive 1312, if present) are directly or indirectly connected to each other through a bus, a peripheral component interconnect (PCI) such as PCI Express, a universal serial bus (USB), an optical bus or other similar bus structures. In one embodiment, at least some of these components can be wirelessly connected, for example, through a network such as the Internet or a cloud computing network. It will be appreciated by those skilled in the art that. Fig.13 Data processing system 1300 shown in FIG. 1 is merely an example, and in other implementations data processing system 1300 may have a different configuration (eg, include additional components, have fewer components, etc.).

[0116] It should be understood that in the case where the methods and systems of the embodiments of the present invention are fully or partially implemented by a computing system, any suitable computing system architecture may be used, including stand-alone computers, network computers, dedicated or non-dedicated hardware devices, etc. In this document, the terms "computing system" and "computing device" (or similar terms) are intended to include, but are not limited to, computers or information processing hardware (any suitable arrangement) capable of implementing the functions described. The encoding device of the present invention may be implemented using software and / or hardware. The decoding device of the present invention may be implemented using software and / or hardware.

[0117] Those skilled in the art will appreciate that the present invention as shown in the specific embodiment can be subjected to different changes and / or modifications to provide other embodiments of the present invention. Therefore, the embodiments of the present invention described should be considered to be illustrative and not restrictive in all respects. The example optional features of some aspects of the present invention are set forth in the above summary of the invention. Some embodiments of the present invention may include one or more of these optional features (some of which are not specifically shown in the drawings). Some embodiments of the present invention may lack one or more of these optional features (some of which are not specifically shown in the drawings). One or more features in an embodiment and one or more features in another embodiment can be combined to provide further embodiments of the present invention.

Claims

1. A coding device, comprising: An image encoder, for extracting image coding features from the image; A quantization module, configured to quantize the image coding features based on a potential embedding space to convert the image coding features into discrete latent variables; A sampling module, configured to sample the discrete latent variables using a sampling mask associated with a sampling factor to obtain a sampled latent variable; as well as A compression module, used for compressing the sampled latent variables; The compressed sampled latent variables may be used to generate a bit stream associated with the image.

2. The encoding device according to claim 1, characterized in that The sampling module is also used to: extract a mask from the image to separate foreground information and background information related to the image, process the extracted mask to obtain an anchor mask, generate an interval mask according to a sampling factor, and combine the anchor mask and the interval mask to generate the sampling mask.

3. The encoding device according to claim 2, characterized in that The sampling module adjusts the size of the extracted mask and binarizes the adjusted mask to process the extracted mask.

4. The encoding device according to claim 2, characterized in that At least the image encoder and the latent embedding space are trained via a loss function.

5. The encoding device according to claim 4, characterized in that The training performed by the loss function is diffusion model training based on vector quantization.

6. The encoding device according to claim 5, characterized in that When training the vector quantization based diffusion model, the parameters of the image encoder and the latent embedding space are fixed.

7. The encoding device according to claim 6, characterized in that The sampling module uses a saliency detection algorithm to separate foreground information and background information related to the discrete latent variables.

8. The encoding device according to any one of claims 2 to 7, characterized in that: The mask extracted from the image is a saliency mask, which is used to localize semantic information in feature space, thereby separating foreground elements from background elements.

9. The encoding device according to any one of claims 2 to 7, characterized in that: The interval mask is an interval mask that is controlled by a sampling factor, where a larger sampling factor represents a larger interval and fewer latent variables are sampled.

10. A decoding device, comprising: an image decoder for decoding a bit stream associated with an image to obtain a sampled latent variable; as well as A vector quantization-based diffusion module is used to process the sampled latent variables at least through a back-diffusion process to reconstruct the image.

11. The decoding device according to claim 10, characterized in that: The vector quantization based diffusion module comprises a semantic information input for editing the background of the image to be reconstructed.

12. The decoding device according to claim 10 or 11, characterized in that: The vector quantization-based diffusion module is used to process the sampled latent variables through a forward diffusion process and a backward diffusion process to reconstruct the image; and The vector quantization-based diffusion module uses a Markov chain to sequentially sample data and gradually introduce noise into the data during the forward diffusion process.

13. The decoding device according to claim 12, characterized in that: The vector quantization-based diffusion module is a Gaussian transfer distribution at each time step in the forward diffusion process.

14. The decoding device according to claim 13, characterized in that: After a fixed number of T time steps have passed, the vector quantization-based diffusion module generates a latent variable sequence x1,…,x with gradually increasing noise levels in the forward diffusion process. T .

15. A coding and decoding system, comprising: The encoding device according to any one of claims 1 to 9; as well as A decoding device according to any one of claims 10 to 14.

16. A coding method comprising: Extracting image encoding features from the image; quantizing the image encoding features based on a latent embedding space to convert the image encoding features into discrete latent variables; Sampling the discrete latent variables using a sampling mask associated with a sampling factor to obtain sampled latent variables; and compressing the sampled latent variables; The compressed sampled latent variables may be used to generate a bit stream associated with the image.

17. The encoding method according to claim 16, characterized in that: Also included is generating the sampling mask, which includes: extracting a mask from the image to separate foreground information and background information associated with the image; processing the extracted mask to obtain an anchor mask; generating an interval mask based on the sampling factor; and The anchor point mask and the interval mask are combined to generate the sampling mask.

18. The encoding method according to claim 17, characterized in that: Processing the extracted mask to obtain the anchor mask includes: The extracted mask is resized and the resized mask is binarized to obtain the anchor mask.

19. The encoding method according to claim 17 or 18, wherein: The mask extracted from the image is a saliency mask for localizing semantic information in feature space, thereby separating foreground elements from background elements; and / or The interval mask is an interval mask that is controlled by a sampling factor, where a larger sampling factor represents a larger interval and fewer latent variables are sampled.

20. A decoding method, comprising: decoding a bitstream associated with the image to obtain sampled latent variables; as well as The sampled latent variables are processed at least through a back-diffusion process using a diffusion model based on vector quantization to reconstruct the image.

21. The decoding method according to claim 20, characterized in that: The vector quantization based diffusion model includes a semantic information input, which is used to edit the background of the image to be reconstructed.

22. The decoding method according to claim 20 or 21, characterized in that: The reconstructing the image by using the diffusion model based on vector quantization includes processing the sampled latent variables by a forward diffusion process and a reverse diffusion process based on vector quantization to reconstruct the image; The forward diffusion process uses a Markov chain to sequentially sample data and gradually introduce noise into the data.

23. The decoding method according to claim 22, characterized in that: Each time step of the forward diffusion process is a Gaussian transfer distribution.

24. The decoding method according to claim 23, characterized in that: After a fixed number T of time steps have passed, the forward diffusion process produces a latent variable sequence x1,…,x with increasing noise levels. T .

Images