Generating potential representations of images using machine learning models

Abstract

This disclosure describes techniques for generating a potential representation of an image using a machine learning model. The image is segmented and flattened into a series of blocks. A series of blocks are spliced with the potential tag sequence. And inputting the spliced blocks and the potential markers into an encoder of a machine learning model. A one-dimensional potential representation of an image is generated by an encoder. Vector quantization is performed on the one-dimensional potential representation of the image by a vector quantizer of a machine learning model to generate quantized potential markers. The image is reconstructed by a decoder of the machine learning model based on the quantized potential markers.

Description

Background Technology

[0001] Machine learning models are increasingly being used across various industries to perform a wide range of tasks. These tasks can include content generation. There is a growing expectation to leverage machine learning models to improve content generation techniques. Attached Figure Description

[0002] The following detailed embodiments can be better understood when read in conjunction with the accompanying drawings. For illustrative purposes, exemplary embodiments of various aspects of this disclosure are shown in the drawings; however, the invention is not limited to the specific methods and tools disclosed.

[0003] Figure 1 An example system for generating potential representations of images using a machine learning model, according to this disclosure, is shown.

[0004] Figure 2 An example system for generating potential representations of images using a machine learning model, according to this disclosure, is shown.

[0005] Figures 3A to 3B An example process for image tokenization according to this disclosure is shown.

[0006] Figure 4 An example process for generating potential representations of images using a machine learning model, according to this disclosure, is shown.

[0007] Figure 5 An example process for generating potential representations of images using a machine learning model, according to this disclosure, is shown.

[0008] Figure 6 An example process for generating potential representations of images using a machine learning model, according to this disclosure, is shown.

[0009] Figure 7 An example process for training a machine learning model according to this disclosure is shown.

[0010] Figure 8 Example graphs are shown to illustrate the evaluation results according to this disclosure.

[0011] Figure 9 An example table showing the evaluation results according to this disclosure is provided.

[0012] Figure 10 An example table showing the evaluation results according to this disclosure is provided.

[0013] Figures 11A to 11C Example tables are shown illustrating the results of ablation studies based on this disclosure.

[0014] Figure 12An example computing device is shown that can be used to perform any of the techniques disclosed herein. Detailed Implementation

[0015] In recent years, the field of image generation has made significant progress, driven by major advancements in transformer and diffusion models. These developments have ultimately enabled the generation of images that are more realistic and detailed than ever before. Similar to the trend in generative language models, many existing image generation models incorporate standardized image tokenizers and de-tokenizers in their architecture. This array of models utilizes tokenized image representations (from continuous vectors to discrete vectors) to perform a key function: transforming raw pixels into latent image representations. The latent space is much more compact than the original image space (e.g., 32×32 vs. 256×256×3), which not only facilitates efficient and effective training of generative models but also paves the way for scaling to larger models by providing concise and expressive representations for image synthesis.

[0016] While image taggers have proven successful in image generation workflows, they encounter fundamental limitations inherent in their design. These systems assume that the latent space retains a two-dimensional structure, maintaining a direct mapping to the corresponding image patches (e.g., assuming the latent tag in the top-left corner directly corresponds to the image patch in the top-left corner). This assumption limits the tagger's ability to effectively utilize the inherent redundancy of images to generate more compressed latent spaces. Consequently, this limitation also hinders the ability to increase image resolution or context length. For example, assuming the image tagger downsamples the image by eight times, Llama2 with a context length of 4096 can manage at most four 256×256 resolution images or one 512×512 resolution image. While 4096 tags may contain rich and diverse textual context information, this capacity is significantly limited in the highly redundant domain of image tagging. Even commercial large language models (LLMs) with context lengths of up to one million can handle the entire Harry Potter book series, but they are almost incapable of handling a two-and-a-half-hour Harry Potter movie, which could easily generate millions of tags using existing tagging models.

[0017] Therefore, there is a need to improve the techniques for generating latent representations of images. This paper describes an improved technique for generating latent representations of images. The technique described in this paper utilizes a compact one-dimensional (1D) sequence as a latent representation of the image in the context of image reconstruction and generation. Compared with existing techniques that utilize downsampled two-dimensional grids to generate latent image representations, the technique described in this paper utilizes a transformer-based framework configured to tokenize the input image into a one-dimensional discrete sequence, which can then be decoded back into an image via a de-tokenizer. The transformer-based framework described in this paper may be referred to as a Transformer image tokenizer. The transformer-based framework described in this paper may include a visual transformer encoder, a ViT decoder, and a vector quantizer. In the tokenization stage, the image can be segmented and flattened into a series of blocks, which are then concatenated with the latent token sequence. The latent tokens generated by the encoder during feature encoding can construct the latent representation of the image. After the vector quantization step, the input image can be reconstructed from the masked token sequence using the decoder. The transformer-based framework described in this paper demonstrates significant efficiency in one-dimensional image tokenization.

[0018] Figure 1 An example system 100 for generating latent representations of an image is illustrated. System 100 may include a machine learning model 105. Machine learning model 105 may include an encoder 108, a vector quantizer 110, and a decoder 112. Image 102 may be segmented and / or flattened into a series of blocks 104. A series of blocks 104 may be concatenated with a sequence of latent labels 106. The concatenated blocks and latent labels may be fed into the encoder 108 of the machine learning model. Encoder 108 may generate a one-dimensional latent representation 107 of image 102. Encoder 108 may generate a one-dimensional latent representation 107 of image 102 based on the concatenated blocks and latent labels. Generating a one-dimensional latent representation 107 may decouple the size of the one-dimensional latent representation 107 from the resolution of image 102. Generating a one-dimensional latent representation 107 may include generating a one-dimensional sequence. The one-dimensional sequence may have the same size as the sequence of latent labels 106.

[0019] A one-dimensional latent representation 107 can be input into a vector quantizer 110. The vector quantizer 110 can perform vector quantization on the one-dimensional latent representation 107 to generate quantized latent markers 111. The quantized latent markers 111 can be input into a decoder 112. The decoder 112 can reconstruct an image based at least on the quantized latent markers 111. For example, the decoder 112 can generate a reconstructed image 116 based at least on the quantized latent markers 111. The reconstructed image 116 can be a reconstruction of image 102. A mask marker sequence can be merged into the quantized latent markers 111. The merged mask markers and the quantized latent markers 111 can be input into the decoder 112. The decoder 112 can generate the reconstructed image 116 based on the mask marker sequence and the quantized latent markers 111.

[0020] In this embodiment, a two-stage training paradigm can be used to train the machine learning model 105. The first stage of the two-stage training paradigm may include training the machine learning model using proxy code to bypass complex loss functions and generative adversarial network (GAN) architectures. The proxy code may include discrete code. The discrete code can be generated using an off-the-shelf vector quantization GAN (VQGAN) model. The second stage of the two-stage training paradigm may include training the machine learning model to fine-tune the decoder 112. The second stage of the training paradigm may include training the decoder 112 only in the pixel space using VQGAN loss while keeping the encoder 108 and vector quantizer 110 frozen.

[0021] Figure 2 An example system 200 for generating latent representations of images is illustrated. System 200 may include a machine learning model. The machine learning model may include a transformer encoder 208, a vector quantizer 210, and a transformer decoder 212. Transformer encoder 208 may include a ViT encoder. Transformer decoder 212 may include a ViT decoder. Transformer encoder 208 may be referred to herein as Enc, vector quantizer 210 may be referred to herein as Quant, and transformer decoder 212 may be referred to herein as Dec.

[0022] Image 202 can be represented as I∈R H×W×3 Where H and W represent the height and width of image 202, respectively. Image 202 can be segmented and / or flattened into a series of blocks 204. A series of blocks 204 can be represented as P, where (Block size equals downsampling factor f and dimension C). A series of blocks 204 can be concatenated with a sequence of latent labels 206. The sequence of latent labels 206 can include K latent labels. The sequence of latent labels 206 can be represented as L∈R K×C .

[0023] The concatenated blocks and latent labels can be fed into the transformer encoder 208 of the machine learning model. In the input to the transformer encoder 208, only the latent labels are retained as the latent representation 207 of the image, thus achieving a more compact latent representation 207. The latent representation 207 can include a one-dimensional sequence Z. 1D (Length is K). This adjustment decouples the potential size from the resolution of image 202 and allows for greater flexibility in design choices. That is, in This indicates concatenation. The latent representation 207 can be input into the vector quantizer 210. The vector quantizer 210 can perform vector quantization on the latent representation 207 to generate a quantized latent label 211.

[0024] In the de-marking stage, the sequence of mask markers 232 can be merged into the quantized latent markers 211. Merging the sequence of mask markers 232 into the quantized latent markers 211 can include concatenating the quantized latent markers 211 with the sequence of mask markers 232. The sequence of mask markers 232 can be represented as M, where... It can be achieved by copying a single mask marker. The sequence of mask markers 232 is obtained next. Transformer decoder 212 can generate a series of reconstruction blocks 224. Transformer decoder 212 can generate a series of reconstruction blocks 224 based on the concatenated sequence of mask markers 232 and the quantized latent markers 211. The series of reconstruction blocks 224 can be a reconstruction of a series of blocks 204. Image 202 can be reconstructed based on the series of reconstruction blocks 224. For example, a reconstructed image 216 can be generated based on the series of reconstruction blocks 224. The reconstructed image 216 can be represented as... in

[0025] A two-stage training paradigm can be used to train a machine learning model comprising a transformer encoder 208, a vector quantizer 210, and a transformer decoder 212. Given the complexity of the loss function and the large number of hyperparameters involved in the machine learning model, training may be unstable, especially when the goal is compact one-dimensional tokenization.

[0026] To mitigate training instability, a two-stage training paradigm utilizing proxy codes can be used to train the machine learning model. The two-stage training strategy comprises a "warm-up" phase and a "decoder fine-tuning" phase. Specifically, in the first "warm-up" phase, the machine learning model is trained using discrete codes (which can be referred to as proxy codes) generated by an off-the-shelf VQGAN model, instead of directly regressing the red, green, and blue (RGB) values ​​in pixel space and applying various loss functions (as in existing methods). This allows bypassing complex loss functions and generative adversarial network (GAN) architectures. This modification does not affect the functionality of the transformer encoder 208 and vector quantizer 210, which still fully perform image tokenization and detoxification; the main adjustment involves processing the output of the transformer decoder 212. The output of the transformer decoder 212 (which may include a set of proxy codes) can then be fed into the same off-the-shelf VQGAN decoder to generate the final RGB output.

[0027] After the first stage of training using the proxy code, a second optional "decoder fine-tuning" stage can be performed to improve the quality of the reconstructed image. The transformer encoder 208 and vector quantizer 210 can be kept frozen, and the transformer decoder 212 can be trained only in the pixel space using typical VQGAN losses (including L2 loss, perceptual loss, and adversarial loss). This two-stage training strategy significantly improves training stability and the quality of the reconstructed image.

[0028] In existing two-dimensional latent representation generation techniques, the latent representation is typically conceived as a static two-dimensional grid. This configuration inherently assumes a strict one-to-one mapping between the latent grid and the original image patches. This assumption limits the model's ability to fully utilize redundancy present in the image (such as similarity between adjacent patches). Furthermore, this approach limits flexibility in choosing the latent size; the most common configurations are f=4, f=8, or f=16, resulting in 4096, 1024, or 256 labels for an image of size 256×256×3. Using one-dimensional latent representations to reconstruct images is not only more efficient but also produces higher-quality images. Labeling an image as a compact one-dimensional sequence with fewer latent labels (e.g., 32 latent labels instead of the standard 256) can be used to reconstruct the image (e.g.,...). Figure 3A (as shown in the simplified diagram 300) or generated image (such as...) Figure 3B (As shown in the simplified diagram 301). This compact latent space can improve the efficiency and effectiveness of generative model training.

[0029] Figure 4 An example process 400 for generating efficient and effective latent representations of images using a machine learning model according to this disclosure is demonstrated. Although in Figure 4The operations are described as a series of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder or modify the operations described.

[0030] At 402, the image (e.g., image 102 or image 202) can be segmented and / or flattened into a series of blocks (e.g., a series of blocks 104 or a series of blocks 204). At 404, the series of blocks can be concatenated with a sequence of latent labels (e.g., a sequence of latent labels 106 or a sequence of latent labels 206). At 406, the concatenated blocks and latent labels can be fed into the encoder (e.g., encoder 108 or transformer encoder 208) of a machine learning model (e.g., machine learning model 105).

[0031] At position 408, a one-dimensional latent representation of the image (e.g., latent representation 107 or latent representation 207) can be generated by an encoder. Generating a one-dimensional latent representation may include decoupling the size of the one-dimensional latent representation from the resolution of the image. Generating a one-dimensional latent representation may include generating a one-dimensional sequence. The one-dimensional sequence may have the same size as the sequence of latent labels. The one-dimensional latent representation may be fed into a vector quantizer of a machine learning model (e.g., vector quantizer 110 or vector quantizer 210).

[0032] At 410, vector quantization can be performed on the one-dimensional latent representation of the image using a vector quantizer to generate quantized latent labels (e.g., quantized latent label 111, quantized latent label 211). The quantized latent labels can be input into the decoder of a machine learning model (e.g., decoder 112 or transformer decoder 212). At 412, the image can be reconstructed. The image can be reconstructed based on the quantized latent labels. The image can be reconstructed using the decoder.

[0033] Figure 5 An example process 500 for generating efficient and effective latent representations of images using a machine learning model according to this disclosure is demonstrated. Although in Figure 5 The operations are described as a series of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder or modify the operations described.

[0034] At 502, the image (e.g., image 102 or image 202) can be segmented and / or flattened into a series of blocks (e.g., a series of blocks 104 or a series of blocks 204). At 504, the series of blocks can be concatenated with a sequence of latent markers (e.g., a sequence of latent markers 106 or a sequence of latent markers 206).

[0035] The concatenated blocks and latent labels can be input into the encoder (e.g., encoder 108 or transformer encoder 208) of a machine learning model (e.g., machine learning model 105). At 506, a one-dimensional latent representation of the image can be generated, e.g., latent representation 107 or latent representation 207. The one-dimensional latent representation can include a one-dimensional sequence. The one-dimensional sequence can have the same size as the sequence of latent labels. The one-dimensional latent representation can be generated by the encoder. The one-dimensional latent representation can be generated by the encoder based on the concatenated blocks and latent labels. At 508, the size of the one-dimensional latent representation can be decoupled from the resolution of the image.

[0036] Figure 6 An example process 600 for generating efficient and effective latent representations of images using a machine learning model according to this disclosure is demonstrated. Although in Figure 6 The operations are described as a series of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder or modify the operations described.

[0037] At 602, vector quantization can be performed on a one-dimensional latent representation (e.g., latent representation 107 or latent representation 207) of an image (e.g., image 102 or image 202) by a vector quantizer (e.g., vector quantizer 110 or vector quantizer 210) of a machine learning model to generate quantized latent labels (e.g., quantized latent label 111, quantized latent label 211). At 604, a mask label sequence (e.g., a sequence of mask labels 232) can be generated. The mask label sequence can be generated by copying individual mask labels. At 606, the mask label sequence can be merged into the quantized latent labels. Merging the mask label sequence into the quantized latent labels can include concatenating the quantized latent labels with the mask label sequence. At 608, the merged mask labels and quantized latent labels can be input into a decoder (e.g., decoder 112 or transformer decoder 212) of a machine learning model. At 610, the image can be reconstructed. The image can be reconstructed based on the merged mask labels and quantized latent labels. Images can be reconstructed using the decoder of a machine learning model.

[0038] Figure 7 An example process 700 for training a machine learning model according to this disclosure is shown. Although in Figure 7 The operations are described as a series of operations, but those skilled in the art will understand that various embodiments may add, remove, reorder or modify the operations described.

[0039] At 702, the machine learning model (e.g., machine learning model 105) can be trained using a two-stage training paradigm. At 704, the first stage of training can be performed on the machine learning model. The first stage of training can be performed using proxy code to bypass complex loss functions and generative adversarial network (GAN) architectures. The proxy code can include discrete code generated via an off-the-shelf vector quantization GAN (VQGAN) model. At 706, the second stage of training can be performed on the machine learning model. The second stage of training can be performed to fine-tune the decoder of the machine learning model (e.g., decoder 112 or transformer decoder 212). The second stage of training can include training the decoder only in pixel space using VQGAN loss while keeping the encoder (e.g., encoder 108 or transformer encoder 208) and the vector quantizer (e.g., vector quantizer 110 or vector quantizer 210) of the machine learning model frozen.

[0040] The machine learning model described in this paper (e.g., machine learning model 105) can be configured in a variety of different ways, can have a variety of different sizes, and can utilize different numbers of latent labels. Experiments were conducted to determine the most efficient and effective settings for the machine learning model described in this paper.

[0041] In the experiments, the machine learning model was trained using images with resolutions of H=256 and W=256. The block size for both the tagger and detagner was set to 16, with the codebook C configured to have N=1024 entries and dimension D=16, and L2 normalization applied to the codebook embeddings. Training augmentations were limited to random pruning and flipping. The training scheme employed compressed scheduling with a batch size of 256, conducted over 500k steps, equivalent to approximately 100 epochs. The optimizer's initial learning rate was 1×10⁻⁶. -4 The weight decays to 1×10 -4 The learning rate was scheduled using cosine decay. Only the first phase of training (i.e., only the "warm-up" training phase) was used in the experiment.

[0042] Three model sizes were evaluated: small, basic, and large transformer image labelers (models). These included 22M, 86M, and 307M parameters for the encoder and decoder, respectively. The impact of changing the number of latent labels K from 16 to 256 was evaluated. During the generative training phase, all images were pre-labeled using center cropping and random flipping enhancement, and then processed via a masked image modeling procedure to generate images. An eight-step cosine masking schedule was employed. The batch size for training the generative model was 2048, with 500k steps. Additionally, a dropout probability of 0.1 was applied to the class condition. During evaluation, a classifier-free guidance system with linear scheduling was used.

[0043] To comprehensively evaluate the model, multiple metrics were employed, including reconstruction and generation FID metrics (i.e., rFID and gFID), which measure performance in image reconstruction and generation tasks, respectively. Given that the one-dimensional VQ model is essentially a compact form of image compression, the semantic information preserved by the model was investigated through linear probing. Additionally, training throughput was examined to directly compare the training efficiency of generative models at different latent sizes.

[0044] It is well known that image representations inherently contain redundancy, as evidenced by practices such as masking large portions of an image to accelerate the training process without adversely affecting performance. This strategy has been validated in various computer vision tasks that rely on high-level image features. However, the effectiveness of this approach in the context of image reconstruction and generation, where low-level details are crucial for creating realistic reconstructed and generated outputs, remains to be fully explored. Therefore, experiments were conducted to determine the minimum number of labels required for reconstructing and generating high-quality images.

[0045] Figure 8 The results for rFID and gFID are shown for different numbers of potential markers. (800) Figure 8 As shown in (a), although model performance gradually improves with increasing latent label size, it is observed that the performance improvement is most significant when K ranges from 16 to 128. After this point, increasing the latent space size only yields marginal benefits. With only 32 latent labels, the model achieves performance comparable to existing models, suggesting that as few as 32 labels may be sufficient as an effective latent representation of an image, optimizing the utilization of image redundancy.

[0046] Enlarging the marker can achieve a more compact potential size. For example... Figure 8 As shown in graph (a), larger models facilitate more compact representations. Specifically, the base model with 64 latent labels achieves performance comparable to a smaller model with 128 latent labels, while the large model with 32 latent labels achieves performance comparable to the base model with 64 latent labels. This pattern suggests that with each increment in model size (e.g., from S to B, from B to L), it is possible to reduce the size of the latent image representation without compromising performance. This trend highlights the potential benefits of scaling up the labeler to achieve more compact image representations.

[0047] Linear probing experiments were conducted on the image markers, such as... Figure 8 The curve is shown in (b). Specifically, a batch normalization layer without affine transformation is added on top of the frozen encoded features, followed by a linear layer following the MAE protocol. Figure 8As shown in graph (b), as the size of the latent representation decreases, the model increasingly tends to learn semantically rich representations during the tokenization process, as evidenced by the improvement in linear detection accuracy. This indicates that in scenarios with limited representation space, the model learns high-level information, while in scenarios with expanded latent space, the model tends to focus more on local texture.

[0048] In addition to reconstruction capabilities, the gFID performance of the markers was evaluated to assess their effectiveness in generative downstream tasks, such as... Figure 8 The curves are shown in (c) and (d). When the number of latent labels is sufficiently large (K≥128), the results produced by variants of different model sizes are comparable. However, within the range of compact latent sizes, larger models significantly improve performance. Furthermore, the adaptation of one-dimensional labeling in the transformer image labeler improves the efficiency and effectiveness of generative model training. For example, the model variant with K=32, despite poorer reconstruction quality, exhibits significantly better generative performance, highlighting the advantages of using a more concise and semantically rich latent space in the training of generative models. In addition, reducing the number of latent labels significantly speeds up training; when using K=32 instead of K=256, the training speed is improved by 9.1 times (1397.3 vs. 152.9 samples / GPU), and the sampling speed is improved by 5.9 times (generating 128 samples takes 83.7 seconds vs. 14.3 seconds).

[0049] Figure 9 Example Table 900 is shown, illustrating evaluation results according to this disclosure. The ability of machine learning model 105, as well as the ability of existing diffusion generative models and transformer generative models to reconstruct 256×256 images, is evaluated. The results are shown in Table 900. The comparisons shown in Table 900 are based on the best variants using no classifier guidance or rejection sampling.

[0050] As shown in Table 900, the transformer-based framework described in this paper achieves a similar level of reconstructed FID (rFID) using a significantly fewer latent labels than other VQ models. Specifically, the large model achieves an rFID of 2.22 using only 32 labels. Furthermore, when using the same generator framework and the same sampling stride, the large model significantly improves upon existing models (gFID from 4.02 to 2.88), demonstrating the benefits of more efficient generator training using compact one-dimensional labels. This model also achieves similar performance compared to other diffusion-based generative models, while enjoying over 100x speedup (measured by throughput) during the sampling process.

[0051] Figure 10Example Table 1000 is shown, illustrating the evaluation results according to this disclosure. The capabilities of machine learning model 105, as well as existing diffusion generative models and transformer generative models, were evaluated to reconstruct 512×512 images. The results are shown in Table 1000. The comparisons shown in Table 1000 are based on the best variants using no classifier guidance or rejection sampling. P represents the number of parameters of the generator, S represents the number of sampling steps, and T represents the samples per second throughput on A100.

[0052] As shown in Table 1000, the transformer-based framework described in this paper offers a significantly better trade-off between accuracy and cost. Considering that the model uses fewer labels (i.e., a higher compression ratio), it maintains a considerably good rFID compared to other methods. For generation, all variants described in this disclosure outperform existing techniques. The large model of size L-64 exhibits performance comparable to existing models, while running at a speed improvement of 455.6x, compared to diffusion-based models. The best base model variant of size B-128 with 64 steps shows a significant performance improvement (gFID 2.31 vs. 3.04) and a faster sampling rate (7.35 vs. 0.09 samples / sec) compared to existing models.

[0053] Figures 11A-11C The results of ablation studies using the transformer-based framework described in this paper are presented. Figure 11A Table 1100 shows the results of the ablation marker design (reconstruction task) on the validation set. A large model with 32 markers achieved an rFID of 6.59. Using a larger codebook size improved the rFID by 0.74, while further increasing the number of training iterations (from 100 epochs to 200 epochs) improved it by another 0.37. In addition, “decoder fine-tuning” (the second-stage training strategy described in this paper) significantly improved the overall reconstruction performance to an rFID of 2.21.

[0054] Figure 11B Table 1101 illustrates the effects of different masking schedules. As shown in Table 1101, the model changes its preference to inverse cosine or linear scheduling. This performance improvement can be attributed to the model's ability to provide more compact and semantically meaningful labels than existing models, with linear and inverse cosine scheduling showing lower masking rates in earlier steps compared to cosine masking scheduling. This is consistent with the observation that masking rates are generally higher for redundant signals (e.g., 75% masking rate in images) and relatively lower for semantically meaningful inputs (e.g., 15% masking rate in language).

[0055] Figure 11CTable 1102 shows the experimental results using the advanced transformer backbone. As shown in Table 1102, this resulted in a significant improvement for the small model with 128 labels. This improvement can be attributed to the compact nature of the one-dimensional latent representation and the small dataset (approximately 1.28M images). Specifically, the large model with 32 labels uses only 32 labels to represent each image, thus effectively compressing the entire training dataset.

[0056] Figure 12 The illustration shows computing devices that can be used in various fields, such as... Figures 1 to 2 The models, components, and / or devices depicted herein. Regarding... Figures 1 to 2 Any or all components can be freely separated. Figure 12 This is implemented using one or more instances of the computing device 1200. Figure 12 The computer architectures shown illustrate conventional server computers, workstations, desktop computers, laptop computers, tablet computers, network facilities, PDAs, e-readers, digital cellular phones, or other computing nodes, and can be used to perform any aspect of the computers described herein, such as implementing the methods described herein.

[0057] The computing device 1200 may include a substrate or “motherboard,” which is a printed circuit board to which multiple components or devices may be connected via a system bus or other electrical communication paths. One or more central processing units (CPUs) 1204 may operate in conjunction with a chipset 1206. The CPUs 1204 may be standard programmable processors that perform the arithmetic and logic operations required to operate the computing device 1200.

[0058] Multiple CPUs 1204 can perform necessary operations by manipulating switching elements to transition from one discrete physical state to the next. These switching elements can distinguish and change these states. Switching elements typically include electronic circuitry (such as flip-flops) that holds one of two binary states, and electronic circuitry (such as logic gates) that provides an output state based on a logical combination of the states of one or more other switching elements. These basic switching elements can be combined to create more complex logic circuits, including registers, adder-subtractor units, arithmetic logic units, floating-point units, etc.

[0059] The (multiple) CPUs 1204 can be enhanced or replaced by other processing units (such as (multiple) GPUs 1205). The (multiple) GPUs 1205 may include processing units specifically designed for, but not necessarily limited to, highly parallel computing (such as graphics and other visualization-related processing).

[0060] Chipset 1206 may provide an interface between CPU(s) 1204 and the remaining components and devices on the substrate. Chipset 1206 may provide an interface to random access memory (RAM) 1208, which is used as the main memory in computing device 1200. Chipset 1206 may further provide an interface to computer-readable storage media (such as read-only memory (ROM) 1220 or non-volatile RAM (NVRAM) (not shown)) for storing basic routines that can help start computing device 1200 and transfer information between various components and devices. ROM 1220 or NVRAM may also store other software components required for the operation of computing device 1200 according to the various aspects described herein.

[0061] Computing device 1200 can operate in a networked environment using a logical connection via a local area network (LAN) to remote computing nodes and computer systems. Chipset 1206 may include functionality for providing network connectivity via a network interface controller (NIC) 1222 (such as a Gigabit Ethernet adapter). NIC 1222 may be able to connect computing device 1200 to other computing nodes via network 1216. It should be understood that multiple NICs 1222 may be present in computing device 1200, thereby connecting the computing device to other types of networks and remote computer systems.

[0062] Computing device 1200 can be connected to mass storage device 1228, which provides non-volatile storage for the computer. Mass storage device 1228 can store system programs, application programs, other program modules, and data, which are described in more detail herein. Mass storage device 1228 can be connected to computing device 1200 via storage controller 1224 connected to chipset 1206. Mass storage device 1228 can consist of one or more physical storage units. Mass storage device 1228 may include management unit 1210. Storage controller 1224 can interface with physical storage units via a Serial Amount SCSI (SAS) interface, a Serial Advanced Technology Attachment (SATA) interface, a Fibre Channel (FC) interface, or other types of interfaces used for physical connections and data transfer between the computer and physical storage units.

[0063] The computing device 1200 can store data on the mass storage device 1228 by changing the physical state of the physical storage units to reflect the stored information. The specific changes in the physical state can depend on various factors and different implementations of this specification. Examples of such factors may include, but are not limited to, the technology used to implement the physical storage units and whether the mass storage device 1228 is characterized as a primary storage device or a secondary storage device.

[0064] For example, computing device 1200 can issue instructions via storage controller 1224 to change the magnetic properties of a specific location within a disk drive unit, the reflection or refraction properties of a specific location within an optical storage unit, or the electrical properties of a specific capacitor, transistor, or other discrete component in a solid-state storage unit, thereby storing information in mass storage device 1228. Other transformations of the physical medium are also possible without departing from the scope and spirit of this specification, wherein the foregoing examples are provided merely to assist this specification. Computing device 1200 can further read information from mass storage device 1228 by detecting the physical state or characteristics of one or more specific locations within the physical storage unit.

[0065] In addition to the high-capacity storage device 1228 described above, the computing device 1200 can also access other computer-readable storage media to store and retrieve information, such as program modules, data structures, or other data. Those skilled in the art will understand that a computer-readable storage medium can be any available medium that provides non-transitory data storage and can be accessed by the computing device 1200.

[0066] By way of example and not limitation, computer-readable storage media may include volatile and non-volatile computer-readable storage media, transient computer-readable storage media and non-transitory computer-readable storage media, and removable and non-removable media implemented in any method or technology. Computer-readable storage media include, but are not limited to, RAM, ROM, erasable programmable ROM (“EPROM”), electrically erasable programmable ROM (“EEPROM”), flash memory or other solid-state memory technologies, compact disc ROM (“CD-ROM”), digital versatile optical disc (“DVD”), high-definition DVD (“HD-DVD”), BLU-RAY, or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices, other magnetic storage devices, or any other medium that can be used to store desired information in a non-transitory manner.

[0067] High-capacity storage devices (such as) Figure 12 The mass storage device 1228 depicted herein can store an operating system used to control the operation of the computing device 1200. The operating system may include a version of the LINUX operating system. The operating system may include a version of the WINDOWS SERVER operating system from Microsoft Corporation. Depending on other aspects, the operating system may include a version of the UNIX operating system. Various mobile phone operating systems, such as iOS and Android, may also be utilized. It should be understood that other operating systems may also be utilized. The mass storage device 1228 can store other systems, applications, and data used by the computing device 1200.

[0068] Mass storage device 1228 or other computer-readable storage media may also be encoded with computer-executable instructions that, when loaded into computing device 1200, transform the computing device from a general-purpose computing system into a special-purpose computer capable of implementing the various aspects described herein. These computer-executable instructions transform computing device 1200 by specifying the state transitions of CPU(s)1204, as described above. Computing device 1200 can access the computer-readable storage medium storing the computer-executable instructions, which, when executed by computing device 1200, can perform the methods described herein.

[0069] Computing devices (e.g.) Figure 12 The computing device 1200 depicted may also include an input / output controller 1232 for receiving and processing input from multiple input devices (such as a keyboard, mouse, touchpad, touchscreen, or electronic stylus) or other types of input devices. Similarly, the input / output controller 1232 may provide output to a display (such as a computer monitor, flat panel display, digital projector, printer, or plotter) or other types of output devices. It should be understood that the computing device 1200 may not include... Figure 12 All components shown may include Figure 12 Other components not explicitly shown in the document, or those that can be utilized with Figure 12 The architecture shown is completely different.

[0070] As described in this article, a computing device can be a physical computing device, such as... Figure 12 The computing device 1200. A computing node may also include virtual machine host processes and one or more virtual machine instances. Computer-executable instructions may be indirectly executed by the physical hardware of the computing device by interpreting and / or executing instructions stored and executed in the context of the virtual machine.

[0071] It should be understood that the methods and systems are not limited to any particular method, component, or implementation. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.

[0072] Unless the context clearly specifies otherwise, the singular forms “a,” “an,” and “(the)” as used in this specification and the appended claims include a plural of indicators. Scopes herein may be expressed as from “about” a specific value and / or to “about” other specific values. When such scopes are expressed, another embodiment includes from one specific value and / or to another specific value. Similarly, when a value is expressed as an approximation using the antecedent “about,” it will be understood that a particular value forms another embodiment. It will be further understood that the endpoints of each scope are significant relative to and independent of the other endpoints.

[0073] "Optional" or "optionally" means that the event or situation described below may or may not occur, and the description includes instances of the event or situation occurring and not occurring.

[0074] Throughout the description and claims of this specification, the word "comprise" and variations thereof (such as "comprising" and "comprises") mean "including but not limited to" and are not intended to exclude, for example, other components, integrals, or steps. "Exemplary" means "an example of..." and is not intended to convey indications of preferred or ideal embodiments. "For example" is not used in a limiting sense but for interpretive purposes.

[0075] Components that can be used to perform the described methods and systems are described. When describing combinations, subsets, interactions, groups, etc., of these components, it should be understood that while specific references to each of the various individual and collective combinations and arrangements of these components may not be explicitly described, each component is specifically conceived and described herein for all methods and systems. This applies to all aspects of this application, including but not limited to operations in the described methods. Therefore, if various additional operations exist that can be performed, it should be understood that each of these additional operations can be performed using any specific embodiment or combination of embodiments of the described methods.

[0076] The method and system can be more readily understood by referring to the following detailed description of preferred embodiments and examples included therein, as well as the accompanying drawings and their descriptions.

[0077] As will be understood by those skilled in the art, the methods and systems may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the methods and systems may take the form of a computer program product on a computer-readable storage medium having computer-readable program instructions (e.g., computer software) embodied within the storage medium. More specifically, the methods and systems may take the form of computer software implemented on the web. Any suitable computer-readable storage medium may be utilized, including hard disks, CD-ROMs, optical storage devices, or magnetic storage devices.

[0078] Embodiments of methods and systems are described below with reference to block diagrams and flowcharts illustrating methods, systems, apparatuses, and computer program products. It should be understood that each block in the block diagrams and flowcharts, as well as combinations of blocks in the block diagrams and flowcharts, can be implemented by computer program instructions. These computer program instructions can be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute on the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart.

[0079] These computer program instructions may also be stored in a computer-readable storage medium, which may instruct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of writing comprising computer-readable instructions for implementing the functions specified in one or more boxes of a flowchart. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more boxes of a flowchart.

[0080] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Additionally, in some embodiments, certain method or process blocks may be omitted. The methods and processes described herein are not limited to any particular order, and blocks or states associated with methods and processes may be performed in other suitable orders. For example, described blocks or states may be performed in an order different from that specifically described, or multiple blocks or states may be combined in a single block or state. Example blocks or states may be performed serially, in parallel, or in some other manner. Blocks or states may be added to or removed from the described example embodiments. The example systems and components described herein may be configured differently than described. For example, elements may be added, removed, or rearranged compared to the described example embodiments.

[0081] It should also be understood that the items are illustrated as being stored in memory or storage devices during use, and that these items, or portions thereof, may be transferred between memory and other storage devices for memory management and data integrity purposes. Alternatively, in other embodiments, some or all of the software modules and / or systems may execute in memory on another device and communicate with the illustrated computing system via inter-computer communication. Furthermore, in some embodiments, some or all of the systems and / or modules may be implemented or provided in other ways, such as at least in part as firmware and / or hardware, including but not limited to one or more application-specific integrated circuits (“ASICs”), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field-programmable gate arrays (“FPGAs”), complex programmable logic devices (“CPLDs”), etc. Some or all of the modules, systems, and data structures may also be stored (e.g., as software instructions or structured data) on computer-readable media, such as hard disks, memory, networks, or portable media articles to be read by appropriate devices or via appropriate connections. The system, modules, and data structures can also be transmitted as generated data signals (e.g., as part of a carrier or other analog or digital propagation signal) over various computer-readable transmission media, including wireless and wired / cable-based media, and can take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). In other embodiments, such a computer program product can also take other forms. Therefore, the invention can be practiced with other computer system configurations.

[0082] Although the methods and systems have been described in conjunction with preferred embodiments and specific examples, this is not intended to limit the scope to the specific embodiments illustrated, as the embodiments herein are intended to be illustrative rather than restrictive in all respects.

[0083] Unless otherwise expressly stated, it should not be construed that any method described herein requires its operations to be performed in a particular order. Therefore, no order is intended to be inferred in any way where a method claim does not actually describe the order in which its operations are followed, or where the claims or specification do not otherwise specifically state that the operations are limited to a particular order. This applies to any possible non-express basis for interpretation, including: logical questions concerning the arrangement of steps or the flow of operations; direct meaning derived from grammatical organization or punctuation; and the number or type of embodiments described in the specification.

[0084] It will be apparent to those skilled in the art that various modifications and variations can be made without departing from the scope or spirit of this disclosure. Other embodiments will be apparent to those skilled in the art in light of the description and practice described herein. This specification and example figures are to be considered exemplary only, and the true scope and spirit are indicated by the appended claims.

Claims

1. A method of generating a latent representation of an image using a machine learning model, the method comprising: segmenting and flattening an image into a series of patches; concatenating the series of patches with a sequence of latent tokens; inputting the concatenated patches and latent tokens into an encoder of the machine learning model; generating a one-dimensional latent representation of the image by the encoder; performing vector quantization on the one-dimensional latent representation of the image by a vector quantizer of the machine learning model to generate quantized latent tokens; and reconstructing the image based on the quantized latent tokens by a decoder of the machine learning model.

2. The method of claim 1, wherein generating the one-dimensional latent representation of the image by the encoder further comprises: decoupling a size of the one-dimensional latent representation from a resolution of the image; and generating the one-dimensional latent representation comprising a one-dimensional sequence, wherein the one-dimensional sequence has a same size as the sequence of latent tokens.

3. The method of claim 1, further comprising: merging a sequence of mask tokens into the quantized latent tokens; and inputting the merged mask tokens and the quantized latent tokens into the decoder.

4. The method of claim 3, further comprising: generating the sequence of mask tokens by replicating a single mask token.

5. The method of claim 1, wherein the encoder comprises a vision transformer (ViT) encoder, and wherein the decoder comprises a ViT decoder.

6. The method of claim 1, further comprising: training the machine learning model using a two-stage training paradigm.

7. The method of claim 6, further comprising: performing a first stage of training on the machine learning model using a proxy code to bypass a complex loss function and a generative adversarial network (GAN) architecture, wherein the proxy code comprises discrete codes generated by an off-the-shelf vector quantization GAN (VQGAN) model.

8. The method of claim 7, further comprising: performing a second stage of training on the machine learning model to fine-tune the decoder, and wherein the second stage of training comprises training only the decoder in pixel space using a VQGAN loss while keeping the encoder and the vector quantizer frozen.

9. A system for generating a latent representation of an image using a machine learning model, the system comprising: at least one processor; and at least one memory communicatively coupled to the at least one processor and comprising computer-readable instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: segmenting and flattening an image into a series of patches; concatenating the series of patches with a sequence of latent tokens; inputting the concatenated patches and latent tokens into an encoder of the machine learning model; generating a one-dimensional latent representation of the image by the encoder; performing vector quantization on the one-dimensional latent representation of the image by a vector quantizer of the machine learning model to generate quantized latent tokens; and ​ ​ ​ ​ reconstructing the image based on the quantized latent markers by a decoder of the machine learning model.

10. The system of claim 9, wherein the generating, by the encoder, the one-dimensional latent representation of the image further comprises: decoupling a size of the one-dimensional latent representation from a resolution of the image; and generating the one-dimensional latent representation comprising a one-dimensional sequence, wherein the one-dimensional sequence has a same size as the sequence of latent markers.

11. The system of claim 9, the operations further comprising: generating a sequence of mask markers by replicating a single mask marker; merging the sequence of mask markers into the quantized latent markers; and inputting the merged mask markers and the quantized latent markers into the decoder.

12. The system of claim 9, wherein the encoder comprises a vision transformer (ViT) encoder, and wherein the decoder comprises a ViT decoder.

13. The system of claim 9, the operations further comprising: training the machine learning model using a two-stage training paradigm.

14. The system of claim 13, the operations further comprising: performing a first stage training of the machine learning model using a proxy code to bypass a complex loss function and a generative adversarial network (GAN) architecture, wherein the proxy code comprises discrete codes generated by an off-the-shelf vector quantization GAN (VQGAN) model; and performing a second stage training of the machine learning model to fine-tune the decoder, and wherein the second stage training comprises training only the decoder in pixel space while keeping the encoder and the vector quantizer frozen using a VQGAN loss.

15. A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a processor, cause the processor to perform operations comprising: segmenting and flattening an image into a series of patches; pasting the series of patches with a sequence of latent markers; inputting the pasted patches and the latent markers into an encoder of the machine learning model; generating a one-dimensional latent representation of the image by the encoder; performing vector quantization of the one-dimensional latent representation of the image by a vector quantizer of the machine learning model to generate quantized latent markers; and reconstructing the image based on the quantized latent markers by a decoder of the machine learning model.

16. The non-transitory computer-readable storage medium of claim 15, wherein the generating, by the encoder, the one-dimensional latent representation of the image further comprises: decoupling a size of the one-dimensional latent representation from a resolution of the image; and generating the one-dimensional latent representation comprising a one-dimensional sequence, wherein the one-dimensional sequence has a same size as the sequence of latent markers.

17. The non-transitory computer-readable storage medium of claim 15, the operations further comprising: generating a sequence of mask markers by replicating a single mask marker; merging the sequence of mask markers into the quantized latent markers; and inputting the merged mask markers and the quantized latent markers into the decoder. The merged mask label and quantized latent label are input into the decoder.

18. The non-transitory computer-readable storage medium of claim 15, wherein the encoder comprises a vision transformer (ViT) encoder, and wherein the decoder comprises a ViT decoder.

19. The non-transitory computer-readable storage medium of claim 15, the operations further comprising: training the machine learning model using a two-stage training paradigm.

20. The non-transitory computer-readable storage medium of claim 19, the operations further comprising: performing first stage training of the machine learning model using a proxy code that bypasses a complex loss function and generative adversarial network (GAN) architecture, wherein the proxy code comprises discrete codes generated by an off-the-shelf vector quantization GAN (VQGAN) model; and performing second stage training of the machine learning model to fine-tune the decoder, and wherein the second stage training comprises training only the decoder in pixel space while keeping the encoder and the vector quantizer frozen using a VQGAN loss.

Images