Visual cue adjustment for generative transfer learning

Abstract

A system and method for training and using a cue element generator to generate a set of cue elements that, when fed into a pre-trained generative image transformer (e.g., an autoregressive transformer, a continuous diffusion model, a non-autoregressive transformer, or a discrete diffusion model), may bias the output of the generative image transformer to a particular domain (e.g., a non-autoregressive transformer, a continuous diffusion model, a non-autoregressive transformer, or a discrete diffusion model). ), towards a specific image class, towards a specific training instance, etc. In some examples, a cue element generator may be used to generate a set of different cue element sequences, which may then be sequentially fed to a pre-trained non-autoregressive generative image transformer when the pre-trained non-autoregressive generative image transformer iteratively generates each image in each time step, and therefore, more diversity can be introduced into the final output of the transformer.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of the filing date of U.S. Provisional Application No. 63 / 406,841, filed on September 15, 2022, the entire disclosure of which is incorporated herein by reference. Background Art

[0002] There are many different types of generative image models that are capable of generating diverse and semantically meaningful images that look real and are free of obvious visual artifacts. Generative adversarial networks ("GANs") can provide state-of-the-art speed, but have some limitations in the diversity and realism of the images they can generate. Likelihood-based models such as autoregressive transformers and continuous diffusion models can provide improved image quality over GANs, but may require hundreds of steps to synthesize an image, making them several orders of magnitude slower. Recently, developments in non-autoregressive transformers and discrete diffusion models have provided a promising middle ground, achieving image quality comparable to state-of-the-art autoregressive transformer and continuous diffusion models, while achieving such image quality up to two orders of magnitude faster than autoregressive transformer and continuous diffusion models. However, as the quality of such models continues to improve across multiple domains, attention is increasingly turning to how such models, once trained, can be efficiently adapted to generate images in new domains. Summary of the invention

[0003] The technology relates to systems and methods for training and using a hint token generator to generate a hint token set that, when fed into a pre-trained generative image transformer (e.g., an autoregressive transformer, a continuous diffusion model, a non-autoregressive transformer, or a discrete diffusion model), can bias the output of the generative image transformer toward a particular domain (e.g., toward a particular image class, toward a particular training instance, etc.). In some aspects, the technology relates to systems and methods for training a hint token generator using training examples, each of which includes a target token sequence representing a first vector quantized image and a first set of one or more identifiers (e.g., a class identifier, an instance identifier, etc.), at least one of the first set of one or more identifiers being related to a subject of the first vector quantized image. In such cases, the hint token generator can generate a first hint token sequence based at least in part on the first set of one or more identifiers, and the pre-trained generative image transformer can generate a first output token sequence based at least in part on the first hint token sequence. This first output word-gram sequence will represent the second vector quantized image and can be generated over any suitable number of time steps. The processing system can compare the first output word-gram sequence to the target word-gram sequence to generate a loss value for the training example in question, which can then be used (alone, as part of an aggregated loss value representing multiple training examples, and / or together with other types of loss values) to modify one or more parameters of the prompt word-gram generator. In addition, this process can be repeated using any suitable optimization routine (e.g., stochastic gradient descent) until the prompt word-gram generator learns to generate prompts that cause the pre-trained generative image transformer to generate a first output word-gram sequence that is very close to (or the same as) the target word-gram sequence for each training example.

[0004] In addition, in some aspects, the technology relates to systems and methods for generating images using a trained hint word-gram generator and a pre-trained generative image transformer that will be biased toward the specific domain on which the word-gram generator was trained. In some examples, after the training just described, the hint word-gram generator can generate a second hint word-gram sequence based at least in part on a second set of one or more identifiers (e.g., class identifiers, instance identifiers, etc.), and then the pre-trained generative image transformer can generate a second output word-gram sequence based at least in part on the second hint word-gram sequence. The second output word-gram sequence will also represent a second vector quantized image and can be generated by any suitable number of steps. In this way, by using a specific class identifier from the training set, the hint word-gram generator can cue the pre-trained generative image transformer to generate a second output word-gram sequence that, when converted to a second vector quantized image, will look similar to an image in this specific class. Similarly, by using a particular instance identifier, the prompted word-gram generator can prompt a pre-trained generative image transformer to generate a second output word-gram sequence that, when converted to a second vector quantized image, will look similar to the particular training example with this instance identifier. Notably, by using the prompted word-gram generator and training methods of the present technique with a pre-trained generative image transformer, significantly better and more efficient knowledge transfer can be achieved than is possible with GANs, and this can be achieved in a wide range of new domains. For example, in some aspects, a prompted word-gram generator trained according to the present technique using only 5 training images per class can enable a pre-trained generative image transformer to produce images with significantly lower Frechet Inception Distance (“FID”) scores than is possible using 20 to 100 times more images per class according to GAN-based transfer learning methods.

[0005] Further, in some aspects, the technology relates to systems and methods for sequentially feeding a set of different prompt word-gram sequences to a pre-trained non-autoregressive generative image transformer (e.g., a non-autoregressive transformer or a discrete diffusion model) as the pre-trained non-autoregressive generative image transformer iteratively generates each image at each time step. This process can be used to introduce more diversity into the output of the generative image transformer. For example, by interpolating between a second prompt word-gram sequence for a given instance (e.g., a given picture of a dog) and a third prompt word-gram sequence for the class of the given instance (e.g., dog), a set of prompt word-gram sequences can be generated that will cause the pre-trained generative image transformer to generate a final output word-gram sequence that, when converted to a final image, appears to be a consistent variation of the class of the training example with the instance identifier (e.g., different dogs of the same breed, color, face shape, etc.). Similarly, by interpolating between a second clue word-gram sequence for one instance (e.g., a picture of a golden retriever) and a third clue word-gram sequence for another instance (e.g., a picture of a Swiss mountain dog), a set of clue word-gram sequences can be generated that will cause the pre-trained generative image transformer to generate a final output word-gram sequence that, when converted to a final image, appears to be a mixture of the visual characteristics of the two training examples (e.g., a dog that looks like a mix of a golden retriever and a Swiss mountain dog).

[0006] In one aspect, the present disclosure describes a computer-implemented method comprising: (1) for each given training example in a plurality of training examples, the given training example comprising a target word-gram sequence representing a first vector quantized image and a first set of one or more identifiers, at least one of the first set of one or more identifiers being related to a subject of the first vector quantized image: using a hint word-gram generator to generate a first hint word-gram sequence based at least in part on the first set of one or more identifiers; using a pre-trained generative image transformer to generate a first output word-gram sequence based at least in part on the first hint word-gram sequence, the first output word-gram sequence representing a second vector quantized image; and using one or more processors of a processing system to compare the first output word-gram sequence with the target word-gram sequence to generate a loss value for the given training example; and (2) using the one or more processors to modify one or more parameters of the hint word-gram generator based at least in part on the loss values ​​generated for the plurality of training examples. In some aspects, the hint word-gram generator comprises two or more multilayer perceptrons, and modifying the one or more parameters of the hint word-gram generator comprises modifying one or more parameters of each of the two or more multilayer perceptrons. In some aspects, the first set of one or more identifiers for a given training example includes a class identifier associated with a subject of the first vector quantized image. In some aspects, the first set of one or more identifiers for a given training example includes an instance identifier associated with the first vector quantized image. In some aspects, the method further includes: generating a second hint word-gram sequence based at least in part on the second set of one or more identifiers using a hint word-gram generator; and generating a second output word-gram sequence based at least in part on the second hint word-gram sequence using a pre-trained generative image transformer, the second output word-gram sequence representing a third vector quantized image. In some aspects, the method further includes: generating a second hint word-gram sequence using a hint word-gram generator based at least in part on a second set of one or more identifiers; generating a third hint word-gram sequence using the hint word-gram generator based at least in part on a third set of one or more identifiers, the third set of one or more identifiers being different from the second set of one or more identifiers due to at least one identifier; generating one or more intermediate hint word-gram sequences based on the second hint word-gram sequence and the third hint word-gram sequence using one or more processors; generating a second output word-gram sequence based at least in part on the second hint word-gram sequence using a pre-trained generative image transformer, the second output word-gram sequence representing a third vector quantized image; and generating a third output word-gram sequence based at least in part on the second output word-gram sequence and one of the one or more intermediate hint word-gram sequences using a pre-trained generative image transformer, the third output word-gram sequence representing a fourth vector quantized image.In some aspects, the method further includes: using a pre-trained generative image transformer to generate a fourth output word-gram sequence based at least in part on one of the one or more intermediate prompt word-gram sequences, the fourth output word-gram sequence representing a fifth vector quantized image; and using a pre-trained generative image transformer to generate a fifth output word-gram sequence based at least in part on the fourth output word-gram sequence and the third prompt word-gram sequence, the fifth output word-gram sequence representing a sixth vector quantized image. In some aspects, the method further includes generating an output image based on the fifth output word-gram sequence.

[0007] In another aspect, the present disclosure describes a non-transitory computer program product comprising computer readable instructions that, when executed by a processing system, cause the processing system to perform any of the methods described in the previous paragraphs.

[0008] In another aspect, the present disclosure describes a processing system comprising: (1) a memory storing a pre-trained generative image transformer and a hint word-gram generator; and (2) one or more processors coupled to the memory and configured to train the hint word-gram generator according to a training method, the training method comprising: (a) for each given training example of a plurality of training examples, the given training example comprising a target word-gram sequence representing a first vector quantized image and a first set of one or more identifiers, at least one identifier in the first set of one or more identifiers being related to a subject of the first vector quantized image: using the hint word-gram generator to generate a first hint word-gram sequence based at least in part on the first set of one or more identifiers; using the pre-trained generative image transformer to generate a first output word-gram sequence based at least in part on the first hint word-gram sequence, the first output word-gram sequence representing a second vector quantized image; and comparing the first output word-gram sequence to the target word-gram sequence to generate a loss value for the given training example; and (b) modifying one or more parameters of the hint word-gram generator based at least in part on the loss values ​​generated for the plurality of training examples. In some aspects, the hint word-gram generator comprises a multilayer perceptron. In some aspects, the hint word-gram generator comprises two or more multilayer perceptrons. In some aspects, the one or more processors are configured to modify one or more parameters of the hint word-gram generator, including modifying one or more parameters of each of the two or more multilayer perceptrons. In some aspects, the one or more processors are further configured to: generate a second hint word-gram sequence based at least in part on a second set of one or more identifiers using the hint word-gram generator; and generate a second output word-gram sequence based at least in part on the second hint word-gram sequence using a pre-trained generative image transformer, the second output word-gram sequence representing a third vector quantized image.In some aspects, the one or more processors are further configured to: generate a second hint word-gram sequence using a hint word-gram generator based at least in part on a second set of one or more identifiers; generate a third hint word-gram sequence using the hint word-gram generator based at least in part on a third set of one or more identifiers, the third set of one or more identifiers being different from the second set of one or more identifiers due to at least one identifier; generate one or more intermediate hint word-gram sequences based on the second hint word-gram sequence and the third hint word-gram sequence; generate a second output word-gram sequence based at least in part on the second hint word-gram sequence using a pre-trained generative image transformer, the second output word-gram sequence representing a third vector quantized image; and generate a third output word-gram sequence based at least in part on the second output word-gram sequence and one of the one or more intermediate hint word-gram sequences using a pre-trained generative image transformer, the third output word-gram sequence representing a fourth vector quantized image. In some aspects, one or more processors are further configured to: generate a fourth output word-gram sequence using a pre-trained generative image transformer based at least in part on one of the one or more intermediate prompt word-gram sequences, the fourth output word-gram sequence representing a fifth vector quantized image; and generate a fifth output word-gram sequence based at least in part on the fourth output word-gram sequence and the third prompt word-gram sequence using a pre-trained generative image transformer, the fifth output word-gram sequence representing a sixth vector quantized image. In some aspects, one or more processors are further configured to generate an output image based on the fifth output word-gram sequence. In some aspects, one or more processors are configured to generate an output image using a decoder of a pre-trained generative image transformer. In some aspects, the pre-trained generative image transformer is an autoregressive image transformer. In some aspects, the pre-trained generative image transformer is a non-autoregressive image transformer. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a functional diagram of an example system in accordance with aspects of the present disclosure.

[0010] Figure 2 is a functional diagram of an example system in accordance with aspects of the present disclosure.

[0011] Figure 3 is a flow chart illustrating an exemplary process flow for generating images using an autoregressive transformer and a sequence of cue tokens in accordance with aspects of the present disclosure.

[0012] Figure 4is a flow chart illustrating an exemplary process flow for generating an image using a non-autoregressive transformer and a sequence of cue tokens in accordance with aspects of the present disclosure.

[0013] Figure 5 is a flow chart illustrating an exemplary process flow for generating a cue word-gram sequence using a cue word-gram generator in accordance with aspects of the present disclosure.

[0014] Figure 6 is a flow chart illustrating an exemplary process flow for generating an output word-gram sequence based on a prompt word-gram sequence using a pre-trained generative image transformer according to aspects of the present disclosure.

[0015] Figure 7 is a diagram illustrating an exemplary image generated by a pre-trained generative image transformer based on a single example prompt in accordance with aspects of the present disclosure.

[0016] Figure 8 is a diagram illustrating an exemplary image generated by a pre-trained generative image transformer based on a prompt set interpolated between a prompt set for a single instance and a prompt set for a class, in accordance with aspects of the present disclosure.

[0017] Fig. 9 is a diagram illustrating an exemplary image generated by a pre-trained generative image transformer based on a prompt set interpolated between prompt sets for two different instances in accordance with aspects of the present disclosure.

[0018] Fig.10 An exemplary method for training a cue word-gram generator according to aspects of the present disclosure is depicted.

[0019] Fig.11 Depicted are exemplary methods for generating a sequence of cue word-grams using a trained cue word-gram generator and generating an output word-gram sequence based on the sequence of cue word-grams using a pre-trained generative image transformer in accordance with aspects of the present disclosure.

[0020] Fig. 12A Depicted are exemplary methods for generating two hint word-gram sequences using a trained hint word-gram generator, generating one or more intermediate hint word-gram sequences based on the two hint word-gram sequences generated by the word-gram generator, and sequentially feeding two of the hint word-gram sequences to a pre-trained generative image transformer to generate two consecutive output word-gram sequences in accordance with aspects of the present disclosure.

[0021] Fig. 12B Describes various aspects of the present disclosure. Fig. 12A An exemplary method constructed by feeding two other prompt word-gram sequences in the generated prompt word-gram sequence sequentially to a pre-trained generative image transformer to generate two additional consecutive output word-gram sequences. DETAILED DESCRIPTION

[0022] The technology will now be described with respect to the following exemplary systems and methods.Common reference numerals between the figures depicted and described below are intended to identify like features. Example System

[0023] Figure 1 A high-level system diagram 100 of an exemplary processing system 102 for performing the methods described herein is shown. The processing system 102 may include one or more processors 104 and a memory 106 storing instructions 108 and data 110. The instructions 108 and data 110 may include a pre-trained generative image transformer (e.g., an autoregressive transformer, a continuous diffusion model, a non-autoregressive transformer, or a discrete diffusion model) and / or a hint word generator (e.g., one or more multilayer perceptrons), as further described below. In addition, the data 110 may store training examples to be used in training the hint word generator, outputs from the hint word generator and / or the pre-trained generative image transformer generated during training, training signals and / or loss values ​​generated during such training, and / or outputs from the hint word generator and / or the pre-trained generative image transformer generated during inference.

[0024] The processing system 102 may reside on a single computing device. For example, the processing system 102 may be a server, a personal computer, or a mobile device, and the pre-trained generative image transformer and / or the prompt word unit generator may therefore be local to the single computing device. Similarly, the processing system 102 may reside on a cloud computing system or other distributed system. In such cases, the pre-trained generative image transformer and / or the prompt word unit generator may be distributed across two or more different physical computing devices. For example, the processing system may include a first computing device storing layers 1-n of a pre-trained generative image transformer and / or a prompt word unit generator having m layers and a second computing device storing layers nm of the pre-trained generative image transformer and / or the prompt word unit generator. In such cases, the first computing device may be a device (e.g., a personal computer, a mobile phone, a tablet computer, etc.) with less memory and / or processing power than the memory and / or processing power of the second computing device, and vice versa. Likewise, in some aspects of the technology, the processing system may include one or more computing devices storing pre-trained generative image transformers, and one or more separate computing devices storing hint word-unit generators. Further, in some aspects of the technology, data (e.g., training examples, model outputs, loss values, etc.) used and / or generated during training or inference of the generative image transformer and / or hint word-unit generator may be stored on a computing device different from the generative image transformer and / or hint word-unit generator.

[0025] Further in this regard, Figure 2A high-level system diagram 200 is shown in which the exemplary processing system 102 just described is distributed across two computing devices 102a and 102b, each of which may include one or more processors (104a, 104b) and memory (106a, 106b) storing instructions (108a, 108b) and data (110a, 110b). The processing system 102, including the computing devices 102a and 102b, is shown as communicating with one or more websites and / or remote storage systems including a website 204 and a remote storage system 212 via one or more networks 202. In this example, the website 204 includes one or more servers 206a-206n. Each of the servers 206a-206n may have one or more processors (e.g., 208) and associated memory (e.g., 210) storing instructions and data (including the content of one or more web pages). Likewise, although not shown, the remote storage system 212 may also include one or more processors and memory storing instructions and data. In some aspects of the technology, the processing system 102, including the computing devices 102a and 102b, can be configured to retrieve data from one or more of the website 204 and / or the remote storage system 212 for use during the training of the prompt word-gram generator. For example, in some aspects, the first computing device 102a can be configured to retrieve training images or target word-gram sequences and associated identifiers (e.g., class identifiers, instance identifiers, etc.) from the remote storage system 212. These training images or target word-gram sequences and associated identifiers can then be fed to the prompt word-gram generator hosted on the first computing device 102a to generate prompts, which in turn will be fed into the pre-trained generative image transformer hosted on the second computing device 102b.

[0026] The processing systems described herein may be implemented on any type of computing device, server, or collection thereof, such as any type of general purpose computing device, and may further include other components typically found in a general purpose computing device or server. Likewise, the memory of such a processing system may be of any non-transitory type capable of storing information accessible by a processor of the processing system. For example, the memory may include non-transitory media such as hard drives, memory cards, optical disks, solid-state, tape storage, and the like. Computing devices suitable for the roles described herein may include different combinations of the foregoing, whereby different portions of instructions and data are stored on different types of media.

[0027] In all cases, the computing devices described herein may further include any other components commonly used in conjunction with computing devices such as user interface subsystems. The user interface subsystem may include one or more user inputs (e.g., a mouse, keyboard, stylus, touch screen, and / or microphone) and one or more electronic displays (e.g., a monitor with a screen or any other electrical device operable to display information). Output devices such as speakers, lights, and vibration, pulse, or tactile elements other than electronic displays may also be included in the computing devices described herein.

[0028] The one or more processors included in each computing device may be any conventional processor, such as a commercially available central processing unit ("CPU"), a graphics processing unit ("GPU"), a tensor processing unit ("TPU"), and the like. Alternatively, one or more processors may be dedicated devices, such as ASICs or other hardware-based processors. Each processor may have multiple cores capable of operating in parallel. The processor, memory, and other elements of a single computing device may be stored in a single physical housing, or may be distributed between two or more housings. Similarly, the memory of a computing device may include a hard drive or other storage medium located in a housing different from the housing of the processor (such as in an external database or networked storage device). Therefore, references to processors or computing devices will be understood to include a collection of processors or computing devices or memories that may or may not operate in parallel, as well as references to one or more servers of a load-balanced server farm or a cloud-based system.

[0029] The computing device described herein can store instructions (such as machine code) that can be directly executed by a processor or instructions (such as scripts) that are indirectly executed. The computing device can also store data, which can be retrieved, stored or modified by one or more processors according to the instructions. The instructions can be stored on a computing device readable medium as a computing device code. In this regard, the terms "instructions" and "programs" can be used interchangeably herein. Instructions can also be stored in an object code format to be directly processed by a processor, or stored in any other computing device language, which includes a collection of scripts or independent source code modules that are interpreted or compiled in advance on demand. For example, the programming language can be C#, C++, JAVA or another computer programming language. Similarly, any component of an instruction or program can be implemented with a computer scripting language (such as JavaScript, PHP, ASP or any other computer scripting language). In addition, any of these components can be implemented using a combination of a computer programming language and a computer scripting language. Example Method

[0030] Figure 3 is a flow chart illustrating an exemplary process flow 300 for generating an image using an autoregressive transformer and a sequence of cue tokens in accordance with aspects of the present disclosure.

[0031] In this regard, Figure 3 Exemplary outputs representing an autoregressive transformer (or continuous diffusion model) at time steps t = 0, t = 1, t = 2, t = 100, t = 160, and t = 256 are shown. In this example, as shown at time step t = 0, the autoregressive transformer will begin by accepting as input a prompt word-gram sequence 302. Then, at time step t = 1, the autoregressive transformer will predict a first word-gram 304 based on the prompt word-gram sequence 302. Next, in time step t = 2, the autoregressive transformer will predict a second word-gram 306 based on the prompt word-gram sequence 302 and the predicted first word-gram 304. Next, in time step t = 3 (not shown), the autoregressive transformer will predict a third word-gram based on the prompt word-gram sequence 302 and a sequence of previously predicted words (first word-gram 304 and second word-gram 306). This process is repeated sequentially for each next token until the autoregressive transformer has predicted the complete sequence of tokens, as shown at time step t = 256.

[0032] The resulting final output word-gram sequence can then be converted to an image in any suitable manner. For example, in some aspects, the output word-gram sequence can represent a vector quantized image, and thus can be converted to a corresponding image by processing the sequence through a decoder of a vector quantized autoencoder. In this regard, in some aspects, each word-gram of the output vector can correspond to a different pixel of the image. Likewise, in some aspects, each word-gram of the output vector can correspond to a group of pixels. For example, in some aspects, the final image can be a 256×256 pixel image, and Figure 3 Each element of the 256-element output word-gram sequence shown in can correspond to a different 16×16 pixel block. Similarly, in some aspects, the final image can be a 512×512 pixel image, and Figure 3 Each element of the 256-element output word-gram sequence shown in may correspond to a different 32×32 pixel block. Furthermore, although the output word-gram sequence is shown in Figure 3 , but it will be appreciated that it may have any other suitable format. Thus, in some aspects of the technology, the output word-gram sequence may be a flattened sequence representing a left-to-right, top-to-bottom scan of a grid covering the desired output image.

[0033] In some aspects of the technology, the autoregressive transformer can include a vector quantization autoencoder capable of decoding the output word-gram sequence into a corresponding image. Likewise, in some aspects of the technology, the output word-gram sequence from the autoregressive transformer can be converted into a corresponding image by a decoder of a separate vector quantization autoencoder.

[0034] Figure 4 is a flow chart illustrating an exemplary process flow 400 for generating an image using a non-autoregressive transformer and a sequence of cue tokens in accordance with aspects of the present disclosure.

[0035] In this regard, Figure 4 An exemplary output for time steps t = 0 to t = 8 representing a non-autoregressive transformer (or discrete diffusion model) is shown. In this example, as shown at time step t = 0, the non-autoregressive transformer will start by accepting a prompt word sequence 402 and a fully masked vector 404 as input. Then, in time step t = 1, the non-autoregressive transformer will predict the value of each of the masked words in the vector 404 based on the prompt word sequence 402, and will retain a predetermined number of these predicted values. The non-autoregressive transformer can determine which of the predicted values ​​to retain based on any suitable criteria. For example, in some aspects, the non-autoregressive transformer can be configured to also generate a confidence score for each of the values ​​it predicts, and can be configured to select which values ​​to retain based on which values ​​have the highest confidence scores. Likewise, in some aspects, a separate model (e.g., a learned word-gram critic) can be configured to process the output of the non-autoregressive transformer at each time step and predict which word-gram values ​​are considered to be most realistic and should therefore be retained in the next time step. Notably, where a separate model is used, the separate model can be configured to examine all word-grams of the vector 404 at each time step (those retained from the previous time step and those predicted in the current time step), thereby allowing word-grams retained in one time step to be masked in the next time step when other predicted word-grams are considered more realistic. This can minimize the "anchoring" effect in which word-grams saved from earlier time steps end up overly influencing the final output, and can therefore improve the variability and / or quality of the final output of the non-autoregressive transformer.

[0036] exist Figure 4In the example of , it is assumed that the values ​​predicted for the three word-grams 406 will be retained in unmasked form as input to time step t = 2. Therefore, at time step t = 2, the non-autoregressive transformer will predict the value of each of the masked word-grams in vector 404 based on the prompt word-gram sequence 402 and the values ​​of the three word-grams 406 retained from step t = 1, and a predetermined number of these predicted values ​​will be retained for use as input to time step t = 3. This process will be repeated according to the appropriate masking schedule until the final time step (in this example, time step t = 8), where all predictions of the non-autoregressive transformer will be included in the final output vector. Here again, the output word-gram sequence in this final output vector can then be converted to an image in any suitable manner. For example, in some aspects, the output word-gram sequence can represent a vector quantized image, and can therefore be converted to a corresponding image by processing the sequence through a decoder of a vector quantized autoencoder. In some aspects of the technology, the non-autoregressive transformer can include a vector quantized autoencoder capable of decoding the output word-gram sequence into a corresponding image. Likewise, in some aspects of the technique, the output token sequence from a non-autoregressive transformer can be converted to a corresponding image by a decoder of a separate vector quantization autoencoder.

[0037] Figure 5 is a flow chart illustrating an exemplary process flow 500 for generating a cue word-gram sequence using a cue word-gram generator 502 in accordance with aspects of the present disclosure.

[0038] exist Figure 5 In the example of FIG. 5 , it is assumed that the hint word unit generator 502 will include four separate multi-layer perceptrons MLP C (504), MLP P (506), MLP F (508) and MLP T (510). In this case, the first multi-layer perceptron MLP C (504) is shown as accepting information about a given class identifier and / or instance identifier 505 for each training example. Figure 5 It is assumed that there will be a batch of training examples. Therefore, MLP CThe output of (504) is shown as a vector of dimensions B×1×P×F, where B represents the number of training examples in the batch, P is the specified hidden dimension of the hint token generator 502, and F is a factor value (509) that can be set to any value greater than or equal to 1 to effectively increase the number of parameters without requiring all of them to be learnable. For example, in some aspects, a value of 1 can be used for non-autoregressive transformers (e.g., similar to Figure 4 ), while larger values ​​(e.g., 16) can be used for autoregressive transformers (e.g., similar to Figure 3 Those autoregressive transformers shown in ).

[0039] In this case, the second multilayer perceptron MLP P (506) is shown as accepting a position vector 507, where each element corresponds to the position of a different word in the expected final output hint word sequence S. Thus, the MLP P The output of (506) is shown as a vector of dimensions 1 x S x P x F, where S represents the number of word-grams in the expected final output prompt word-gram sequence.

[0040] like Figure 5 As shown, MLP C (504) and MLP P The outputs of (506) are then combined to generate a vector of dimension B×S×P×f. This can be done in any suitable manner. For example, in some aspects, the MLP C The output of (504) may be replicated B times in the first dimension (thus generating a replicated vector of dimensions B×S×P×F), and the MLP P The output of (506) can be replicated S times in the second dimension (thus generating another replicated vector of dimensions B×S×P×F), and then the two replicated vectors can be summed element-wise (thus generating a summed vector of dimensions B×S×P×F).

[0041] Further, in Figure 5 In the example, the third multilayer perceptron MLP F (508) is shown to accept a predetermined factor value 509. Thus, the MLP F The output of is shown as a vector of dimension 1×1×1×F, where F represents the factor value 509. This vector can then be combined in any suitable way with the vector generated by the combined MLP C (504) and MLP P The summed vector combination generated by the replicated output of (506). For example, Figure 5 As shown, by combining MLP C (504) and MLP P The summed vector produced by the replicated output of (506) can be element-wise multiplied by the MLP F The output of (508) is then further summed in the F dimension, thus producing an output vector having dimensions B×S×P.

[0042] The resulting B×S×P dimensional vector can then be fed to the fourth multi-layer perceptron MLP T (510) to produce a final hint word-gram sequence 512. In this case, the final hint word-gram sequence 512 is assumed to have a length of S words, where each word-gram has a dimension of D. Therefore, in some aspects of the technique, the final hint word-gram sequence 512 can be represented as a vector of dimension S×D.

[0043] Figure 6 is a flow chart illustrating an exemplary process flow 600 for generating an output word-gram sequence based on a prompt word-gram sequence using a pre-trained generative image transformer 601 in accordance with aspects of the present disclosure.

[0044] The pre-trained generative image transformer 601 can be any suitable type of transformer, such as (for example, as described above with respect to Figure 3 ) autoregressive transformer or continuous diffusion model (e.g., as discussed above with respect to Figure 4 Although the pre-trained generative image transformer 601 can have any suitable number of layers and parameters, in Figure 6 In the example of , for simplicity, it is assumed that the pre-trained generative image transformer 601 will have two layers 601-1, 601-2. Thus, the prompt word sequence 602 (e.g., consisting of Figure 5The final hint word-gram sequence 512 output by the hint word-gram generator 502 is appended to an empty vector or fully masked vector 603, which includes an element for each word-gram representing the expected final output word-gram sequence. The first layer 601-1 of the pre-trained generative image transformer 601 will then produce an intermediate vector. In this case, assuming that the first layer 601-1 of the generative image transformer 601 is configured to generate an intermediate vector of the same dimension as its input, the intermediate vector will therefore include a word-gram sequence 604 of length S based on the initial sequence of S hint words 602, and a word-gram set 605 based on the initial values ​​in the vector 603. The intermediate vector will then be passed to the second layer 601-2 of the pre-trained generative image transformer 601 that produces the final vector. Here again, assume that the second layer 601-2 of the generative image transformer 601 is configured to generate a final vector of the same dimension as its input (intermediate vector), which will therefore include a word-gram sequence 606 of length S based on the intermediate word-gram sequence 604, and a word-gram set 607 based on the intermediate group word-gram 605. In this example, assume that the word-gram sequence 606 will be discarded, thereby producing a final output word-gram sequence 607, which can then be decoded to generate the corresponding image. Here again, as described above with respect to Figure 3 and Figure 4 As discussed, this may be done in any suitable manner, such as by processing the final output word-gram sequence 607 through a decoder of a vector quantized autoencoder.

[0045] Figure 7 is a diagram 700 illustrating an exemplary image generated by a pre-trained generative image transformer based on a single example prompt in accordance with aspects of the present disclosure.

[0046] In this regard, Figure 7 The example of assumes that a prompt word unit generator (not shown) has been trained on a training sample set, and image 702 is an example in the training sample set. Example diagram 700 shows a prompt word unit generator (e.g., Figure 5 The prompt word-gram generator 502 will be based on at least the instance identifier of the image 702 (e.g., according to Figure 5 500) to generate a prompt word sequence 704. In this example, at each time step, the prompt word sequence 704 is used as (e.g., as described above with respect to Figure 4The input of a pre-trained non-autoregressive generative image transformer or discrete diffusion model (configured as discussed above). Thus, the prompt word sequence 704 is shown as first appended to an empty vector or fully masked vector 706, which includes an element for each word of the expected output word sequence. Then, as described above with respect to Figure 4 In the same manner as described in process flow 400 of , the prompt word sequence 704 and vector 706 will be fed to a pre-trained non-autoregressive generative image transformer, which will predict the value of each of the word-grams in vector 706 based on the prompt word sequence 704, and will retain a predetermined number of these predicted values. In this example, it is assumed that the value predicted for one word-gram 708 is retained from the prediction at time step t=1. Then, at time step t=2, the non-autoregressive transformer will predict the value of each of the masked word-grams in vector 706 based on the prompt word sequence 704 and the value of one word-gram 708 retained from step t=1, and will retain a predetermined number of these predicted values ​​(again shown as shaded). As described above, this process will be repeated according to the appropriate masking schedule until the final time step (in this example, time step t=12), where all of the predictions of the non-autoregressive transformer will be included in the final output vector, which includes the final output word-gram sequence.

[0047] Here again, the output word-gram sequence generated in any or all of the time steps can be converted into a corresponding image, and this can be done as above with respect to Figure 4 The method may be performed in any suitable manner as described. For example, in some aspects, the output word-meta sequence at each time step may represent a vector quantized image, and thus may be converted to a corresponding image by processing the sequence through a decoder of a vector quantized autoencoder. In some aspects of the technology, a non-autoregressive transformer or a discrete diffusion model may include a vector quantized autoencoder capable of decoding the output word-meta sequence into a corresponding image. Similarly, in some aspects of the technology, the output word-meta sequence from a non-autoregressive transformer may be converted to a corresponding image by a decoder of a separate vector quantized autoencoder.

[0048] To show how the output of a non-autoregressive transformer or discrete diffusion model can evolve at each time step, Figure 7Exemplary images corresponding to the output at each time step (positioned below the grid representing vector 706 for that time step) are shown. Thus, the prediction of the generative image transformer in the first time step t = 1 is shown as image 710, and the prediction of the transformer in the final time step t = 12 is shown as image 712.

[0049] Figure 8 is a diagram 800 illustrating an exemplary image generated by a pre-trained generative image transformer based on a prompt set interpolated between a prompt set for a single instance and a prompt set for a class, in accordance with aspects of the present disclosure.

[0050] In this regard, Figure 8 Assume that a hint word unit generator (not shown) has been trained on a training sample set including two or more images with a class identifier of "dog", and image 802 is an example in the training sample set. Example diagram 800 shows a hint word unit generator (e.g., Figure 5 The prompt word-gram generator 502 will be based on at least the instance identifier of the image 802 (e.g., according to Figure 5 500) to generate a first prompt word sequence 804, and based at least on the class identifier 803 of "dog" (e.g., also based on Figure 5 The first hint word-gram sequence 804 and the second hint word-gram sequence 806 are then used to generate a hint word-gram sequence set (804, 810, 812, 806), which can be used as a hint word-gram sequence in successive time steps according to a suitable schedule (e.g., as described above with respect to Figure 4 The processing system may be configured as discussed above to provide an input to a pre-trained non-autoregressive generative image transformer or discrete diffusion model. The set of prompt word-gram sequences may be generated in any suitable manner. For example, in some cases, the processing system may interpolate between the first prompt word-gram sequence 804 and the second prompt word-gram sequence 806 to generate one or more intermediate prompt word-gram sequences (e.g., prompt word-gram sequences 810, 812).

[0051] In this case, the first cue word-gram sequence 804 is shown as being used in the first time step, followed by the first intermediate cue word-gram sequence 810 being used in the second time step, followed by the second intermediate cue word-gram sequence 812 being used in the third time step, followed by the second cue word-gram sequence 806 being used in all subsequent time steps (the fourth time step through the twelfth time step). As will be appreciated, any suitable number of intermediate cue word-gram sequences may be generated and used, and each cue word-gram sequence may be used in one or more time steps according to any suitable schedule.

[0052] Here again, to show how the output of a non-autoregressive transformer or discrete diffusion model may evolve at each time step when using such a set of different cue-word sequences, Figure 8 An exemplary image corresponding to the output for each time step (positioned below the grid representing the prompt word-gram sequence for that time step) is shown. As can be seen, by interpolating between a first prompt word-gram sequence 804 for a given instance (e.g., an image 802 of a particular dog) and a second prompt word-gram sequence 806 for the class 803 of this given instance (e.g., "dog"), a set of prompt word-gram sequences (804, 810, 812, 806) can be generated that can be used to influence a pre-trained non-autoregressive generative image transformer or discrete diffusion model to generate a final output word-gram sequence that, when converted to a final image 814, shows another image in this class that is similar to the image of the original instance. In this case, the final image 814 shows a dog having the same colors as those shown in image 802, but in a slightly different pose and with a slightly different face shape.

[0053] Fig. 9 is a diagram 900 illustrating an exemplary image generated by a pre-trained generative image transformer based on a prompt set interpolated between prompt sets for two different instances in accordance with aspects of the present disclosure.

[0054] In this regard, Fig. 9 Assume that a prompt word unit generator (not shown) has been trained on a training sample set, and images 902 and 903 are two examples in the training sample set. Example diagram 900 shows a prompt word unit generator (e.g., Figure 5 The prompt word-gram generator 502 will be based on at least the instance identifier of the image 902 (e.g., according to Figure 5 500) to generate a first prompt word sequence 904, and based at least on the instance identifier of the image 903 (e.g., also based on Figure 5Here again, the first hint word-gram sequence 904 and the second hint word-gram sequence 906 will then be used to generate a hint word-gram sequence set (904, 910, 912, 914, 916, 906), which can be used as a hint word-gram sequence in successive time steps according to a suitable schedule (e.g., as described above with respect to Figure 4 The processing system may be configured as discussed above to provide an input to a pre-trained non-autoregressive generative image transformer or discrete diffusion model. The set of prompt word-gram sequences may be generated in any suitable manner. For example, in some cases, the processing system may interpolate between the first prompt word-gram sequence 904 and the second prompt word-gram sequence 906 to generate one or more intermediate prompt word-gram sequences (e.g., prompt word-gram sequences 910, 912, 914, 916).

[0055] In this case, the first cue word-gram sequence 904 is shown as being used in the first time step, followed by the first intermediate cue word-gram sequence 910 being used in the second time step, followed by the second intermediate cue word-gram sequence 912 being used in the third time step, followed by the third intermediate cue word-gram sequence 412 being used in the fourth time step, followed by the fourth intermediate cue word-gram sequence 916 being used in the fifth time step, followed by the second cue word-gram sequence 906 being used in all subsequent time steps (the sixth time step through the twelfth time step). As will be appreciated, any suitable number of intermediate cue word-gram sequences may be generated and used, and each cue word-gram sequence may be used in one or more time steps according to any suitable schedule.

[0056] Here again, to show how the output of a non-autoregressive transformer or discrete diffusion model may evolve at each time step when using such a set of different cue-word sequences, Fig. 9An exemplary image corresponding to the output for each time step (positioned below the grid representing the prompt word-gram sequence for that time step) is shown. As can be seen, by interpolating between a first prompt word-gram sequence 904 for a given first instance (e.g., first dog image 902) and a second prompt word-gram sequence 906 for a given second instance (e.g., second dog image 903), a set of prompt word-gram sequences (904, 910, 912, 914, 916, 906) can be generated that can be used to influence a pre-trained non-autoregressive generative image transformer or discrete diffusion model to generate a final output word-gram sequence that blends the visual characteristics of the two given instances when converted to a final image 918. In this case, the final image 918 shows a dog having the same colors as those shown in image 903 but having a pose and face shape similar to that of image 902.

[0057] Fig.10 An exemplary method 1000 for training a prompt word unit generator according to aspects of the present disclosure is depicted. In this aspect, the method 1000 can be used to train Figure 5 The prompt word unit generator 502.

[0058] In step 1002, a processing system (e.g., Figure 1 or Figure 2 The processing system 102 selects a given training sample from a plurality of training samples, the given training sample comprising a target word sequence representing a first vector quantized image and a first set of one or more identifiers, at least one identifier in the first set of one or more identifiers being related to a subject of the first vector quantized image.

[0059] The target word sequence may represent the first vector quantized image in any suitable manner, including as described above with respect to Figure 3 , Figure 4 and Figures 7 to 9 The first vector quantized image is represented in the same manner as described. Thus, for example, in some aspects, each word in the target word-unit sequence can correspond to a different pixel of the image. Likewise, in some aspects, each word in the target word-unit sequence can correspond to a group of pixels. For example, in some aspects, for a 256×256 pixel image, the target word-unit sequence can be a 256-element vector, where each element corresponds to a different 16×16 pixel block. Similarly, in some aspects, for a 512×512 pixel image, the target word-unit sequence can be a 256-element vector, where each element corresponds to a different 32×32 pixel block.

[0060] The first set of one or more identifiers may include any suitable type of identifier related to the subject matter of the first vector quantized image that the prompt word-mem generator may be configured to accept. For example, in some aspects of the technology, the first set of one or more identifiers may include the identifiers described above with respect to Figure 5 In addition, in some aspects, the first set of one or more identifiers may include additional identifiers that are not related to the subject matter of the first vector quantized image, such as (for example, as described above with respect to Figure 5 507) and / or (eg, as described above with respect to the position vector Figure 5 The factor value 509 describes a predetermined factor value.

[0061] In step 1004, the processing system generates a first hint word-gram sequence based at least in part on the first set of one or more identifiers using a hint word-gram generator. The hint word-gram generator can be any suitable type of model configured to generate a hint word-gram sequence based on a set of one or more identifiers, and can have any suitable number of parameters. For example, in some aspects of the technology, the hint word-gram generator can include the hint word-gram generator described above. Figure 5 The one or more multilayer perceptrons described by the prompt word unit generator 502 of , for which the number of trainable parameters is P·(F·(C+S)+D). Similarly, the prompt word unit generator can also be configured to generate a first prompt word unit sequence of words of any suitable length and with any suitable dimension. For example, in some aspects of the technology, as described above with respect to Figure 5 As described in the final hint word-gram sequence 512 of FIG. 5 , the first hint word-gram sequence can be a sequence of length S, where each word-gram has a dimension D. Each of P, F, C, S, and D can be set to any suitable value. For example, in some aspects of the technology, S can have a value of 128, P can have a value of 768, D can have a value of 768, C can have a value of 100, and the value of F can be based on the type of transformer used (e.g., as described above with respect to Figure 5 The factor value 509 describes an integer between 1 and 16.

[0062] In step 1006 , the processing system generates a first output word-gram sequence based at least in part on the first prompt word-gram sequence using a pre-trained generative image transformer, the first output word-gram sequence representing a second vector quantized image.

[0063] Here again, the first output word-gram sequence may represent the second vector quantized image in any suitable manner, including representing the second vector quantized image in the same manner as described above (in step 1002) with respect to how the target word-gram sequence may represent the first vector quantized image.

[0064] The pre-trained generative image transformer can be any suitable type of transformer, such as (for example, as described above with respect to Figure 3 ) autoregressive transformer or continuous diffusion model (e.g., as discussed above with respect to Figure 4 The pre-trained generative image transformer may have any suitable number of layers and parameters. For example, in some aspects of the technology, the pre-trained generative image transformer may be an autoregressive transformer or a continuous diffusion model with 24 transformer layers and 306 million parameters trained on 256×256 pixel images. Similarly, in some aspects of the technology, the pre-trained generative image transformer may be a non-autoregressive transformer or a discrete diffusion model with 24 transformer layers and 172 million parameters trained on 256×256 pixel images.

[0065] In step 1008, the processing system compares the first output word-gram sequence to the target word-gram sequence to generate a loss value for a given training example. The processing system can make this comparison and use any suitable loss function to generate the loss value in any suitable manner. For example, in some aspects of the technology, the processing system can be configured to use a binary cross entropy loss function to compare the first output word-gram sequence to the target word-gram sequence to generate the loss value. Likewise, it will be appreciated that other types of classification losses may be used instead.

[0066] In step 1010, the processing system determines whether there are additional training examples in the batch. In this aspect, the plurality of training examples may be divided into multiple batches, or maintained as a whole, in which case there will be a single "batch" containing each of the plurality of first training examples. In either case, as indicated by the "yes" arrow, if the processing system determines that there are additional training examples in the batch, it will proceed to step 1012. In step 1012, the processing system will select the next given training example from the batch and then repeat steps 1004 to 1010 for the newly selected training example. This process will then be repeated for each next given training example in the batch until the processing system determines at step 1010 that there are no additional training examples in the batch and therefore proceeds to step 1014 (as indicated by the "no" arrow).

[0067] As shown in step 1014, after the loss value has been generated for each given training example in the batch (in step 1008), the processing system modifies one or more parameters of the prompt word-unit generator based at least in part on the generated loss value. The processing system can be configured to modify one or more parameters in any suitable manner and at any suitable interval based on these generated loss values. For example, an optimization routine such as stochastic gradient descent can be applied to the generated loss values ​​to determine the parameter modifications. In some aspects of the technology, each "batch" can include a single training example, so that the processing system will perform a back-propagation step in which the processing system modifies one or more parameters of the prompt word-unit generator each time a loss value is generated. Further, in this aspect, the processing system can be configured to combine (e.g., add) the loss values ​​generated for each given training example to generate a single aggregate loss value for the given training example, and modify one or more parameters based on this aggregate loss value. Similarly, where each "batch" includes two or more training examples, the processing system can be configured to combine the generated loss values ​​into an aggregate loss value for the batch (e.g., by summing or averaging the multiple loss values), and to modify one or more parameters of the hint word generator based on this aggregate loss value.

[0068] In step 1016, the processing system determines whether there are additional batches in the plurality of training examples. In the event that the plurality of training examples have not been split, and therefore there is a single "batch" containing each of the plurality of training examples, the determination in step 1016 will automatically be "no", and the method 1000 will then end as shown in step 1020. However, in the event that the plurality of training examples have been split into two or more batches, the processing system will follow the "yes" arrow to step 1018 to select the next given training example from the plurality of training examples. This will then begin another set of passes through steps 1004 to 1010 for each training example in the next batch, and another modification of one or more parameters of the prompt word-unit generator in step 1014. This process will continue until there are no additional batches remaining, at which point the processing system will follow the "no" arrow to step 1020.

[0069] Although method 1000 is shown as ending at step 1020 once all of the plurality of training examples have been used to tune the parameters of the cue word-gram generator, it will be understood that method 1000 may be repeated any suitable number of times using the same plurality of training examples until the first output word-gram sequence predicted by each of its predicted first output word-gram sequences is sufficiently close to its corresponding target word-gram sequence in each training example. In this regard, in some aspects of the technology, the processing system may be configured to repeat method 1000 for the plurality of training examples a predetermined number of times. Further, in some aspects, the processing system may be configured to aggregate all of the loss values ​​generated during a given pass through method 1000 and determine whether to repeat method 1000 for the plurality of training examples based on this aggregated loss value. For example, in some aspects of the technology, if the aggregated loss value for the most recent pass through method 1000 is greater than some predetermined threshold, the processing system may be configured to repeat method 1000 for the plurality of training examples. Likewise, in some aspects, the processing system can be configured to use gradient descent, and thus repeat method 1000 for multiple training examples until the aggregated loss value on a given pass through method 1000 is equal to or greater than the aggregated loss value from its previous pass.

[0070] Fig.11 Depicted is an exemplary method 1100 for generating a sequence of cue word-grams using a trained cue word-gram generator and generating an output word-gram sequence based on the sequence of cue word-grams using a pre-trained generative image transformer in accordance with aspects of the present disclosure.

[0071] In this aspect, as shown in step 1102, assume that a prompt word-gram generator is trained to generate a prompt word-gram sequence for use as an input to a pre-trained generative image transformer. This can be done in any suitable manner. For example, in some aspects of the technology, a processing system (e.g., Figure 1 or Figure 2 The processing system 102) can be based on Fig.10 Method 1000 to train a prompt word generator.

[0072] Here again, the prompt word-gram generator can be any suitable type of model configured to generate a sequence of prompt word-grams based on a set of one or more identifiers, and can have any suitable number of parameters. For example, in some aspects of the technology, the prompt word-gram generator can include the above-mentioned Figure 5 The one or more multilayer perceptrons described by the prompt word unit generator 502 of , for which the number of trainable parameters is P·(F·(C+S)+D). Similarly, the prompt word unit generator can also be configured to generate a sequence of prompt word units of any suitable length and with any suitable dimension. For example, in some aspects of the technology, as described above with respect to Figure 5 As described in the final prompt word sequence 512 of the generated prompt word sequence, the generated prompt word sequence can be a sequence of length S, where each word has a dimension of D. Each of P, F, C, S, and D can be set to any suitable value. For example, in some aspects of the technology, S can have a value of 128, P can have a value of 768, D can have a value of 768, C can have a value of 100, and the value of F can be based on the type of transformer used (e.g., as described above with respect to Figure 5 The factor value 509 describes an integer between 1 and 16.

[0073] In step 1104, a processing system (e.g., Figure 1 or Figure 2 The processing system 102 uses a hint word-gram generator to generate a first hint word-gram sequence based at least in part on the first set of one or more identifiers.

[0074] Here again, the first set of one or more identifiers may include any suitable type of identifier that the prompt word-gram generator may be configured to accept. For example, in some aspects of the technology, the first set of one or more identifiers may include the identifiers described above with respect to Figure 5 In addition, in some aspects, the first set of one or more identifiers may include additional identifiers that are not related to the subject matter of the image on which the word-unit generator is trained, such as (for example, as described above with respect to Figure 5 507) and / or (e.g., as described above with respect to Figure 5 The factor value 509 describes a predetermined factor value.

[0075] In step 1106 , the processing system generates a first output word-gram sequence based at least in part on the first prompt word-gram sequence using a pre-trained generative image transformer, the first output word-gram sequence representing a first vector quantized image.

[0076] Here again, the pre-trained generative image transformer can be any suitable type of transformer, such as (for example, as described above with respect to Figure 3 ) autoregressive transformer or continuous diffusion model (e.g., as discussed above with respect to Figure 4 The pre-trained generative image transformer may have any suitable number of layers and parameters. For example, in some aspects of the technology, the pre-trained generative image transformer may be an autoregressive transformer or a continuous diffusion model with 24 transformer layers and 306 million parameters trained on 256×256 pixel images. Similarly, in some aspects of the technology, the pre-trained generative image transformer may be a non-autoregressive transformer or a discrete diffusion model with 24 transformer layers and 172 million parameters trained on 256×256 pixel images.

[0077] Furthermore, the first output word sequence may represent the first vector quantized image in any suitable manner, including as described above with respect to Figure 3 , Figure 4 and Figures 7 to 9 The first vector quantized image is represented in the same manner as described. Thus, for example, in some aspects, each word-element of the first output word-element sequence can correspond to a different pixel of the image. Likewise, in some aspects, each word-element of the first output word-element sequence can correspond to a group of pixels. For example, in some aspects, for a 256×256 pixel image, the first output word-element sequence can be a 256-element vector, where each element corresponds to a different 16×16 pixel block. Similarly, in some aspects, for a 512×512 pixel image, the first output word-element sequence can be a 256-element vector, where each element corresponds to a different 32×32 pixel block.

[0078] Fig. 12ADepicted is an exemplary method 1200-1 for generating two hint word-gram sequences using a trained hint word-gram generator, generating one or more intermediate hint word-gram sequences based on the two hint word-gram sequences generated by the word-gram generator, and sequentially feeding two of the hint word-gram sequences to a pre-trained generative image transformer to generate two consecutive output word-gram sequences in accordance with aspects of the present disclosure.

[0079] In this aspect, as shown in step 1202, assume that a prompt word-gram generator is trained to generate a prompt word-gram sequence for use as an input to a pre-trained generative image transformer. This can be done in any suitable manner. For example, in some aspects of the technology, a processing system (e.g., Figure 1 or Figure 2 The processing system 102) can be based on Fig.10 Method 1000 to train a prompt word generator.

[0080] Here again, the prompt word-gram generator can be any suitable type of model configured to generate a sequence of prompt word-grams based on a set of one or more identifiers, and can have any suitable number of parameters. For example, in some aspects of the technology, the prompt word-gram generator can include the above-mentioned Figure 5 The one or more multilayer perceptrons described by the prompt word unit generator 502 of , for which the number of trainable parameters is P·(F·(C+S)+D). Similarly, the prompt word unit generator can also be configured to generate a sequence of prompt word units of any suitable length and with any suitable dimension. For example, in some aspects of the technology, as described above with respect to Figure 5 As described in the final prompt word sequence 512 of the generated prompt word sequence, the generated prompt word sequence can be a sequence of length S, where each word has a dimension of D. Each of P, F, C, S, and D can be set to any suitable value. For example, in some aspects of the technology, S can have a value of 128, P can have a value of 768, D can have a value of 768, C can have a value of 100, and the value of F can be based on the type of transformer used (e.g., as described above with respect to Figure 5 The factor value 509 describes a value between 1 and 16.

[0081] In step 1204, a processing system (e.g., Figure 1 or Figure 2 The processing system 102 generates a first sequence of cue word-grams using a cue word-gram generator based at least in part on a first set of one or more identifiers.

[0082] Here again, the first set of one or more identifiers may include any suitable type of identifier that the prompt word-gram generator may be configured to accept. For example, in some aspects of the technology, the first set of one or more identifiers may include the identifiers described above with respect to Figure 5 In addition, in some aspects, the first set of one or more identifiers may include additional identifiers that are not related to the subject matter of the image on which the word-unit generator is trained, such as (for example, as described above with respect to Figure 5 507) and / or (e.g., as described above with respect to Figure 5 The factor value 509 describes a predetermined factor value.

[0083] In step 1206, the processing system generates a second cue word-gram sequence using a cue word-gram generator based at least in part on a second set of one or more identifiers that differs from the first set of one or more identifiers by at least one identifier.

[0084] Here again, the second set of one or more identifiers may include any suitable type of identifiers that the prompt word-gram generator may be configured to accept, including any of the options described above in step 1204 with respect to the first set of one or more identifiers. Furthermore, the second set of one or more identifiers may differ from the first set of one or more identifiers in any suitable manner. For example, the first set of one or more identifiers may include an instance identifier of a first image on which the prompt word-gram generator was trained (e.g., Figure 8 802), and the second set of one or more identifiers may include class identifiers associated with a plurality of images on which the prompt word-mem generator was trained (e.g., Figure 8 Likewise, in another example, the first set of one or more identifiers may include an instance identifier of a first image that prompted the word-unit generator to be trained on (e.g., Fig. 9 ), and the second set of one or more identifiers may include an instance identifier of a second image that prompted the word-mem generator to be trained on (e.g., Fig. 9 Instance identifier 903).

[0085] In step 1208, the processing system generates one or more intermediate cue word-gram sequences based on the first cue word-gram sequence and the second cue word-gram sequence. The processing system can be configured to use the first cue word-gram sequence and the second cue word-gram sequence in any suitable manner to generate these one or more intermediate sequences. For example, in some aspects of the technology, as described above with respect to Figure 8The intermediate hint word sequences 810 and 812 and Fig. 9 As described above, the processing system may interpolate between the first cue word-gram sequence and the second cue word-gram sequence to generate one or more intermediate sequences.

[0086] In step 1210 , the processing system generates a first output word-gram sequence based at least in part on the first prompt word-gram sequence using a pre-trained generative image transformer, the first output word-gram sequence representing a first vector quantized image.

[0087] For example, the processing system can use a pre-trained generative image transformer to Figure 8 The processing system may generate a first output word-gram sequence based at least in part on the first prompt word-gram sequence in the same manner that the first prompt word-gram sequence 804 is used to generate an output word-gram sequence representing an exemplary image at time step t=1. Similarly, the processing system may use a pre-trained generative image transformer to generate a first output word-gram sequence based at least in part on the first prompt word-gram sequence. Fig. 9 The first output word-gram sequence is generated at least in part based on the first hint word-gram sequence in the same manner that the first hint word-gram sequence 904 is used to generate an output word-gram sequence representing an exemplary image at time step t=1.

[0088] Here again, the pre-trained generative image transformer can be any suitable type of transformer, such as (for example, as described above with respect to Figure 3 ) autoregressive transformer or continuous diffusion model (e.g., as discussed above with respect to Figure 4 The pre-trained generative image transformer may have any suitable number of layers and parameters. For example, in some aspects of the technology, the pre-trained generative image transformer may be an autoregressive transformer or a continuous diffusion model with 24 transformer layers and 306 million parameters trained on 256×256 pixel images. Similarly, in some aspects of the technology, the pre-trained generative image transformer may be a non-autoregressive transformer or a discrete diffusion model with 24 transformer layers and 172 million parameters trained on 256×256 pixel images.

[0089] Furthermore, the first output word sequence may represent the first vector quantized image in any suitable manner, including as described above with respect to Figure 3 , Figure 4and Figures 7 to 9 The first vector quantized image is represented in the same manner as described. Thus, for example, in some aspects, each word-element of the first output word-element sequence can correspond to a different pixel of the image. Likewise, in some aspects, each word-element of the first output word-element sequence can correspond to a group of pixels. For example, in some aspects, for a 256×256 pixel image, the first output word-element sequence can be a 256-element vector, where each element corresponds to a different 16×16 pixel block. Similarly, in some aspects, for a 512×512 pixel image, the first output word-element sequence can be a 256-element vector, where each element corresponds to a different 32×32 pixel block.

[0090] In step 1212, the processing system generates a second output word-gram sequence using a pre-trained generative image transformer based at least in part on the first output word-gram sequence and one of the one or more intermediate prompt word-gram sequences, the second output word-gram sequence representing a second vector quantized image.

[0091] For example, the processing system can use a pre-trained generative image transformer to Figure 8 In the same manner that some or all of the output word-gram sequence at time step t=1 can be used together with the intermediate prompt word-gram sequence 810 to generate an output word-gram sequence representing a second exemplary image at time step t=2, the second output word-gram sequence is generated based at least in part on the first output word-gram sequence and one of the intermediate prompt word-gram sequences. Similarly, the processing system can use a pre-trained generative image transformer to generate a second output word-gram sequence based at least in part on the first output word-gram sequence and one of the intermediate prompt word-gram sequences. Fig. 9 In the same manner as some or all of the output word-gram sequence at time step t=1 can be used together with the intermediate prompt word-gram sequence 910 to generate an output word-gram sequence representing a second exemplary image at time step t=2, a second output word-gram sequence is generated based at least in part on the first output word-gram sequence and one of the intermediate prompt word-gram sequences. In all cases, the pre-trained generative image transformer can generate the second output word-gram sequence based on the entire first output word-gram sequence, or based on a portion of the first output word-gram sequence. For example, in some aspects of the technology, as described above with respect to Figure 4 and Figure 7 As described, a pre-trained generative image transformer can generate a second output word-gram sequence based on a masked version of the first output word-gram sequence.

[0092] Here again, the second output word-gram sequence may represent the second vector quantized image in any suitable manner, including any of the options described above in step 1210 with respect to how the first vector quantized image may represent the first vector quantized image.

[0093] Fig. 12B Describes various aspects of the present disclosure. Fig. 12A The exemplary method 1200-1 constructed by the exemplary method 1200-2 is used to sequentially feed another two prompt word-gram sequences in the generated prompt word-gram sequence to a pre-trained generative image transformer to generate two additional consecutive output word-gram sequences.

[0094] In this regard, as shown in step 1214, it is assumed that the above-described manner is performed. Fig. 12A Each of steps 1202 to 1212 of method 1200-1.

[0095] Then, in step 1216, the processing system uses a pre-trained generative image transformer based at least in part on ( Fig. 12A The pre-trained generative image transformer may generate a third output word sequence based on a selected one of the one or more intermediate prompt word sequences (generated in step 1208 of the present invention), the third output word sequence representing a third vector quantized image. As will be appreciated, the pre-trained generative image transformer may generate the third output word sequence based on more than just the selected intermediate prompt word sequence. Thus, for example, the pre-trained generative image transformer may generate the third output word sequence based on a selected one of the one or more intermediate prompt word sequences and some or all of the output word sequences from the previous time step (e.g., as described above with respect to Figure 4 and Figure 7 described, a masked version of the second output word-gram sequence) to generate a third output word-gram sequence.

[0096] For example, the processing system can use a pre-trained generative image transformer to Figure 8 In the same manner that one of the intermediate hint word-gram sequences 810 or 812 can be used (along with a masked version of the output word-gram sequence from the previous time step) to generate an output word-gram sequence representing the exemplary image at time step t=2 or t=3, the processing system can generate a third output word-gram sequence based at least in part on one of the intermediate hint word-gram sequences. Similarly, the processing system can use a pre-trained generative image transformer to generate a third output word-gram sequence based at least in part on one of the intermediate hint word-gram sequences. Fig. 9In the same manner as one of the intermediate hint word-gram sequences 910, 912, 914, or 916 can be used to generate an output word-gram sequence representing an exemplary image at time step t = 2, t = 3, t = 4, or t = 5 (together with a masked version of the output word-gram sequence from the prior time step), a third output word-gram sequence is generated based at least in part on one of the intermediate hint word-gram sequences.

[0097] Here again, the third output word-gram sequence may represent the third vector quantized image in any suitable manner, including any of the options described above in step 1210 with respect to how the first vector quantized image represents the first vector quantized image.

[0098] At step 1218, the processing system generates a fourth output word-gram sequence based at least in part on the third output word-gram sequence and the second hint word-gram sequence using a pre-trained generative image transformer, the fourth output word-gram sequence representing a fourth vector quantized image.

[0099] For example, the processing system can use a pre-trained generative image transformer to Figure 8 In the same manner that some or all of the output word-gram sequence at time step t=3 can be used together with the second prompt word-gram sequence 806 to generate an output word-gram sequence representing the exemplary image at time step t=4, a fourth output word-gram sequence can be generated based at least in part on the third output word-gram sequence and the second prompt word-gram sequence. Similarly, the processing system can use a pre-trained generative image transformer to generate a fourth output word-gram sequence based at least in part on the third output word-gram sequence and the second prompt word-gram sequence. Fig. 9 In the same manner as the output word-gram sequence at time step t = 5 of , some or all of the output word-gram sequence can be used together with the second prompt word-gram sequence 906 to generate an output word-gram sequence representing the exemplary image at time step t = 6, to generate a fourth output word-gram sequence based at least in part on the third output word-gram sequence and the second prompt word-gram sequence. Here again, in all cases, the pre-trained generative image transformer can generate the fourth output word-gram sequence based on the entire third output word-gram sequence or based on a portion of the third output word-gram sequence. For example, in some aspects of the technology, as described above with respect to Figure 4 and Figure 7 As described, a pre-trained generative image transformer can generate a fourth output word-unit sequence based on a masked version of the third output word-unit sequence.

[0100] Here again, the fourth output word-gram sequence may represent the fourth vector quantized image in any suitable manner, including any of the options described above in step 1210 with respect to how the first vector quantized image represents the first vector quantized image.

[0101] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the exemplary systems and methods should be made by way of illustration rather than by way of limitation of the subject matter defined by the claims. In addition, the examples described herein and the provision of clauses expressed as "such as," "including," "comprising," etc. should not be interpreted as limiting the subject matter of the claims to specific examples; rather, these examples are intended to illustrate only some of the many possible embodiments. Further, the same reference numerals in different figures may identify the same or similar elements.

Claims

1. A computer-implemented method comprising: For each given training example in a plurality of training examples, the given training example comprises a target word-gram sequence representing a first vector quantized image and a first set of one or more identifiers, at least one identifier in the first set of one or more identifiers being related to a subject of the first vector quantized image: generating, using a cue word-gram generator, a first cue word-gram sequence based at least in part on the first set of one or more identifiers; generating a first output word-gram sequence based at least in part on the first prompt word-gram sequence using a pre-trained generative image transformer, the first output word-gram sequence representing a second vector quantized image; as well as using one or more processors of a processing system to compare the first output word-gram sequence with the target word-gram sequence to generate a loss value for the given training example; as well as One or more parameters of the hint word-gram generator are modified using the one or more processors based at least in part on the loss values ​​generated for the plurality of training examples.

2. The method of claim 1, wherein: The hint word unit generator includes two or more multi-layer perceptrons, and Wherein modifying the one or more parameters of the hint word-gram generator comprises modifying one or more parameters of each of the two or more multi-layer perceptrons.

3. The method according to claim 1 or claim 2, wherein: The first set of one or more identifiers for the given training example includes a class identifier associated with the subject matter of the first vector quantized image.

4. The method according to any one of claims 1 to 3, wherein: The first set of one or more identifiers for the given training example includes instance identifiers associated with the first vector quantized image.

5. The method according to any one of claims 1 to 4, further comprising: generating, using the cue word-gram generator, a second cue word-gram sequence based at least in part on a second set of one or more identifiers; as well as A second output word-gram sequence is generated based at least in part on the second prompt word-gram sequence using the pre-trained generative image transformer, the second output word-gram sequence representing a third vector quantized image.

6. The method according to any one of claims 1 to 4, further comprising: generating, using the cue word-gram generator, a second cue word-gram sequence based at least in part on a second set of one or more identifiers; generating, using the cue word-gram generator, a third cue word-gram sequence based at least in part on a third set of one or more identifiers, the third set of one or more identifiers being different from the second set of one or more identifiers due to at least one identifier; generating, using the one or more processors, one or more intermediate cue word-gram sequences based on the second cue word-gram sequence and the third cue word-gram sequence; generating a second output word-gram sequence based at least in part on the second prompt word-gram sequence using the pre-trained generative image transformer, the second output word-gram sequence representing a third vector quantized image; and Generating a third output word-gram sequence based at least in part on the second output word-gram sequence and one of the one or more intermediate prompt word-gram sequences using the pre-trained generative image transformer, the third output word-gram sequence representing a fourth vector quantized image.

7. The method of claim 6, further comprising: generating, using the pre-trained generative image transformer, a fourth output word-gram sequence based at least in part on one of the one or more intermediate cue word-gram sequences, the fourth output word-gram sequence representing a fifth vector quantized image; and A fifth output word-gram sequence is generated using the pre-trained generative image transformer based at least in part on the fourth output word-gram sequence and the third hint word-gram sequence, the fifth output word-gram sequence representing a sixth vector quantized image.

8. The method of claim 7, further comprising: An output image is generated based on the fifth output word-gram sequence.

9. A processing system comprising: A memory storing a pre-trained generative image transformer and a prompt word unit generator; as well as One or more processors, the one or more processors being coupled to the memory and configured to train the prompt word-gram generator according to a training method, the training method comprising: For each given training example in a plurality of training examples, the given training example comprises a target word-gram sequence representing a first vector quantized image and a first set of one or more identifiers, at least one identifier in the first set of one or more identifiers being related to a subject of the first vector quantized image: generating, using the cue word-gram generator, a first cue word-gram sequence based at least in part on the first set of one or more identifiers; generating a first output word-gram sequence based at least in part on the first prompt word-gram sequence using the pre-trained generative image transformer, the first output word-gram sequence representing a second vector quantized image; and comparing the first output word-gram sequence with the target word-gram sequence to generate a loss value for the given training example; and One or more parameters of the hint word-gram generator are modified based at least in part on the loss values ​​generated for the plurality of training examples.

10. The system of claim 9, wherein: The prompt word unit generator includes a multi-layer perceptron.

11. The system of claim 9, wherein: The cue word-gram generator includes two or more multi-layer perceptrons.

12. The system of claim 11, wherein: The one or more processors are configured to modify the one or more parameters of the cue word-gram generator, including modifying one or more parameters of each of the two or more multi-layer perceptrons.

13. The system of any one of claims 9 to 12, wherein: The one or more processors are further configured to: generating, using the cue word-gram generator, a second cue word-gram sequence based at least in part on a second set of one or more identifiers; and A second output word-gram sequence is generated based at least in part on the second prompt word-gram sequence using the pre-trained generative image transformer, the second output word-gram sequence representing a third vector quantized image.

14. The system of any one of claims 9 to 12, wherein: The one or more processors are further configured to: generating, using the cue word-gram generator, a second cue word-gram sequence based at least in part on a second set of one or more identifiers; generating, using the cue word-gram generator, a third cue word-gram sequence based at least in part on a third set of one or more identifiers, the third set of one or more identifiers being different from the second set of one or more identifiers due to at least one identifier; generating one or more intermediate cue word-gram sequences based on the second cue word-gram sequence and the third cue word-gram sequence; generating a second output word-gram sequence based at least in part on the second prompt word-gram sequence using the pre-trained generative image transformer, the second output word-gram sequence representing a third vector quantized image; and Generating a third output word-gram sequence based at least in part on the second output word-gram sequence and one of the one or more intermediate prompt word-gram sequences using the pre-trained generative image transformer, the third output word-gram sequence representing a fourth vector quantized image.

15. The system of claim 14, wherein: The one or more processors are further configured to: generating, using the pre-trained generative image transformer, a fourth output word-gram sequence based at least in part on one of the one or more intermediate cue word-gram sequences, the fourth output word-gram sequence representing a fifth vector quantized image; and A fifth output word-gram sequence is generated using the pre-trained generative image transformer based at least in part on the fourth output word-gram sequence and the third hint word-gram sequence, the fifth output word-gram sequence representing a sixth vector quantized image.

16. The system of claim 15, wherein: The one or more processors are further configured to: An output image is generated based on the fifth output word-gram sequence.

17. The system of claim 16, wherein: The one or more processors are configured to generate the output image using a decoder of the pre-trained generative image transformer.

18. The system of any one of claims 9 to 13, wherein: The pre-trained generative image transformer is an autoregressive image transformer.

19. The system of any one of claims 9 to 17, wherein: The pre-trained generative image transformer is a non-autoregressive image transformer.

20. A non-transitory computer program product comprising computer readable instructions which, when executed by a processing system, cause the processing system to perform the method of any one of claims 1 to 8.

Images