System and method for free localization of text-to-image generation via cross attention

Abstract

Systems and methods for free localization of text-to-image generation via cross attention. A method discloses receiving, at a cross-attention layer of a model, first text data describing a first object and second text data describing a first scene, where the first text data includes a location description of the first object, connecting the first text data and the second text data to generate a prompt using the model having the cross-attention layer; generating a broadcasted location mask constructed from at least the location; generating a broadcasted full 1 matrix associated with second textual data describing the first scene; calculating a key matrix and a value matrix using the prompted individual linear projection; calculating a query matrix by utilizing linear projection; generating a broadcasted location matrix in response to connecting the broadcasted location mask and the broadcasted full 1 matrix; generating a cross attention map using the query matrix, the key matrix, and the broadcast location matrix; and outputting a final image.

Description

Technical Field

[0001] The present disclosure relates to image generation utilizing machine learning networks, such as text-to-image generation systems. Background Art

[0002] In recent years, text-to-image generation models have demonstrated strong performance. Models like StableDiffusion and Dall-E are able to generate high-quality and diverse images from arbitrary text prompts. However, a significant challenge faced by these models is that they rely solely on text prompts alone to exert content control over the generation process, which may not be sufficient for many applications. Specifically, one of the most intuitive and user-friendly ways to exert control over the generation is to provide localization information that guides the model where to generate specific elements within the image. Unfortunately, current pre-trained models face limitations in their ability to perform localized generation. These limitations stem not only from their inability to incorporate positional information as input, but also from the inherent difficulties associated with synthesizability - a known challenge for many multimodal base models.

[0003] Existing approaches to this problem generally fall into three main categories: training completely new models, fine-tuning existing models with additional components (such as task-specific encoders), or strategically combining multiple samples into one. These approaches may often require large amounts of training data, resources, and / or extended inference time, making them impractical for real-life applications due to their time- and resource-intensive nature. On the other hand, prompt-to-prompt image editing can edit generated images based on modified text prompts by manipulating the criss-cross attention maps in text-to-image generation models. Notably, this work also shows that the criss-cross attention layer plays a key role in controlling the spatial layout of generated objects associated with specific phrases in the prompt. Summary of the invention

[0004] According to a first embodiment, a computer-implemented method for generating synthetic image data discloses receiving first text data describing a first object and second text data describing a first scene at a cross-attention layer of a pretrained model that includes an input to the pretrained model, wherein the first text data includes a location description of the first object. The method also includes utilizing a pre-trained model having a cross-attention layer, connecting first text data and second text data to generate a single prompt describing the first text data and the second text data, utilizing the first text data to generate a broadcasted position mask, the position mask consisting of at least the position of the first object indicating pixels associated with the first object, utilizing the second text data to generate a broadcasted all-ones matrix associated with the second text data describing the first scene, wherein the broadcasted all-ones matrix is ​​a mask including all pixels associated with the first scene, utilizing separate linear projections of the single prompt to compute a key matrix and a value matrix, utilizing a linear projection of (i) the output of a previous layer or (ii) an input to the pre-trained model to compute a query matrix, generating a broadcasted position matrix in response to connecting the broadcasted position mask and the broadcasted all-ones matrix, generating a cross-attention map using the query matrix, the key matrix, and the broadcasted position matrix, and outputting a final image in response to performing a fixed number of iterations corresponding to the cross-attention layer of the pre-trained model.

[0005] According to a second embodiment, a computer-implemented method discloses receiving first text data describing a first object and second text data describing a first scene at a cross-attention layer of a model, wherein the first text data includes a position description of the first object; using a model with a cross-attention layer, connecting the first text data and the second text data to generate a prompt describing the first text data and the second text data; using the first text data to generate a broadcasted position mask, which is at least composed of the position of the first object indicating pixels associated with the first object; using the second text data to generate a broadcasted all-1 matrix associated with the second text data describing the first scene, wherein the broadcasted all-1 matrix is ​​a mask including all pixels associated with the first scene; using separate linear projections of the prompts to calculate a key matrix and a value matrix; using (i) the output of a previous layer or (ii) a linear projection to the input of a pre-trained model to calculate a query matrix; generating a broadcasted position matrix in response to connecting the broadcasted position mask and the broadcasted all-1 matrix; generating a cross-attention map using the query matrix, the key matrix, and the broadcasted position matrix; and in response to performing a fixed number of iterations, outputting a final image from the final layer of the model.

[0006] According to a third embodiment, a system includes an input interface configured to receive a query; and a processor in communication with the input interface, the processor being programmed to: receive first text data describing a first object and second text data describing a first scene at a cross-attention layer of a pretrained model including an input to the pretrained model, wherein the first text data includes a location description of the first object, and using the pretrained model with the cross-attention layer, connect the first text data and the second text data to generate a single prompt describing the first text data and the second text data; and generate a broadcasted location mask using the first text data, the location mask consisting of at least locations of the first object indicating pixels associated with the first object. The invention relates to a method for generating a broadcasted all-one matrix associated with the second text data describing the first scene using the second text data, wherein the broadcasted all-one matrix is ​​a mask including all pixels associated with the first scene; determining a key matrix and a value matrix using separate linear projections of a single hint; determining a query matrix using a linear projection of (i) the output of a previous layer or (ii) the input to a pre-trained model; generating a broadcasted position matrix in response to concatenating the broadcasted position mask and the broadcasted all-one matrix; generating a cross-attention map using the query matrix, the key matrix, and the broadcasted position matrix; and outputting a final image in response to performing a fixed number of iterations corresponding to the cross-attention layer of the pre-trained model. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 Illustrated are embodiments of block diagrams showing example architectures of computing devices, such as camera devices, processors, mobile devices, and other computing devices described herein.

[0008] Figure 2 An embodiment of a process utilizing text-to-image generation with crossed attention control (CAC) is illustrated.

[0009] Figure 3 An example of localized text-to-image generation utilizing the illustrative embodiments is illustrated. DETAILED DESCRIPTION

[0010] In one embodiment, the system and method propose the use of cross-attention control (CAC) to provide a pre-trained text-to-image model with better open vocabulary localization capabilities. Given a caption and localization information, such as bounding boxes and semantic segmentation maps, along with their corresponding text descriptions, the system can first construct a new text input by connecting the caption and all prompts associated with the location information. The system can then calculate a cross-attention map based on this new text prompt, and apply localization constraints to the cross-attention map based on the localization information. Such an approach does not require any additional training or model architecture modifications, such as designing task-specific encoders. It also does not impose any language restrictions, such as using a fixed vocabulary set or language parser. In addition, it is highly portable and can be easily integrated into a single forward pass in any cross-attention-based text-to-image generation framework with only a few lines of code, so no additional inference time is required.

[0011] Figure 1 An embodiment of a block diagram of an example architecture of a computing device is illustrated, such as a camera device, a processor, a mobile device, and other computing devices described herein. It will be appreciated that not all user devices will include all components of the architecture 100, and some user devices may include additional components not shown in the architecture 100. The architecture 100 may include one or more processing elements 104 for executing instructions and retrieving data stored in a non-transitory computer-readable storage element 102. The processing element 104 may include at least one processor. Any suitable one or more processors may be used. For example, the processing element 104 may include one or more digital signal processors (DSPs) and / or image signal processors (ISPs). In some examples, the processing element 104 may effectively perform one or more functions of a generator and / or discriminator, as described above. The storage element 102 may include one or more different types of non-transitory computer-readable memories, data storage devices, or computer-readable storage media dedicated to different purposes within the architecture 100. For example, the storage element 102 may include flash memory, random access memory, disk-based storage devices, etc. For example, different portions of memory element 102 may be used for program instructions executed by processing element 104, storage of images or other digital works, and / or removable storage for transferring data to other devices, etc.

[0012] The storage element 102 may also store software executed by the processing element 104. The operating system 122 may provide an interface to a user for operating a user device, and may facilitate communications and commands between applications executed on the architecture 100 and its various hardware. The transmission application 124 may be configured to receive images and / or video from another device (e.g., a mobile device, an image capture device, and / or a display device) or from an image sensor 132 included in the architecture 100. In some examples, the transmission application 124 may also be configured to upload the received image to another device (e.g., a mobile device and / or another computing device) that may perform the processing described herein.

[0013] When implemented in some user devices, the architecture 100 may also include a display component 106. The display component 106 may include one or more light emitting diodes (LEDs) or other suitable display lights. In addition, in some examples, the display component 106 may include, for example, one or more devices such as a cathode ray tube (CRT), a liquid crystal display (LCD) screen, a gas plasma-based flat panel display, an LCD projector, a raster projector, an infrared projector, or other types of display devices.

[0014] The architecture 100 may also include one or more input devices 108, which are operable to receive input from a user. The input device 108 may include, for example, a button, a touchpad, a touch screen, a scroll wheel, a joystick, a keyboard, a mouse, a trackball, a keypad, a light gun, a game controller, or any other such device or element, whereby a user can provide input to the architecture 100. These input devices 108 may be incorporated into the architecture 100, or may be operably coupled to the architecture 100 via a wired or wireless interface. In some examples, the architecture 100 may include a microphone 170 for capturing sounds (such as voice commands). The speech recognition engine 180 may interpret the audio signal of the sound captured by the microphone 170. In some examples, the speech recognition engine 180 may monitor the "wake-up word" to be received by the microphone 170. Upon receiving the wake-up word, the speech recognition engine 180 may stream the audio to a speech recognition server for analysis. In various examples, the speech recognition engine 180 may stream the audio to an external computing device via the communication interface 112.

[0015] When the display assembly 106 includes a touch-sensitive display, the input device 108 may include a touch sensor operating in conjunction with the display assembly 106 to permit a user to interact with images displayed by the display assembly 106 using touch input (e.g., with a finger or a stylus). The architecture 100 may also include a power source 114, such as a wired alternating current (AC) converter, a rechargeable battery operable to be charged by conventional plug-in methods, or by other methods such as capacitive or inductive charging.

[0016] The communication interface 112 may include one or more wired or wireless components operable to communicate with one or more other user devices. For example, the communication interface 112 may include a wireless communication module 136 configured to communicate on a network according to any suitable wireless protocol, such as IEEE 802.11 or another suitable wireless local area network (WLAN) protocol. The short-range interface 134 may be configured to communicate using one or more short-range wireless protocols such as, for example, near field communication (NFC), Bluetooth, Bluetooth LE, etc. The mobile interface 140 may be configured to communicate using a cellular or other mobile protocol. The global positioning system (GPS) interface 138 may communicate with one or more earth orbiting satellites or other suitable location determination systems to identify the location of the architecture 100. The wired communication module 142 may be configured to communicate according to a USB protocol or any other suitable protocol. In various examples, the mobile interface 140 may allow (one or more) computing devices to communicate with one or more other computing devices that may be used to perform various aspects of the technology described herein.

[0017] The architecture 100 may also include one or more sensors 130, such as, for example, one or more position sensors, image sensors, and / or motion sensors. An image sensor 132 may be shown. Some examples of the architecture 100 may include multiple image sensors 132. For example, a panoramic camera system may include multiple image sensors 132, thereby generating multiple images and / or video frames that may be stitched and may be blended to form a seamless panoramic output. Examples of image sensors 132 may be cameras or other image sensors. In various examples, the image sensor 132 may be configured to capture color information, IR image data, image geometry information, and / or ambient light information.

[0018] The motion sensor may include any sensor that senses the motion of the architecture, including, for example, a gyroscope sensor and an accelerometer. In some examples, the motion sensor may be used to determine the orientation of the camera, such as a pitch angle and / or a roll angle. The gyroscope sensor may be configured to generate a signal indicating the rotational motion and / or orientation change of the architecture (e.g., the magnitude and / or direction of the motion or orientation change). Any suitable gyroscope sensor may be used, including, for example, a ring laser gyroscope, a fiber optic gyroscope, a fluid gyroscope, a vibration gyroscope, etc. In some examples, an accelerometer may generate a signal indicating acceleration (e.g., the magnitude and / or direction of acceleration). Any suitable accelerometer may be used, including, for example, a piezoresistive accelerometer, a capacitive accelerometer, etc. In some examples, the GPS interface 138 may be used as a motion sensor. For example, a change in the position of the architecture 100 determined by the GPS interface 138 may indicate the motion of the GPS interface 138. As described, in some examples, the image sensor 132 may effectively detect infrared light. In at least some examples, the architecture 100 may include an infrared light source to illuminate the surrounding environment.

[0019] Figure 2 An illustration of a process for text-to-image generation utilizing cross-attention control (CAC) is disclosed. The system can utilize a standardized set of evaluation metrics for localized text-to-image generation tasks using off-the-shelf large-scale pre-trained recognition models. The system can apply CAC to a variety of state-of-the-art baseline text-to-image generation models and experiment with different forms of localization information including bounding boxes and semantic segmentation maps. The system and method can demonstrate that CAC imparts new localized generation capabilities to pre-trained standard text-to-image models, and further improves models trained specifically for localized generation. In addition, embodiments can show that the method can significantly improve the synthesizability capabilities of text-to-image generation models using simple heuristics to spatially separate components within a text prompt.

[0020] The goal of this work is to perform localized text-to-image generation given a pre-trained text-to-image generation model. The localization information provided by the user should consist of text phrases describing the content in the image space and the constrained spatial locations associated with these contents. Common location information includes bounding boxes and semantic segmentation maps. Furthermore, the system can be designed to perform this task with: (1) no additional training or fine-tuning, (2) no model architecture modifications, and (3) no additional inference time, and (4) no further restrictions on the input text space from the original model. The provided pre-trained model can be trained with localization information, or trained only with text-image pairs.

[0021] Formally, given a pre-trained text-to-image generation model p θ, length η0, text prompt y0∈Y n0 and a set of localized information One goal could be to generate an image that is visually consistent with the overall text description provided in y0 and the localized description provided in g Here γ represents the vocabulary space of textual cues, C, H, W are the dimensions of the output image, and for each g i =(y i , b i )∈Y ni *[0, 1] H*W , where y i is the text description of the ith localized region, and b i is the spatial constraint mask corresponding to the description. θ From p θ (x|y0) or p θ (x|y0, g). The system can assume that the pre-trained model uses a cross-attention mechanism for text-image conditioning.

[0022] State-of-the-art text-to-image generation models can leverage criss-cross attention mechanisms to achieve their success. In one embodiment, the system can select stable diffusion as the backbone model. However, alternative embodiments can allow for a system and method that can also be applied to other criss-cross attention based diffusion models, such as Imagen and GANs, such as GigaGAN.

[0023] For x∈p θ (x\y0)(where x∈X, y0∈Y n ) sampling task, the cross-attention layer l in pθ receives the encoded textual prompt and intermediate samples that have been processed by previous layers in the network and previous diffusion time steps As its input. For example, n0, d e is the text length and text embedding dimension, and C (l) , H (l) , W (l) represents the perceptual dimensions of layer l, which can be different from C′, H′, W′ due to the U-Net structure. The system can then (＜1) Projection to query matrix 211 h is the number of heads of multi-head attention, d is the feature projection dimension of query 211 and key 207, and d v is the feature projection dimension of the value. The cross attention map at layer l is then calculated as:

[0024]

[0025] is the output of l, where l O is another linear projection.

[0026] When generating the jth pixel block in the image, the system can Each entry in It is interpreted as the degree of attention that the rth head gives to the kth token in y0. The layer output is a weighted average of the value features, where the weights are assigned by the attention maps from all heads.

[0027] In one example, a cue 201 may include a text description ("dog") and have an indicator of a location 202 associated with the first cue applied to a matrix or graph. A second cue 203 may include a description "cat" and show a location 204 associated with the first cue as belonging to the object. Another cue 205 may indicate a scene.

[0028] Each localized information pair g i =(y i , b i ) indicates that the model should generate textual hints at pixel locations y i To describe the content, where b i > 0. Therefore, based on the previous explanation and findings, if the spatial constraint mask b i The jth entry b in i,(j) is positive, then the (r, j, k)th element in the attention map should only be from y i The kth token in receives attention.

[0029] As a result, the system can first interpolate the original position mask b i , to obtain The dimension of the receptive field of its matching layer l. Let Mark from The flattened and broadcasted position mask 209 is constructed, and Mark from y i The calculated key matrix 207, which can expand equation 1 to have:

[0030]

[0031] In one example, the system can then combine the m+1 attention maps In one example, attention maps 215a, 215b, 215c may be generated. In one embodiment, the system may calculate an average map However, it is not clear what the “average” value matrix 217 corresponding to this attention map is. Another attempt is to separately compute the matrix in And then calculate the average output matrix or as the output of the layer. While it works well for their standard text-to-image generation task, in our setting the very sparse attention map represented by the localized information associated with small objects may lead to unexpected performance.

[0032] The system can assume that y i is a substring of y0 (for all i=1,…,m): For example, if the user wants to generate “a photo of a restaurant with cups on the table” and provides bounding boxes for “cup” and “table”, the system can use the position information to directly mask the portion of the attention map of the caption associated with the tokens for “cup” and “table” (i.e. ). Formally, assuming y i The j i The token is associated with the (jth i +n i ) tokens, then the system can directly calculate:

[0033]

[0034] in is a mask, where Broadcast only to the jth i to (j i +n i ) submatrices while keeping the remaining elements all zero. The system can then calculate

[0035] However, this assumption may not always hold. For example, a user could request a “photo of a restaurant” without describing all the details of the scene, but they could still specify the locations of the “cup” and “table” with bounding boxes without mentioning them in the caption. Therefore, in order to apply the method to all inputs without this assumption, the system can construct a new text prompt by concatenating all input prompts:

[0036]

[0037] in Mark concatenation. In one embodiment, the system and method can prevent all special token encodings and fill the resulting prompt after concatenation. Similar to the text prompt, the system can also concatenate all masks to create:

[0038]

[0039] In practice, the system and method can use the all-one matrix 213 as a position mask for the caption y0 and special tokens. Similar to the prompt-to-prompt editing, the system can also use a separate set of weights Applied to the attention map to adjust the influence each token has on the resulting generation. The system can calculate the aggregate attention map as

[0040] Finally, the output of layer 219 can be calculated as z (l) = l o (M (l) V (l) ), and the framework is Figure 2 The system and method can only change the forward pass of the pre-trained model at sampling time, and therefore does not require any further training or model architecture modification, and it does not require any other language restrictions or priors, such as a fixed vocabulary set or language parser. It also encapsulates well in the original optimized transformer framework, and therefore does not require additional inference time. Due to the minimal assumptions, our method is a cost-free open-vocabulary plug-in for all text-to-image generation models that use criss-cross attention for text guidance.

[0041] In addition to criss-cross attention, self-attention layers are also essential for many text-to-image generation models to produce coherent spatial layouts. For example, applying self-attention control to a small portion of the diffusion process in addition to criss-cross attention control can further help provide consistent geometry and color palette. While self-attention control is trivial in editing, it becomes complicated in our setting because the position information of different localized cues may overlap with each other, resulting in conflicting signals and ambiguous masks at overlapping pixels.

[0042] One way to incorporate self-attention control is to optimize each region separately based on different localization cues before tying all regions together. The solution to this can be roughly simplified to multi-diffusion when both self-attention control and cross-attention control are applied to all diffusion steps. As a result, the system can first apply multi-diffusion to a small part of the diffusion process, and then perform cross-attention controlled diffusion as described above for the remaining diffusion time steps to achieve the desired effect. The system can also use models like GLIGEN, which are fine-tuned based on localization information to provide learned self-attention control.

[0043] In one illustrative embodiment and method, the system can be considered as a plug-in for MultiDiffusion and GLIGEN to provide better localization capabilities, and it can do so without adding additional cost to both algorithms.

[0044] Figure 3 An example of cross-attention control as a plug-in for existing methods for localized text-to-image generation is shown. Cross-attention control can improve a variety of types of localization (bounding boxes, semantic segmentation maps, and localized styles) using different base models (e.g., stable diffusion and GLIGEN). Figure 3 As shown in , a bounding box can be used as an input. The bounding box can reflect objects like "astronaut on horse" and "pink flag". In one embodiment, the caption can indicate "Mars landscape". The localized hint can indicate "astronaut on horse" and "pink flag" and correspond to the input. The bounding box can include the scene and objects associated with the scene. As shown in the CAC example, the placement of "astronaut on horse" is more accurate than the baseline model. Additionally, the flag can have an appropriate shape and position.

[0045] In another example, the input can utilize a semantic segmentation map. The semantic segmentation map can map out the object or enhance the image accordingly. In one example, the semantic segmentation map can correspond to a cat with one blue eye and one green eye. The caption can be "cute cat". The localized cues can be "blue eyes" and "green eyes". The cross-attention control model can output an image that is more accurate and higher quality than other versions. Of course, Figure 3 Other examples are shown in .

[0046] Embodiments of the present disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The figures are not necessarily to scale; some features may be magnified or minimized to show the details of a particular component. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a representative basis for teaching those skilled in the art to adopt the embodiments in various ways. As will be understood by those of ordinary skill in the art, the various features illustrated and described with reference to any of the figures may be combined with the features illustrated in one or more of the other figures to produce embodiments that are not explicitly illustrated or described. The combination of the illustrated features provides a representative embodiment of a typical application. However, for a particular application or implementation, various combinations and modifications of features consistent with the teachings of the present disclosure may be desired.

[0047] As used herein, "a", "an", and "the" refer to both the singular and the plural, unless the context clearly indicates otherwise. By way of example, a "processor" programmed to perform various functions refers to one processor programmed to perform each function, or more than one processor programmed together to perform each of the various functions.

[0048] Although exemplary embodiments are described above, it is not intended that these embodiments describe all possible forms covered by the claims. The words used in the specification are descriptive rather than restrictive words, and it should be understood that various changes can be made without departing from the spirit and scope of the present disclosure. As previously described, the features of the various embodiments can be combined to form other embodiments of the present invention that may not be explicitly described or illustrated. Although various embodiments may have been described as providing advantages or preferred over other embodiments or prior art implementations in terms of one or more desired characteristics, it is recognized by those of ordinary skill in the art that one or more features or characteristics can be compromised to achieve the desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, applicability, weight, manufacturability, ease of assembly, etc. As such, to the extent that any embodiment is described as less desirable than other embodiments or prior art implementations in terms of one or more characteristics, these embodiments are not outside the scope of the present disclosure and may be desirable for a particular application.

Claims

1. A computer-implemented method for generating composite image data, comprising: receiving, at a criss-cross attention layer of a pre-trained model comprising an input to the pre-trained model, first text data describing a first object and second text data describing a first scene, wherein the first text data comprises a location description of the first object; Concatenating the first text data and the second text data to generate a single prompt describing the first text data and the second text data using a pre-trained model with a crossed attention layer; generating a broadcasted position mask using the first text data, the position mask consisting of at least positions of the first object indicating pixels associated with the first object; generating a broadcasted all-ones matrix associated with the second text data describing the first scene using the second text data, wherein the broadcasted all-ones matrix is ​​a mask including all pixels associated with the first scene; The key and value matrices are computed using separate linear projections of individual cues; Compute the query matrix using (i) the output of the previous layer or (ii) the linear projection of the input to the pre-trained model; generating a broadcasted position matrix in response to concatenating the broadcasted position mask and the broadcasted all-ones matrix; generating a cross-attention map using the query matrix, the key matrix, and the broadcast position matrix; as well as In response to performing a fixed number of iterations corresponding to the criss-cross attention layers of the pre-trained model, a final image is output.

2. The method according to claim 1, wherein: The cross-attention map is generated using at least a softmax function.

3. The method according to claim 1, wherein: The criss-cross attention map is generated using element-wise multiplication with the broadcast position matrix and matrix multiplication of the query matrix and the key matrix.

4. The method according to claim 1, wherein: The pre-trained model at least includes a stable diffusion model.

5. The method according to claim 1, wherein: The pre-trained model includes one or more layers of criss-cross attention layers, where a first layer is configured to generate an output that is sent to a second layer.

6. The method according to claim 1, wherein: The second iteration is configured to generate output that is sent to the third iteration.

7. The method according to claim 1, wherein: The criss-cross attention layer includes a final layer that outputs a final image associated with the first text data and the second text data.

8. The method according to claim 1, wherein: The pre-trained model is configured to adjust the influence of each token in response to a weight associated with the cross-attention map.

9. The method according to claim 1, wherein: The method includes receiving third text data describing a second object.

10. The method according to claim 9, wherein: The third text data describes a second position associated with the second object.

11. A computer-implemented method of generating composite image data, comprising: Receiving, at a cross-attention layer of the model, first text data describing a first object and second text data describing a first scene, wherein the first text data includes a location description of the first object; Concatenating the first text data and the second text data to generate a prompt describing the first text data and the second text data using a model having a crossed attention layer; generating a broadcasted position mask using the first text data, the position mask consisting of at least positions of the first object indicating pixels associated with the first object; generating a broadcasted all-ones matrix associated with the second text data describing the first scene using the second text data, wherein the broadcasted all-ones matrix is ​​a mask including all pixels associated with the first scene; Compute the key and value matrices using separate linear projections of the prompts; Compute the query matrix using (i) the output of the previous layer or (ii) the linear projection of the input to the pre-trained model; generating a broadcasted position matrix in response to concatenating the broadcasted position mask and the broadcasted all-ones matrix; generating a cross-attention map using the query matrix, the key matrix, and the broadcast position matrix; as well as In response to performing a fixed number of iterations, a final image is output from the final layer of the model.

12. The method according to claim 11, wherein: The fixed number of iterations corresponds to the criss-cross attention layer.

13. The method according to claim 11, wherein: The model includes an input image model.

14. The method according to claim 11, wherein: The model is configured to adjust the influence of each token in response to a weight associated with the cross-attention map.

15. The method according to claim 11, wherein: The final image is associated with the first text data and the second text data.

16. The method according to claim 11, wherein: The method includes receiving third text data describing a second object and a second location associated with the second object.

17. A system comprising: an input interface configured to receive queries; and a processor in communication with the input interface, the processor being programmed to: receiving, at a criss-cross attention layer of a pre-trained model comprising an input to the pre-trained model, first text data describing a first object and second text data describing a first scene, wherein the first text data comprises a description of a location of the first object; Concatenating the first text data and the second text data to generate a single prompt describing the first text data and the second text data using a pre-trained model with a crossed attention layer; generating a broadcasted position mask using the first text data, the position mask consisting of at least positions of the first object indicating pixels associated with the first object; generating a broadcasted all-ones matrix associated with the second text data describing the first scene using the second text data, wherein the broadcasted all-ones matrix is ​​a mask including all pixels associated with the first scene; Determine the key matrix and the value matrix using separate linear projections of the individual cues; Determine the query matrix using (i) the output of the previous layer or (ii) the linear projection of the input to the pre-trained model; generating a broadcasted position matrix in response to concatenating the broadcasted position mask and the broadcasted all-ones matrix; generating a cross-attention map using the query matrix, the key matrix, and the broadcast position matrix; as well as In response to performing a fixed number of iterations corresponding to the criss-cross attention layers of the pre-trained model, a final image is output.

18. The system of claim 17, wherein: The processor is programmed to receive third text data describing a second object and a second location associated with the second object.

19. The system of claim 17, wherein: The cross-attention map is generated using at least a softmax function.

20. The system of claim 17, wherein: The criss-cross attention map is generated using element-wise multiplication with the broadcast position matrix and matrix multiplication of the query matrix and the key matrix.

Images