System and method for localized text-to-image generation for free via cross-attention

Abstract

To provide an image generation method and system that utilize a machine learning network, such as a text-to-image generation system.SOLUTION: A method for generating synthetic image data includes: receiving first text data describing a first object and second text data describing a first scene; utilizing a model with cross-attention layers and concatenating the first text data and the second text data to generate a prompt; generating a location mask constructed from a location; generating an all-one matrix associated with the second text data; computing a key matrix and a value matrix utilizing linear projections; computing a query matrix utilizing linear projections; concatenating the location mask and the all-one matrix to generate a location matrix; utilizing the query matrix, the key matrix, and the location matrix to generate a cross-attention map; and outputting a final image.SELECTED DRAWING: None

Description

[Technical field]

[0001] This disclosure relates to image generation using machine learning networks, such as text-to-image generation systems. [Background technology]

[0002] Background technology In recent years, text-to-image generation models have shown impressive performance. Models such as Stable Diffusion and Dall-E are capable of generating high-quality and diverse images from arbitrary text prompts. However, a major challenge these models face is that they rely solely on the text prompt for content control over the generation process, which may be insufficient for many applications. In particular, one of the most intuitive and user-friendly ways to control generation is to provide location-specific information that guides the model on where to generate specific elements in an image. Unfortunately, current pre-trained models face limitations in their ability to perform location-specific generation. These limitations stem not only from their inability to incorporate location information as input, but also from the inherent difficulties associated with composability, a known challenge for many multimodal foundational models. Summary of the Invention [Problem to be solved by the invention]

[0003] Existing methods that address this problem typically fall into three main categories: training an entirely novel model, fine-tuning an existing model with additional components such as task-specific encoders, or strategically combining multiple samples into one. These approaches often require large amounts of training data, resources, and / or extended inference time, making them impractical for real-world applications due to their time- and resource-intensive nature. On the other hand, in inter-prompt image editing, we can edit the images generated based on the modified text prompt by manipulating the mutual attention map in a text-to-image generation model. Of note, this work also shows that the mutual attention layer plays a pivotal role in controlling the spatial layout of the generated objects associated with specific phrases in the prompt. [Means for solving the problem]

[0004] overview According to a first embodiment, a computer-implemented method of generating synthetic image data discloses receiving, at a mutual attention layer of a pre-trained model including an input to the pre-trained model, first text data describing a first object and second text data describing a first scene, the first text data including a description of a location of the first object. The method also includes utilizing the pre-trained model with the mutual attention layer to concatenate the first text data and the second text data to generate a single prompt describing the first text data and the second text data; utilizing the first text data to generate a broadcasted location mask constructed from at least a location of the first object indicative of pixels associated with the first object; and utilizing the second text data to generate a broadcasted all-ones matrix associated with the second text data describing the first scene, the broadcasted all-ones matrix including a location of all pixels associated with the first scene. the broadcasted location mask being a mask including the broadcasted location mask and the broadcasted all-ones matrix; and outputting a final image in response to performing a fixed number of iterations corresponding to the mutual attention layer of the pre-trained model.

[0005] According to a second embodiment, a computer-implemented method includes receiving, at a mutual attention layer of a model, first text data describing a first object and second text data describing a first scene, the first text data including a description of a location of the first object; utilizing a model having a mutual attention layer to concatenate the first text data and the second text data and generate a prompt describing the first text data and the second text data; utilizing the first text data to generate a broadcast location mask constructed from at least a location of the first object indicative of pixels associated with the first object; and utilizing the second text data to generate a broadcast location mask constructed from at least a location of the first object indicative of pixels associated with the second text data describing the first scene. The method also discloses the steps of generating a broadcasted all-ones matrix, where the broadcasted all-ones matrix is ​​a mask that includes all pixels associated with the first scene; computing a key matrix and a value matrix using separate linear projections of the prompt; computing a query matrix using linear projections of either i) the output of a previous layer or (ii) the input to a pre-trained model; generating a broadcasted location matrix in response to concatenating the broadcasted location mask and the broadcasted all-ones matrix; generating a mutual attention map using the query matrix, the key matrix and the broadcasted location matrix; and outputting a final image from the final layer of the model in response to performing a fixed number of iterations.

[0006] According to a third embodiment, a system includes an input interface configured to receive a query, and a processor in communication with the input interface, the processor receiving first text data describing a first object and second text data describing a first scene at a mutual attention layer of a pre-trained model including an input to the pre-trained model, where the first text data includes a description of a location of the first object, utilizing the pre-trained model with the mutual attention layer, concatenating the first text data and the second text data to generate a single prompt describing the first text data and the second text data, utilizing the first text data to generate a broadcasted location mask constructed from at least a location of the first object indicative of pixels associated with the first object, and the first scene, wherein the broadcasted all-ones matrix is ​​a mask including all pixels associated with the first scene; determining a key matrix and a value matrix using separate linear projections of a single prompt; determining a query matrix using linear projections of either (i) an output of a previous layer or (ii) an input to a pre-trained model; generating a broadcasted location matrix in response to concatenating the broadcasted location mask and the broadcasted all-ones matrix; generating a mutual attention map using the query matrix, the key matrix, and the broadcasted location matrix; and outputting a final image in response to performing a fixed number of iterations corresponding to the mutual attention layer of the pre-trained model. [Brief description of the drawings]

[0007] [Figure 1]FIG. 1 illustrates an embodiment of a block diagram showing an example architecture of a computing device, such as a camera device, a processor, a mobile device, and other computing devices described herein. [Diagram 2] FIG. 1 illustrates one embodiment of a process for utilizing text-to-image generation with mutual attention control (CAC). [Diagram 3] FIG. 13 illustrates an example of localized text-to-image generation utilizing the illustrated embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Detailed Description In one embodiment, the present system and method propose to use mutual attention control (CAC) to provide a pre-trained text-to-image model with better open vocabulary localization ability. Given caption and localization information, such as bounding boxes and semantic segmentation maps, along with their corresponding text descriptions, the system can first construct a new text input by concatenating the caption and all the prompts associated with the location information. The system can then compute a mutual attention map from this new text prompt and apply localization constraints to the mutual attention map according to the localization information. Such a method does not require any additional training or model architecture modification, such as designing a task-specific encoder. It also does not impose any language restrictions, such as using a fixed vocabulary set or language analyzer. Moreover, it is highly portable and can be easily integrated into a single forward pass in any mutual attention based text-to-image generation framework with only a few lines of code, and therefore does not require extra inference time.

[0009] FIG. 1 illustrates an embodiment of a block diagram showing an example architecture of a computing device, such as a camera device, a processor, a mobile device, and other computing devices described herein. It is understood that not all user devices include all of the components of architecture 100, and some user devices may include additional components not shown in architecture 100. Architecture 100 may include one or more processing elements 104 for executing instructions and retrieving data stored in non-transitory computer-readable storage element 102. The processing element 104 may include at least one processor. Any suitable processor or processors may be used. For example, processing element 104 may include one or more digital signal processors (DSPs) and / or image signal processors (ISPs). In some examples, processing element 104 may be effective to perform one or more functions of a generator and / or a classifier, as described above. Storage element 102 may include one or more different types of non-transitory computer-readable memory, data storage, or computer-readable storage media devoted to different purposes within architecture 100. For example, the memory element 102 may include flash memory, random access memory, disk-based storage, etc. Different portions of the memory element 102 may be used for, for example, storing program instructions for execution by the processing element 104, images or other digital works, and / or removable storage for transferring data to other devices.

[0010] The memory element 102 may also store software executed by the processing element 104. The operating system 122 may provide a user with an interface for operating the user device and may facilitate communication and commands between applications running on the architecture 100 and its various hardware. The transfer application 124 may be configured to receive images and / or video from other devices (e.g., mobile devices, image capture devices, and / or display devices) or from an image sensor 132 included in the architecture 100. In some examples, the transfer application 124 may be configured to upload received images to other devices (e.g., mobile devices and / or other computing devices) that may perform processing as described herein.

[0011] When implemented in a user device, the architecture 100 may also include a display component 106. The display component 106 may include one or more light emitting diodes (LEDs) or other suitable display lamps, and in some examples, the display component 106 may include one or more devices, such as, for example, a cathode ray tube (CRT), a liquid crystal display (LCD) screen, a gas plasma-based flat panel display, an LCD projector, a raster projector, an infrared projector, or other types of display devices.

[0012] The architecture 100 may also include one or more input devices 108 operable to receive input from a user. The input devices 108 may include, for example, push buttons, touch pads, touch screens, wheels, joysticks, keyboards, mice, trackballs, keypads, light guns, game controllers, or any other such devices or elements by which a user may provide input to the architecture 100. These input devices 108 may be incorporated into the architecture 100 or may be operatively coupled to the architecture 100 via a wired or wireless interface. In some examples, the architecture 100 may include a microphone 170 for capturing audio, such as voice commands. A speech recognition engine 180 may interpret acoustic signals of sounds captured by the microphone 170. In some examples, the speech recognition engine 180 may listen for a "wake word" received by the microphone 170. Upon receiving the wake word, the speech recognition engine 180 may stream the audio to a speech recognition server for analysis. In various examples, the speech recognition engine 180 can stream audio to an external computing device via the communications interface 112.

[0013] If the display component 106 includes a touch-sensitive display, the input device 108 may include a touch sensor operatively associated with the display component 106 to allow a user to use touch input (e.g., with a fingertip or electronic pen) to interact with images displayed by the display component 106. The architecture 100 may also include a power source 114, such as a rechargeable battery operable to be recharged via a wired alternating current (AC) converter, a conventional plug-in approach, or via other approaches, such as capacitive or inductive charging.

[0014] The communication interface 112 may include one or more wired or wireless components operable to communicate with one or more other user devices. For example, the communication interface 112 may include a wireless communication module 136 configured to communicate over a network according to any suitable wireless protocol, such as IEEE 802.11 or other suitable wireless local area network (WLAN) protocol. The short-range interface 134 may be configured to communicate using one or more short-range wireless protocols, such as near field communication (NFC), Bluetooth, Bluetooth LE, etc. The mobile interface 140 may be configured to communicate utilizing a cellular protocol or other mobile protocol. The global positioning system (GPS) interface 138 may communicate with one or more Earth orbiting satellites or other suitable location systems to identify the location of the architecture 100. The wired communication module 142 may be configured to communicate according to a USB protocol or any other suitable protocol. In various examples, the mobile interface 140 may enable a computing device to communicate with one or more other computing devices that may be used to perform various aspects of the techniques described herein.

[0015] The architecture 100 may also include one or more sensors 130, such as one or more position sensors, image sensors, and / or motion sensors. In some cases, one image sensor 132 is shown. Some examples of the architecture 100 may include multiple image sensors 132. For example, a panoramic camera system may include multiple image sensors 132, resulting in multiple images and / or video frames that may be switched and blended to form a seamless panoramic output. An example of an image sensor 132 may be a camera or other image sensor. In various examples, the image sensor 132 may be configured to capture color information, IR image data, image geometry information, and / or ambient light information.

[0016] The motion sensor may include any sensor that senses motion of the architecture, including, for example, a gyro sensor and an accelerometer. The motion sensor may be used in some examples to determine the attitude, such as, for example, pitch angle and / or roll angle, of the camera. The gyro sensor may be configured to generate a signal indicative of the rotational motion and / or change in attitude of the architecture (e.g., the magnitude and / or direction of the motion or the change in attitude). Any suitable gyro sensor may be used, including, for example, a ring laser gyro, a fiber optic gyro, a fluid gyro, a vibration gyro, etc. In some examples, the accelerometer may generate a signal indicative of acceleration (e.g., the magnitude and / or direction of the acceleration). Any suitable accelerometer may be used, including, for example, a piezoresistive accelerometer, a capacitive accelerometer, etc. In some examples, the GPS interface 138 may be utilized as the motion sensor. For example, a change in the position of the architecture 100 as determined by the GPS interface 138 may indicate motion of the GPS interface 138. As mentioned above, in some examples, the image sensor 132 may be effective to detect infrared light. In at least some examples, the architecture 100 may include an infrared light source that illuminates the surrounding environment.

[0017] FIG. 2 discloses a process description of using text-to-image generation with mutual attention control (CAC). The system can utilize a standardized set of evaluation metrics for localized text-to-image generation tasks using off-the-shelf large-scale pre-trained recognition models. The system can apply CAC to various state-of-the-art baseline text-to-image generation models and experiment with different forms of localization information including bounding boxes and semantic segmentation maps. The system and method can demonstrate that CAC provides pre-trained standard text-to-image models with new localized generation capabilities, and further improves models trained specifically for localized generation. Additionally, the present embodiment may show that the method can significantly improve the synthesis capabilities of text-to-image generation models with a simple heuristic that spatially separates components in the text prompt.

[0018] The goal of this work is to perform localized text-to-image generation given a pre-trained text-to-image generation model. The localization information provided by the user should consist of text phrases describing the contents and their associated constrained spatial locations in the image space. Common location information includes bounding boxes and semantic segmentation maps. Furthermore, the system aims to perform this task without (1) additional training or fine-tuning, (2) modifying the model architecture, (3) spending extra inference time, and (4) further restricting the input text space from the original model. The provided pre-trained model can be trained either with the localization information or alone with the text-image pairs.

[0019] Formally, we use a pre-trained text-to-image generation model ρ θ , a text prompt y0∈Y of length η0 n0 , and location information

number

number

[0020] State-of-the-art text-to-image generation models can achieve their success using mutual attention mechanisms. In one embodiment, the system can choose stable diffusion as the backbone model. However, in alternative embodiments, the system and method can be considered as applicable to other mutual attention based diffusion models such as Imagen and GANs, e.g. GigaGAN.

[0021] x∈p θ(x\y0), where x∈X, y0∈Y n For the task of sampling from p θ The mutual attention layer in l is the encoded text prompt.

number

number

number

number

[0022] The following formula,

number

[0023] This system is

number

number

[0024] In one example, a prompt 201 may include a text description ("dog") with an indicator of a location 202 associated with the first prompt that is applied to a matrix or map. A second prompt 203 may include the description "cat" and may indicate a location 204 associated with the first prompt as attached to an object. Another prompt 205 may indicate a scene.

[0025] For each location-specific information pair g i =(y i ,b i ) is a model, b i At pixel locations where y > 0, the text prompts i Therefore, based on the previous interpretation and findings, the (r,j,k)-th element in the attention map is the spatial constraint mask b i The jth entry in b i,(j) y if and only if is positive i Attention should be received from the k-th token in

[0026] As a result, the system first calculates the original location mask b i to match the dimension of the perceptual field at layer l.

number

number

number

number

number

[0027] The system then generates, in one example, m+1 attention maps.

number

number

number

number

number

number

[0028] The system calculates y for all i=1,...,m. i For example, if a user generates a "picture of a dining room with a cup on the dining table" and wants to provide bounding boxes for "cup" and "dining table", the system can use the location information to determine the portion of the attention map for the caption associated with the tokens for "cup" and "dining table" (i.e.,

number

number

[0029] Where:

number

number

number

[0030] However, this assumption does not always hold. For example, a user may request to generate a "dining room photo" without describing all the details of the scene, but they may still specify the locations of the "cups" and the "dining table" using bounding boxes without describing them in the caption. Thus, to apply our method to all inputs without this assumption, our system can:

number

[0031] Where:

number

number

[0032] The present system and method can in fact use the all-ones matrix 213 as a location mask for the caption y0 and special tokens. Similar to prompt-to-prompt editing, the present system can apply weights to the attention map.

number

number

number

[0033] Finally, the output of layer 219 is z (l) =l0(M (l) V (l) ), the framework is depicted in Figure 2. The system and method require neither further training nor model architecture modifications since it only needs to change the forward path of the pre-trained model at sampling time, nor does it require any other language restrictions or pre-settings, such as a fixed vocabulary set or language parser. It is also nicely packaged into the original optimized transformation framework, and therefore does not require additional inference time. Due to the minimal assumptions, the method is an open vocabulary plug-in for all text-to-image generation models that use mutual attention for text guidance, at no additional cost.

[0034] In addition to mutual attention, a self-attention layer is also essential in many text-to-image generation models to generate coherent spatial layouts. For example, applying self-attention control to a small part of the diffusion process in addition to mutual attention control can provide a more consistent geometry and color palette. However, self-attention control introduces complications, although minor in editing, in our setting because location information for different localization prompts may overlap each other, resulting in signal collisions at overlapping pixels and ambiguity of the mask.

[0035] One approach to incorporate self-attention control is to optimize each region separately according to different localization prompts before combining all the regions together. If both self-attention control and mutual attention control are applied to every diffusion step, the solution to this can be reduced to a rough multi-diffusion. As a result, the system can first apply multi-diffusion to a small part of the diffusion process to get the desired effect, and then perform mutual attention control diffusion as described above for the remainder of the diffusion time steps. A model such as GLIGEN that is fine-tuned based on localization information can also be used in the system to provide learned self-attention control.

[0036] In one exemplary embodiment and method, the system may be considered as a plug-in for Multi-Diffusion and GLIGEN to provide better localization capabilities, and such implementation can be done without adding additional cost to the two algorithms.

[0037] FIG. 3 shows an example of mutual attention control as a plug-in to existing methods for localized text-to-image generation. Mutual attention control can improve various types of localization (bounding boxes, semantic segmentation maps, and localized styles) using different base models (e.g., stable diffusion and GLIGEN). As shown in FIG. 3, a bounding box may be used as an input. The bounding box may reflect an object such as an "astronaut riding a horse" or a "pink flag". The caption may show a "Mars landscape" in one embodiment. The localized prompt may show an "astronaut riding a horse" and a "pink flag", corresponding to the input. The bounding box may include the scene and objects associated with the scene. As shown in the CAC example, the placement of the "astronaut riding a horse" is more accurate than the baseline model. Additionally, the flag may be of any suitable shape and location.

[0038] In another example, the input can utilize a semantic segmentation map. The semantic segmentation map can map an object or an augmented object corresponding to the image. In one example, the semantic segmentation map can correspond to a cat with one blue eye and one green eye. The caption can be "cute cat". The localized prompts can be "blue eyes" and "green eyes". The mutual attention control model can output a more accurate and higher quality image than other versions. Of course, other examples are also shown in FIG. 3.

[0039] In this specification, embodiments of the present disclosure are described. However, it should be understood that the disclosed embodiments are merely examples and that other embodiments may take various alternative forms. The drawings are not necessarily to scale, and some features may be exaggerated or minimized to show details of specific components. Therefore, specific structural and functional details disclosed in this specification should not be interpreted as limiting, but rather as a representative basis for teaching those skilled in the art to various uses of these embodiments. As will be understood by those skilled in the art, various features depicted and described in connection with any one of the drawings can be combined with features depicted in one or more other drawings to produce embodiments not explicitly depicted or described. The combination of the depicted features provides a representative embodiment for a typical application. However, various combinations and modifications of features consistent with the teachings of the present disclosure may be desirable for a particular application or implementation.

[0040] As used herein, "A," "an," and "the" refer to both singular and plural referents unless the context clearly indicates otherwise. By way of example, a "processor" programmed to perform various functions may refer to one processor programmed to perform each function, or may refer to multiple processors collectively programmed to perform each of the various functions.

[0041] Although exemplary embodiments have been described above, these embodiments are not intended to describe all possible forms encompassed by the claims. It will be understood that the terms used herein are terms of description rather than limitation, and that various modifications may be made without departing from the spirit and scope of the present disclosure. As previously mentioned, features of various embodiments may be combined to form further embodiments of the present invention that may not be expressly described or depicted. Although various embodiments may be described as offering advantages or being more preferred over other embodiments or prior art implementations with respect to one or more desired characteristics, those skilled in the art will recognize that depending on the particular application and implementation, one or more functions or characteristics may be compromised to achieve the overall desired system attributes. These attributes may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, maintainability, weight, manufacturability, ease of assembly, and the like. Even so, to the extent that any embodiment is described as less desirable than other embodiments or prior art implementations with respect to one or more characteristics, such embodiments are not outside the scope of the present disclosure and may be desirable for a particular application.

Claims

1. 1. A computer-implemented method for generating synthetic image data, comprising: receiving, at a mutual attention layer of a pre-trained model comprising an input to the pre-trained model, first text data describing a first object and second text data describing a first scene, the first text data including a description of a location of the first object; utilizing the pre-trained model with the mutual attention layer to concatenate the first text data and the second text data, and generate a single prompt that describes the first text data and the second text data; utilizing the first text data to generate a broadcast location mask constructed from at least the location of the first object indicative of pixels associated with the first object; utilizing the second text data to generate a broadcasted all-ones matrix associated with the second text data describing the first scene, the broadcasted all-ones matrix being a mask that includes all pixels associated with the first scene; calculating a key matrix and a value matrix using separate linear projections of the single prompt; Computing a query matrix using a linear projection of either (i) the outputs of a previous layer or (ii) the inputs onto the pre-trained model; generating a broadcasted location matrix in response to concatenating the broadcasted location mask and the broadcasted all-ones matrix; generating a mutual attention map using the query matrix, the key matrix, and the broadcasted location matrix; outputting a final image in response to performing a fixed number of iterations corresponding to the mutual attention layer of the pre-trained model; The method includes:

2. The method of claim 1 , wherein the mutual attention map is generated utilizing at least a softmax function.

3. The method of claim 1 , wherein the mutual attention map is generated using element-wise multiplication with the broadcasted location matrix and matrix multiplication of the query matrix and the key matrix.

4. The method of claim 1 , wherein the pre-trained models include at least a stable diffusion model.

5. 2. The method of claim 1 , wherein the pre-trained model includes one or more of the mutual attention layers, a first layer configured to generate an output that is sent to a second layer.

6. The method of claim 1 , wherein the second iteration is configured to generate an output that is sent to the third iteration.

7. The method of claim 1 , wherein the mutual attention layer includes a final layer that outputs the final image associated with the first text data and the second text data.

8. The method of claim 1 , wherein the pre-trained model is configured to adjust an effect of each token according to a weight associated with the mutual attention map.

9. The method of claim 1 , further comprising the step of receiving third text data describing the second object.

10. The method of claim 9 , wherein the third text data describes a second location associated with the second object.

11. 1. A computer-implemented method for generating synthetic image data, comprising: receiving, in a mutual attention layer of the model, first text data describing a first object and second text data describing a first scene, the first text data including a description of a location of the first object; utilizing the model having the mutual attention layer to concatenate the first text data and the second text data, and generating prompts describing the first text data and the second text data; utilizing the first text data to generate a broadcast location mask constructed from at least the location of the first object indicative of pixels associated with the first object; utilizing the second text data to generate a broadcasted all-ones matrix associated with the second text data describing the first scene, the broadcasted all-ones matrix being a mask that includes all pixels associated with the first scene; calculating a key matrix and a value matrix using separate linear projections of the prompts; Computing a query matrix using a linear projection of either (i) the output of a previous layer or (ii) the input to a pre-trained model; generating a broadcasted location matrix in response to concatenating the broadcasted location mask and the broadcasted all-ones matrix; generating a mutual attention map using the query matrix, the key matrix, and the broadcasted location matrix; outputting a final image from a final layer of the model in response to performing a fixed number of iterations; The method includes:

12. The method of claim 11 , wherein the fixed number of iterations corresponds to the mutual attention layer.

13. The method of claim 11 , wherein the model comprises an input image model.

14. The method of claim 11 , wherein the model is configured to adjust the effect of each token according to a weight associated with the mutual attention map.

15. The method of claim 11 , wherein the final image is associated with the first text data and the second text data.

16. 12. The method of claim 11, further comprising receiving third text data describing a second object and a second location associated with the second object.

17. an input interface configured to receive a query; a processor in communication with the input interface; A system comprising: The processor, receiving, at a mutual attention layer of a pre-trained model including an input to the pre-trained model, first text data describing a first object and second text data describing a first scene, the first text data including a description of a location of the first object; utilizing the pre-trained model having the mutual attention layer to concatenate the first text data and the second text data, and generate a single prompt that describes the first text data and the second text data; Utilizing the first text data to generate a broadcast location mask constructed from at least the location of the first object indicative of pixels associated with the first object; generating a broadcasted all-ones matrix using the second text data, the broadcasted all-ones matrix being associated with the second text data describing the first scene, the broadcasted all-ones matrix being a mask including all pixels associated with the first scene; determining a key matrix and a value matrix using separate linear projections of the single prompt; determining a query matrix using a linear projection of either (i) the output of a previous layer or (ii) the input onto the pre-trained model; generating a broadcasted location matrix in response to concatenating the broadcasted location mask and the broadcasted all-ones matrix; generating a mutual attention map using the query matrix, the key matrix, and the broadcasted location matrix; and outputting a final image in response to performing a fixed number of iterations corresponding to the mutual attention layer of the pre-trained model. The system is programmed to do so.

18. 20. The system of claim 17, wherein the processor is programmed to receive third text data describing a second object and a second location associated with the second object.

19. 20. The system of claim 17, wherein the mutual attention map is generated utilizing at least a softmax function.

20. 20. The system of claim 17, wherein the mutual attention map is generated using element-wise multiplication with the broadcasted location matrix and matrix multiplication of the query matrix and the key matrix.

Images