Segmentation-free guidance in diffusion models

Abstract

Certain aspects of the present disclosure provide techniques for generating an output image based on a textual prompt. A method can include receiving a textual prompt, providing a user interface including one or more input elements associated with one or more words of the textual prompt, receiving input corresponding to at least one of the one or more input elements, the input indicating a semantic importance of each of at least one of the one or more words associated with the at least one of the one or more input elements, and generating an output image based on the textual prompt and the input.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application Serial No. 18 / 511,692, filed November 16, 2023, entitled “Segmentation Free Guidance in Diffusion Models,” the entire contents of which are incorporated herein by reference.

[0003] introduction Technical Field

[0004] Various aspects of this disclosure relate to image generation techniques using diffusion models.

[0005] Related technical descriptions

[0006] Diffusion models are a class of generative deep learning models capable of generating high-fidelity images based on text prompts. The basic principle involves the model adding noise to training data, such as original images, and learning to recover the data by reversing the noise processing. For example, during the training phase, the model might add noise to an image of an apple and learn to recover the image of an apple from the noise (e.g., possibly randomly generated noisy data). Thus, the model learns to generate an apple from the noise. The model can be trained on many images of many different objects and thus learn to generate many different images of many different objects. During inference, to generate an image based on a text prompt, the diffusion model starts with a noisy image (e.g., random data) and can subsequently iteratively denoise the noisy image to ultimately generate an output image more aligned with the text prompt. During the iterative process, the diffusion model generates a latent image representation of the output image at each stage of the iteration. While diffusion models can produce a wide variety of images, they are sometimes insufficient in achieving true fidelity to a given prompt. To address this issue, various methods known as guided or conditional techniques have been devised. These strategies enhance image fidelity, such as in terms of quality, but may reduce its diversity, such as in terms of coverage.

[0007] A common guidance approach is classifier guidance, where a separate classifier model is trained alongside the diffusion model. The classifier provides gradients and probabilities to guide the diffusion model to maintain consistency and produce images more consistent with the textual cue. However, introducing an additional classifier model increases computational complexity and latency. Therefore, classifier-free guidance is introduced to simplify the process by using the diffusion model itself as the classifier. This is performed by running the diffusion model twice, once with the full cue and once with an empty cue, to generate two conditional scores. The contrasting conditional scores, reflecting the residual noise levels from the two iterations, are then used as guidance signals.

[0008] While existing methods produce images that follow the input text prompts, limitations remain in terms of local fidelity and semantic consistency. The generated images often contain irrelevant or contradictory visual features from different spatial regions. This degrades quality and alignment with the input text concepts. Summary of the Invention

[0009] One aspect provides a method for generating an output image based on text prompts. The method includes: obtaining text prompts; encoding the text prompts into a plurality of conditionalized lexical units; for each of one or more blocks of a latent image representation: computing a plurality of corresponding cross-attention weights corresponding to the plurality of conditionalized lexical units based on the block as a query and the plurality of conditionalized lexical units as keys; modifying the maximum cross-attention weight among the respective plurality of cross-attention weights to generate modified respective plurality of cross-attention weights; performing a denoising iteration using the modified respective plurality of cross-attention weights for each of the one or more blocks to obtain a modified latent image representation; and generating an output image based on the modified latent representation.

[0010] On the other hand, a method for generating an output image based on a text prompt is provided. The method includes: receiving a text prompt; providing a user interface including one or more input elements associated with one or more words of the text prompt; receiving input corresponding to at least one of the one or more input elements, the input indicating the semantic importance of each word in at least one of the one or more words associated with the at least one of the one or more input elements; and generating an output image based on the text prompt and the input.

[0011] Other aspects provide: an apparatus capable of operating to, being configured to, or otherwise adapted to perform one or more of the methods described herein and / or those elsewhere in the document; a non-transitory computer-readable medium comprising instructions that, when executed by a processor of the apparatus, cause the apparatus to perform the methods described herein and those elsewhere in the document; a computer program product embodied on a computer-readable storage medium comprising: code for performing the methods described herein and those elsewhere in the document; and / or an apparatus comprising components for performing the methods described herein and those elsewhere in the document. By way of example, an apparatus may include a processing system, a device having a processing system, or a processing system cooperating via one or more networks. An apparatus may include: one or more memories; and one or more processors coupled to the one or more memories, the one or more processors being configured (e.g., causing the apparatus) to perform any portion of any of the methods described herein. In some examples, one or more processors may be pre-configured to perform the various functions or operations described herein without being configured by software.

[0012] For illustrative purposes, the following description and figures illustrate certain features. Attached Figure Description

[0013] The accompanying drawings depict certain features of the various aspects described herein and should not be considered as limiting the scope of this disclosure.

[0014] Figure 1 An example user interface for an image generation system is depicted.

[0015] Figure 2 Additional details are depicted of an example image generation system that implements diffusion processing according to examples of this disclosure.

[0016] Figure 3 Additional details regarding the specific implementation of the diffusion processor are described.

[0017] Figure 4 Additional implementation details are described for one or more guidance modules used to generate attention weights.

[0018] Figure 5 Additional implementation details are described for one or more guidance modules used to generate and modify attention weights.

[0019] Figures 6A to 6I An example user interface is depicted for receiving user input and displaying the resulting image.

[0020] Figure 7An example computational flow for generating cross-attention features between two sequences, according to an example of this disclosure, is described.

[0021] Figure 8 A diagram illustrating an example AI architecture that can be used to implement one or more machine learning (ML) models, according to examples of this disclosure.

[0022] Figure 9 This is an exemplary block diagram of an example artificial neural network (ANN) according to the present disclosure.

[0023] Figure 10 An example method for generating images is described.

[0024] Figure 11 Another example method for generating images is described.

[0025] Figure 12 The various aspects of the example processing system are described.

[0026] Figure 13 Various aspects of another example processing system are described. Detailed Implementation

[0027] This disclosure provides apparatus, methods, processing systems, and computer-readable media for generating virtual content using text descriptions. This process (also referred to herein as "text-to-image transformation") can be used in fields such as content creation and art generation. In some aspects, diffusion models transform textual cues into realistic visual representations. These models, at their core, simulate a diffusion process designed to transform a basic initial distribution (e.g., random noise) into a complex data distribution, typically an image.

[0028] The operation of the diffusion model is inherently iterative. The process generally begins with loading a noisy image, which evolves into a sharper, more realistic representation through successive refinement steps. However, to effectively generate images aligned with text cues, a mechanism may be needed to bridge the gap between text and visual data. Cross-attention, for example, can be used. While some aspects of cross-attention are described, other suitable techniques can be used in conjunction with aspects of this paper, including the specific user interface discussed herein.

[0029] In some respects, based on certain techniques discussed in this paper, cross-attention can be used within the context of a diffusion model to measure the relevance between fragments of textual cues and spatial regions (also known as patches) of the evolving image. By measuring this relevance, the diffusion model receives cues about how to refine the image during each iteration step. Instead of a one-size-fits-all approach, this type of cross-attention mechanism discussed in this paper evaluates individual textual keywords and aligns them with corresponding regions in the image, thus ensuring a more accurate representation. Granular attention tailored to specific textual cues, combined with iterative refinement by the diffusion model, facilitates the generation of images that are not only visually striking but also context-aligned with the input text.

[0030] However, challenges exist related to ensuring spatial consistency in the generated images. For example, an image might display or emphasize irrelevant background elements that are not well aligned with the textual intent. Furthermore, uniformly applying cross-attention mechanisms to decode the relevance between text and image regions can potentially overshadow other details due to the dominant theme. These drawbacks underscore the need for more refined methods that tailor the focus on individual image fragments, ensuring that each part more faithfully corresponds to its associated textual cues.

[0031] In some respects, semantic image generation using local cross-attention offers a solution to the limitations of current diffusion models. By adjusting attention weights based on image regions, spatial alignment between visual features and corresponding text is improved. This provides a technological advancement in inconsistent semantics, which is partly due to the dominant theme overriding local details. Specifically, attenuating the highest semantic attention weight for each block reduces irrelevant or unrealistic combinations within each region. In some respects, this approach does not require a separate segmentation model but can achieve localization using cross-attention.

[0032] More generally, certain aspects of this paper provide techniques for decomposing text cues into multiple parts or “conditionalized lexicals” (e.g., each part or “conditionalized lexical” corresponds to one or more words (e.g., words, phrases, etc.) of the text cue). For example, in a simplified example, the text cue “the dog is on the sofa in the office” can be decomposed into conditionalized lexicals for “dog,” “sofa,” and “office.” During one or more iterations of text cue-based denoising, a diffusion model can be configured to determine, for one or more blocks of the current image representation based on the current iteration, which conditionalized lexical is most relevant (e.g., most similar) to a block. For example, one block of the current image may be most closely similar to a dog, while another block may be most closely similar to a sofa. The techniques of this paper are configured to guide the diffusion model and then focus the block on the conditionalized lexical that the block is already most similar to, such that during each iteration, the block continues to be denoised to better resemble the conditionalized lexical that the block is most similar to. It should be noted that a given block may resemble different conditionalized lexicals in different iterations and be denoised accordingly. Therefore, each block can be augmented to different conditionalized lexical units during each iteration, thereby improving the overall output image, as a block resembling a sofa becomes more similar to a sofa, and a block resembling a dog becomes more similar to a dog. For example, a block resembling a sofa might be less affected by the conditionalized lexical units “dog” and “office,” while a block resembling a dog might be less affected by the conditionalized lexical units “sofa” and “office,” resulting in a higher quality overall generated image. This provides an improvement to techniques where the entire image might be tuned more towards a given conditionalized lexical unit, thereby augmenting the entire image to better resemble, for example, “dog,” but at the cost of fidelity in the “sofa” and “office” aspects of the image. Conversely, the technique presented in this paper augments each block of the image to better resemble the corresponding object within the block, such that the “dog” portion of the image looks more like a dog, the “sofa” aspect looks more like a sofa, and the “office” aspect looks more like an office.

[0033] As discussed, some aspects of this paper employ cross-attention techniques to determine the most similar conditionalized lexical for a given block in a single iteration, and adjust the cross-attention weights (such as the cross-attention weights associated with these conditionalized lexicals) to guide the diffusion model to focus the block on the conditionalized lexical. In some respects, these cross-attention weights are computed as part of the operation of the diffusion model anyway, thus the techniques of this paper add very little computational overhead and complexity, providing the technical benefit of computationally efficient image improvement, unlike classifier models with heavy computational requirements. Therefore, the techniques discussed in this paper can even be implemented in devices with lower computational complexity, such as some mobile or edge devices. Images can be implicitly segmented into blocks for classification based on conditionalized lexicals to guide the diffusion model, and thus such techniques can be referred to as segmentation-free guidance.

[0034] In some respects, a graphical user interface (GUI) is provided that allows a user to indicate the importance of different parts of a text prompt for the generated image (such as for the entire image, or for one or more blocks of the image). This GUI can be used in conjunction with the techniques discussed herein regarding diffusion models and / or cross-attention techniques, or it can be used with other image generation techniques, models, etc. For example, a diffusion model can be guided to enhance an image or one or more blocks of an image to more closely resemble the more important parts indicated by the text prompt. For example, the user interface can allow a user to provide the relative emphasis intensity of one or more words or phrases in the text prompt, also known as semantic importance. An ML model, such as a diffusion model, can be guided to generate an image or one or more blocks of an image, as if weighted according to the semantic importance of one or more words to a phrase. For example, a word or phrase can be given twice the relative importance of other words or phrases in the text prompt, causing the image or block to be directed toward that word or phrase. In another example, the first word can have twice the importance compared to the third word, and the second word can have 1.5 times the importance, ensuring that the image or block is guided at most to the first word, more often to the second word, and at least to the third word. This type of technology offers the technical benefit of user control over image generation, even on screens with limited space, by providing input elements associated with words or phrases of textual prompts that can be easily reviewed and manipulated. This further allows for customization of images based on the user's creative direction, a technological improvement over fully automated conditional settings. Specifically, users can adjust the guidance weights to align the visual with the intent. This user-emphasized mapping enhances creative flexibility and efficiency.

[0035] Example environment and interface for performing image generation

[0036] Figure 1 An example user interface 102 of an image generation system 100 according to an example of the present disclosure is illustrated. In some aspects, one or more components of the image generation system 100 may be implemented or run on one or more processors that can be coupled to one or more memories. For example, the one or more processors may be configured to perform one or more functions of the one or more components of the image generation system 100, such as by executing instructions stored in one or more memories, or without requiring instructions from memory. In some aspects, the one or more processors may be part of a single device, such as a computing device, user equipment, vehicle, etc. In some aspects, the device may include one or more components of the image generation system 100, and / or one or more components that can be coupled to the image generation system 100.

[0037] User interface 102 accepts user input in the form of a text prompt 104. The text prompt 104, indicating the desired image, is entered by the user via text input field 106. The submit user interface element 108, located within user interface 102, initiates the sending of the text prompt 104 to one or more components of the image generation system for further processing.

[0038] In some respects, upon submission, text hint 104 is provided to text encoder 110. Text encoder 110 is configured to convert text hint 104 into encoded text embeddings. In some respects, the encoded text embeddings capture the inherent semantic nuances of text hint 104, thereby allowing for the interpretation of the underlying image description. For example, text encoder 110 may be similar to a text encoder used with any diffusion model, such as Dall-E 2, Imagen, stable diffusion, Midjourney, etc.

[0039] The generated encoded text embedding is then provided to image generator 112. In some respects, image generator 112 generates an image based on one or more diffusion processes. For example, image generator 112 may be similar to an image generator used with any diffusion model, such as Dall-E 2, Imagen, stable diffusion, Midjourney, etc. For example, by applying a series of iterative sampling and denoising stages to (e.g., a predetermined) number of time steps, image generator 112 creates an output image 114. Output image 114 is similar to a text prompt 104 in the input text input field 106. The user may then introduce new text prompts 104 into the text input field 106 to generate alternative or additional images.

[0040] Figure 2 Additional details are depicted of an example image generation system 200 implementing diffusion processing according to examples of this disclosure. In some aspects, one or more components of the image generation system 200 may be implemented or run on one or more processors that can be coupled to one or more memories. For example, the one or more processors may be configured to perform one or more functions of the one or more components of the image generation system 200, such as by executing instructions stored in one or more memories, or without requiring instructions from memory. In some aspects, the one or more processors may be part of a single device, such as a computing device, user equipment, vehicle, etc. In some aspects, the device may include one or more components of the image generation system 200, and / or one or more components that can be coupled to the image generation system 200.

[0041] In some respects, an example image generation system 200, which may be similar to or the same as image generation system 100, generates an output image 202 based on one or more conditionals 204. The one or more conditionals 204 may include those... Figure 1 The text prompt 206 is the same as or similar to the text prompt 104. The text prompt 206 can be provided to a text encoder 208 configured to encode the text prompt 206 into a text embedding 210. The text encoder 208 can be used with... Figure 1 The text encoder 208 is the same as or similar to the text encoder 110. The text encoder 208 encodes the text prompt 206 into a latent feature representation that captures the semantic meaning of the text. In some aspects, the text encoder 208 uses a transformer model such as BERT, CLIP, or another language model to encode the input text prompt 206 into embedding vectors. In some aspects, the transformer encodes the text into embedding vectors (e.g., text embedding 210) by passing the text through multiple self-attention layers. Each self-attention layer transforms the text into a progressively more abstract high-dimensional representation that extracts the contextual relationships between words and concepts in the text prompt 206. The resulting output of the text encoder 208 is a dense latent vector representation (e.g., text embedding 210) of the encoded text prompt 206. This text embedding 210 captures the semantic essence of the text prompt 206 in a format that the image generation system 200 can consume.

[0042] The text embedding 210 is then provided to the diffusion processor 212. In some aspects, the diffusion processor 212 includes an encoder-decoder neural network architecture, where the encoder progressively downsamples the noisy image through a series of layers, while the decoder upsamples the less noisy image to reconstruct the output image in the latent space. In some aspects, at each intermediate layer, cross-attention may be applied to adapt the in-process image features obtained by the encoder-decoder architecture using the encoded text embedding 210. In some aspects, the cross-attention aligns the textual semantics of the text embedding 210 with the spatial image region throughout the generation process performed by the diffusion processor 212.

[0043] At the first time step (e.g., time step embedding 216), a noisy image (e.g., 214) is provided to the diffusion processor 212. The diffusion processor 212 can use the encoder-decoder network described previously to estimate or predict ε. θ (x t The noise level is ε(t). The resulting denoised image can be obtained by subtracting or removing the predicted noise ε from the noisy image. θ (x t The result is obtained by using ,t). For example, as depicted at 218, the result corresponding to the noisy image X can be predicted. 1000 noise level ε θ (x t ,t). Denoising images (e.g., X 999 ) by extracting from noisy images (X 1000 Subtract or remove the prediction noise ε from the result. θ (x tThe latent vector representation of the image (e.g., a noisy image) is obtained by using each denoising loop. After many time steps, the amount of prediction noise is reduced to an acceptable level, and a denoised image 211 is obtained in the latent space. The denoised image 211 in the latent space is provided to a decoder 220, which then decodes the latent representation of the denoised image 211 into the pixel space, thereby producing an output image 202 based on (e.g., matching) conditional text cue 206.

[0044] Figure 3 Additional details are depicted regarding specific implementations of the diffusion processor 302 according to examples of this disclosure. In some aspects, one or more components of the diffusion processor 302 may be implemented or operated on one or more processors that can be coupled to one or more memories. For example, the one or more processors may be configured to perform one or more functions of the one or more components of the diffusion processor 302, such as by executing instructions stored in one or more memories, or without requiring instructions from memory. In some aspects, the one or more processors may be part of a single device, such as a computing device, user equipment, vehicle, etc. In some aspects, a device may include one or more components of the diffusion processor 302, and / or one or more components that can be coupled to the diffusion processor 302.

[0045] In some respects, the diffusion processor 302 can be used with Figure 2 The diffusion processor 212 is the same as or similar to the diffusion processor 212. In the example, and similar to... Figure 2 One or more conditionalization parameters 304 can be used to customize, refine, or guide the image generation process. For example, conditionalization parameters 304 may include semantic graph conditionalization parameter 306, text conditionalization parameter 308, representation conditionalization parameter 310, and image conditionalization parameter 312. Semantic graph conditionalization parameter 306 refers to a latent feature map encoding semantic information about the image; this conditionalization parameter can provide additional guidance regarding the content and style of the desired image beyond the text conditionalization parameter 308. Representation conditionalization parameter 310 refers to an encoded representation of the image capturing style, text, composition, etc., which allows the model to copy elements from a reference image. Image conditionalization parameter 312 refers to a representation of the image capturing style, text, composition, etc. (e.g., in pixel space), which allows the model to copy elements from a reference image. In addition to the conditions described above, other conditionalizations that set conditions or modify the diffusion process are envisioned.

[0046] In some respects, it will correspond to Figure 1The text conditionalization parameters 308 of the text prompt 104 are provided to the encoder 314. The encoder 314 may be the same as or similar to the previously described text encoder 208. The encoder 314 generates a text embedding 316 that may be the same as or similar to the previously described text embedding 210. The text embedding 316 is provided to the diffusion processor 302. In some respects, the diffusion processor 302 may refer to a specific implementation of the diffusion model. Although Figure 3 This description pertains to the diffusion model architecture depicted, but it should be understood that the techniques described herein are equally applicable to other diffusion models and / or architectures.

[0047] The diffusion processor 302 utilizes an encoder-decoder framework 320 to estimate the noise level present within the input latent image representation 318. The encoder-decoder framework 320 may include an encoder section 322 that progressively downsamples the noisy image (e.g., 318) using multiple layers 324, thereby reducing the resolution of the noisy image (e.g., 318). The decoder section 326 then restores the resolution of the downsampled image to reconstruct the input latent image representation 318 and provides a noise estimate corresponding to the amount of predicted noise within the noisy image (e.g., 318). Subsequently, the encoder-decoder framework 320 may extract (e.g., subtract or remove) the predicted noise from the input latent image representation 318 to obtain an output latent image representation 328 (e.g., the output latent image representation 328 as the output of a given iteration of the encoder-decoder framework 320). i In this encoder-decoder framework, there are additional iterations to occur, where the output latent image representation from the previous iteration is 328. i The input representation 318 used as the next iteration, or the output latent image representation 328 as the output of the final iteration of the encoder-decoder framework 320. f The amount of noise eliminated can vary depending on factors such as noise prediction from the encoder-decoder frame 320, time-specific embedding, and / or the mean and bias of the noise distribution.

[0048] In some respects, guidance module 330 j -330 j+n Cross-attention can be applied to each block or region of the evolving image. In each intermediate layer, the guidance module 330... j -330 j+nEncoded text embeddings 316 are used to refine the image features derived from the encoder-decoder framework 320. In some respects, cross-attention aligns the textual semantics of the text embeddings 316 with spatial image regions throughout the generation process of the encoder-decoder framework 320. That is, at each intermediate layer, the text embedding (e.g., 316) can be mapped to an intermediate layer of the encoder-decoder framework 320 via one or more cross-attention layers. These cross-attention layers implement cross-attention of Q (query), K (key), and V (value) as a learnable projection matrix, where K and V are obtained from the text embeddings 316, and Q is obtained from one or more blocks of the input (e.g., 318). The cross-attention weights generated by the cross-attention layers indicate how much each spatial region of the image is associated with different words or phrases in the prompt (e.g., specific lemmas in the text embeddings 316).

[0049] At an initial time step of 321, the input latent image representation 318 can be derived from the amount of randomly generated noise. After T iterations (e.g., time steps 321, 323, and 325), the amount of predicted noise can fall below a certain threshold, causing the diffusion processor 302 to output the final latent image representation 328. f The decoder 332 then outputs a latent image representation 328. f Decode to generate output image 334.

[0050] In some respects, the diffusion processor 302 can implement a classifier-free guided approach by executing the encoder-decoder framework 320 twice (once with text embedding 316 and once with an empty cue (e.g., an empty text embedding)). The difference in noise obtained from the two runs is used as an estimated or predicted noise amount that can be removed. The encoder-decoder framework 320 can then remove this predicted noise amount from the input latent image representation 318 to obtain the output latent image representation 328 (e.g., 328...). i Or 328 f Using a classifier-free method, the text embedding 316 influences the location and amount of noise removed from the input latent image representation 318. In some respects, the amount of noise removed is based on the predicted noise amount determined by the classifier-free method, the temporal embedding, and the mean and / or bias of the noise distribution.

[0051] In some respects, for each block or region of the image in progress, the guidance module 330 j -330 j+n The modified cross-attention process is performed by identifying the most relevant text terms (conditional terms) with the highest attention weight for a block or region from the text embedding 316. In some respects, the guidance module 330 j -330 j+nThe influence of the weighted text terminator is then reduced specifically for that block, such as by scaling its attention weight (e.g., by a negative factor), setting the attention weight to 0, or otherwise reducing the attention weight relative to other attention weights. For each image block, the process of selectively modifying, attenuating, or masking the text terminator with the highest significant attention weight given by the cross-attention process emphasizes its role in removing noise from the input latent image representation 318. This adjustment to the noise level refines the predicted noise level or associated conditional signal for each block, thereby enhancing its alignment with local semantics. By modifying or attenuating the dominant semantic concept of each block, the guidance module 330... j -330 j+n This reduces interference from text terms unrelated to the block content. It emphasizes features relevant to the textual cues while ignoring irrelevant or unrealistic combinations of cues. While some aspects of classifier-free methods have been discussed, in which the difference between noise from two runs is used as an estimate or prediction of the amount of noise that can be removed, thus reducing the attention weight of the most relevant text terms to emphasize features related to those terms, it should be understood that other types of models can be used where the attention weight of the most relevant text terms is increased to emphasize features related to those terms. For example, the attention weight of the most relevant text terms can be increased to emphasize features related to those terms, where a single run is used to estimate or predict the amount of noise.

[0052] In the segmentation-free method, the diffusion processor 302 runs the encoder-decoder framework 320 twice: first using the unmodified text embedding 316 without modifying the cross-attention weights, and then again using the unmodified text embedding 316 but with modified cross-attention weights, such as modifying (e.g., reducing) the cross-attention weights associated with the text words with the highest cross-attention weights in the text words used for each block. The noise variance between these two runs is used as an estimate or prediction of the noise to be extracted. The encoder-decoder framework 320 can then remove this predicted noise from the input latent image representation 318 to obtain the output latent image representation 328 (e.g., 328...). i Or 328 f Using a segmentation-free method, the text embedding 316 influences the location and amount of noise removed from the input latent image representation 318. The amount of noise removed can be based on the predicted noise amount provided by the classifier-free method, the temporal embedding, and the mean and / or bias of the noise distribution.

[0053] Figure 4 A guide module 402 is depicted according to an example of this disclosure. j -402 j+n Additional specific implementation details. Guidance Module 402 j-402 j+n Compatible with guidance module 330 j -330 j+n Similar or identical, with the addition of attention weight modifier 404. Guidance module 402. j -402 j+n Associating textual semantics with spatial image regions through cross-attention involves generating cross-attention weights based on a key vector K derived from text lexicons and a query vector Q derived from blocks. For each block 408, a cross-attention weight generator 406 computes a cross-attention weight 416 between a query 410 derived from block 408 and a corresponding key 412 derived from text embedding 316. Each of the cross-attention weights 416 can be associated with a corresponding key 412, and thus with a corresponding text lexicon. An attention weight modifier 418 then adjusts the cross-attention weights 416 according to a local conditionalization technique. In some aspects, the local conditionalization technique may include the attention weight modifier 418 identifying a maximum attention weight 420 among the cross-attention weights 416, the maximum attention weight 420 representing the most relevant semantic concept for that block 408. The attention weight modifier 418 then modifies (e.g., by scaling) the maximum attention weight 420 to guide an intensity 422 (e.g., a negative scalar) to specifically modify its effect on that block 408. In some respects, the attention weight modifier 418 scales the maximum attention weight 420 by the negative guidance strength (e.g., "-a") to specifically reduce its effect on the block 408. In some respects, the negative guidance strength (e.g., "a") can be set to zero.

[0054] For example, attention weight modifier 418 can determine the corresponding text embedding 316 (e.g., including text lexical units C1-C1) for the i-th block in j cross-attention levels / layers. x The lexical C in ) n attention weights It has the highest attention weight. Cross-attention weight. It is the i-th block and the text word C n A measure of relevance, where j is indexed across the encoder-decoder layers. Attention weights. The attention weights are then modified (e.g., reduced, such as by multiplying by a scalar "a") to obtain the modified attention weights. The other weights in the cross-attention weights 416 may be unmodified. In some aspects, the resulting scaled weights 424 are provided to the aggregator along with the unmodified cross-attention weights 416. In some aspects, the scaled weights 424 customize the semantic blending to emphasize the fidelity of locally inferred semantics within each block 408 (e.g., image region). In some aspects, this is achieved without utilizing a separate segmentation model that incorporates the cross-attention mechanism including the attention weight modifier 404.

[0055] Figure 5 A guide module 402 is depicted according to an example of this disclosure. j -402 j+n Additional specific implementation details are provided. In some aspects, the cross-attention weight 416 can be modified by a user-selectable weight modifier 502, which receives a user-initiated or user-specified weight 506 to be applied to the cross-attention weight 416 as input 504. Thus, one or more lemmas corresponding to the text embedding 316 can be modified based on the user input 504. The user-selectable weight modifier 502 modifies the cross-attention weight 416 with the user-initiated or user-specified weight 506 to obtain a cross-attention weight 508. In some aspects, the attention weight modifier 510 identifies a maximum attention weight 512 for the cross-attention weight 508 and then modifies that maximum attention weight 512, such as by a guidance strength 514, to specifically modify its effect on the block 408. This resulting attention weight can be... Other attention weights are equal to ,in The scaling input 504 is provided by the user and / or received from the user interface. In some aspects, the attention weight modifier 510 scales the maximum attention weight 512 with a negative guidance intensity (e.g., "-a") to specifically reduce its impact on the block 408. In some aspects, the resulting scaled cross-attention weight 516 is provided to the aggregator along with the unmodified cross-attention weight 416. In some aspects, the scaled cross-attention weight 516 customizes the semantic blending according to a user-specified or user-selectable scaling to emphasize the fidelity of locally inferred semantics within each block (e.g., image region). In some aspects, the negative guidance intensity (e.g., "a") may be set to zero.

[0056] In one example, Equation 1 involves modifying the noise predictions of the diffusion model to guide it towards generating images more aligned with a given class. The score (e.g., the output) of the diffusion model can be provided according to Equation 1: (1) in It is for a given image and class The modified noise prediction; It is a diffusion model in the image and its class Initial noise prediction (e.g., output or score) when both conditions are met; It is a weight or scaling factor that determines the degree to which the gradient of the classifier model affects the final noise prediction; Indicates a specific time step The noise level or standard deviation at that location; It is relative to the image The gradient; and This indicates the classifier's ability to predict images. class Confidence level at time (e.g., class) Given image The logarithm of the conditional probability, such as through (Predicted by the parameterized classifier model).

[0057] In the example, the second term of Equation 1 for classifier-free guidance can be the scaling difference between the conditional model score and the unconditional model score, and can be provided by Equation 2: (2) in It is relative to the image The gradient; This indicates the classifier's ability to predict images. class Confidence level (class) Given image The logarithm of the conditional probability, such as through (Predicted by the parameterized classifier model) It is a scaling factor, such as a specific time step. The noise level or standard deviation at that location; It is a diffusion model in the image and its class Initial noise prediction (e.g., output or score) when both are conditions; and It is an image The unconditional fraction of the diffusion model.

[0058] In one example, the "modified" score of the classifierless diffusion model is related to Equation 3: (3) in It is for a given image and class The modified noise prediction incorporates guidance from the classifier model; It is a weight or scaling factor that determines the degree to which the gradient of the classifier model affects the final noise prediction; It is a diffusion model in the image and its class Initial noise prediction (output or score) when both conditions are met; and It is an image The unconditional fraction of the diffusion model.

[0059] In the example, Equation 4 relates to the segment-free modified score used for denoising: (4) in It is for a given image and class The modified noise prediction incorporates guidance from the classifier model; It is a diffusion model in the image and its class Initial noise prediction (output) when both conditions are met; and The cross-attention weights are computed by running the model on its initial prompts, but at the i-th cross-attention module and for the j-th block, the cross-attention weights are... Modify according to Equation 5: (5) The maximum attention weight (Excluding the first key or BoT terminology) is identified and multiplied by a scalar (e.g., -a) to become .

[0060] In one example, Algorithm 1 involves performing a partition-free instruction: _______________________________________________________________ Algorithm 1: No partitioning guidance _______________________________________________________________ 01:

[0061] 02: for t = T , · · · , 1 do

[0062] 03: if t ≥ T - t s

[0063] #compute classifier-free score

[0064] 04:

[0065] 05: else

[0066] # compute segmentation-free score

[0067] 06:

[0068] # sample z t-1

[0069] 07: if t > 1

[0070] 08:

[0071] 09: else

[0072] 10:

[0073] 11: return x = z 0

[0074] _______________________________________________________________

[0075] Based on having a sequence from λ1 to λ T The model for the noise level of the logarithmic SNR scheduling indicates that the noise level starts low at λ1 and moves towards λ. T Increase, among which It is a classifier-free guided strength (e.g., 7.5), t s is the number of iterations using the classifier-free guidance before switching to the segmentation-free guidance, 'a' is the segmentation-free ratio (e.g., 10.0), and 'c' is the cue text embedding, such as the CLIP text embedding. It is an undivided guide strength (e.g., 2.5). These are latent vector samples from a standard normal distribution, where t is the diffusion time step. These are modified scores used for guidance. This is the initial score output by the diffusion model. It is a potential sample at time step t-1. It is the mean prediction of the diffusion model, and It is the covariance matrix used to determine noise characteristics. This is the final output. It is a potential sample at time step t=1. It is the standard deviation of the noise from the scheduler at t=1. It is the modified score at t=1. It is the standard deviation of the positive process noise at t=1. The diffusion model is for image patches. And the initial unmodified score output by c is indicated. and It is done by using an empty prompt The score is transmitted based on the diffusion model.

[0076] To evaluate images generated according to the proposed segmentation-free guidance technique, a subset of cues and corresponding images can be sampled from a validation set (such as the COCO-30K evaluation set). Evaluating all cues (e.g., the 30,000 cues in the COCO-30K validation set) would likely require an impractical number of manual evaluations. Therefore, this disclosure provides an evaluation method for selecting a representative and reasonable subset of cues while keeping the number of manual evaluations manageable.

[0077] Therefore, the diversity of a randomly sampled subset of prompts can first be quantified by calculating the Fréchet distance between the CLIP text codes of the prompt subset and the CLIP text codes of the full validation set. The Fréchet distance is a measure used to quantify the similarity between the text codes of two sets of prompts. Based on the analysis of Fréchet distances for different sample subset sizes, a subset size (such as 5,000 prompts) can be used as a subset that provides a sufficiently diverse sample size to approximate the diversity of the full validation set.

[0078] Next, an image can be generated for each of the prompts in the sampled subset using classifier-free guidance. The generated images can be ranked based on their CLIP scores, which provides an indication of model performance. Prompts can be selected from the 90th, 50th, and 10th percentiles of the ranking to represent high, medium, and low performance prompts, respectively. Selecting prompts from these performance percentiles provides a reasonable sample of the model's capabilities. The selected prompts can be reduced to a final subset of prompts (e.g., 150 prompts) for human evaluation. According to the examples disclosed herein, the final subset of prompts can be evaluated by human evaluators, each rating the image pairs (e.g., images generated by classifier-free guidance techniques and images generated by segmentation-free guidance techniques). Human evaluators can select one rating from five ratings, which indicate the preference between the segmentation-free guidance method of this disclosure and classifier-free guidance methods known in the art. The five ratings can be, for example, much better, slightly better, no preference, slightly worse, and much worse. Thus, this subjective human evaluation allows for a comparison of the two guidance methods in terms of image quality based on the matched text prompts.

[0079] Figure 6A An example interface 600A is depicted, allowing a user to adjust the importance of different parts of a text prompt. Within this interface, a user can select a specific element, such as a word or phrase, from a text prompt 104 displayed on a user interface 102. For example, the user has selected the word "dog" from the text prompt 104. Following this selection, a context menu 604 pops up on the user interface 102. This context menu 604 provides options to change the semantic importance of the selected word or phrase (e.g., corresponding to weight or guidance strength), such as influencing how the concept will be emphasized during the image generation process. It provides the user with a direct way to make localized improvements and fine-tune aspects of interest. For example, as shown, selecting a larger version of the word or phrase may imply relatively greater semantic importance, while selecting a smaller version may imply relatively less semantic importance.

[0080] Figure 6BAnother example interface 600B is presented, which displays the result on user interface 102 after generating image 608. Image 608 reflects the element selected by the user from text prompt 104 (in this example, the word "dog" identified as element 602), which is assigned its specified semantic importance (represented as 606). In context menu 604, the user can determine the semantic importance by selecting the representation size of the selected element. Thus, the resulting image 608 illustrates improved or reduced precision in the region connected to the selected element, such as depicting the dog with more or less detail depending on the selected semantic importance level. In some respects, interface 600B may display the transition from a noisy image to output image 608, or from an output image associated with a first semantic importance of one or more words to an output image associated with a second semantic importance of one or more words, such as a video transition showing frames that transform (e.g., morphing) between output images.

[0081] like Figure 6C The depicted alternative interface 600C allows users to precisely adjust the semantic importance of individual words or elements (such as words or phrases). Within interface 600C, the user can select elements, such as words or phrases, from text prompts 104 displayed on user interface 102. For example, as depicted in interface 600C, the user has selected the word "dog" from text prompt 104. Upon making this selection, user interface 102 displays a weight menu 610. This weight menu 610 is characterized by a dial 612, which the user can select or rotate to set the semantic importance (e.g., emphasis weight) of their selected word or phrase. Figure 6D As depicted, the user has set an increased weight indicated by a value 616 of "10". Therefore, interface 600D presents a newly generated image 618, which is influenced by the user-defined weight from menu 610. The final image 618 may exhibit enhanced clarity in relation to the emphasized concept or element 602, such as depicting a higher-quality dog. In some respects, interface 600D may display a transition from a noisy image to the output image 618, or a transition from an output image associated with a first semantic importance of one or more words to an output image associated with a second semantic importance of one or more words, such as a video transition showing frames that transform (e.g., morphing) between output images.

[0082] Figure 6E The display allows the user to change the interface 600E of the text prompt 104 by selecting a concept from the image 620 using spatial selection 622. By doing so, a portion of the text prompt 104 associated with the element 624 can be adjusted, thereby indicating a change in its associated semantic importance. For example... Figure 6FAs shown in interface 600F, if the user makes the same spatial selection 622 again, that part of the text tooltip 104 linked to element 624 can be adjusted again to indicate a further change in its relevant semantic importance.

[0083] Figure 6G The interface 600G displays changes to the visual features of the text prompt 104 based on semantic importance tailored to specific elements of the prompt. For example, the visual aspects of elements 628 and 630 are adjusted before or after the creation of image 626. Examples of such adjustments may include, but are not limited to, bolding, underlining, highlighting, changing the font, etc. These adjustments may stem from user changes or selections made to the semantic importance of elements linked to text prompt 104. In some respects, the interface 600G may display a transition from a noisy image to output image 626, or a transition from an output image associated with a first semantic importance of one or more words to an output image associated with a second semantic importance of one or more words, such as a video transition showing frames that transition (e.g., morphing) between output images.

[0084] like Figure 6H The depicted alternative interface 600H allows users to precisely adjust the semantic importance of individual words or elements (such as words or phrases). Within interface 600H, the user can select elements, such as words or phrases, from text prompts 104 displayed on user interface 102. For example, as depicted in interface 600H, the user has selected the word "dog" from text prompt 104. Upon making this selection, user interface 102 displays a slider 632. This slider 632 is a graphical control element that the user can use to set the semantic importance (e.g., emphasis weight) of their selected word or phrase by moving an indicator 634. In some cases, the user can also click a point on the slider to change the settings to adjust the semantic importance (e.g., emphasis weight) of their selected word or phrase. Figure 6I As depicted, the user has set an increased weight indicated by a value 636 of "16". Therefore, interface 600I presents a newly generated image 638, which is influenced by the user-defined weight from slider 632. The final image 638 may exhibit enhanced clarity in relation to the emphasized concept or element 602, such as depicting a higher-quality dog. In some respects, interface 600I may display a transition from a noisy image to the output image 638, or a transition from an output image associated with a first semantic importance of one or more words to an output image associated with a second semantic importance of one or more words, such as a video transition showing frames that transform (e.g., morphing) between output images.

[0085] Figure 7 An example is shown for using the guidance module 330. j -330j+n and / or guidance module 402 j -402 j+n An example computational flow for generating cross-attention features between two datasets (such as sequences 702 and 704 as examples). The first sequence 702 represents, for example, text embedding 316. The second sequence 704 represents, for example, block 408. Cross-attention is applied to sequences 702 and 704 to indicate how much each spatial region of the image (e.g., block 408) is associated with different words in the text prompt (e.g., specific terms in text embedding 316). As shown, value weights 706 are applied to the first sequence 702 to transform its features into value sequence 718. Key weights 708 are applied to the first sequence 702 to transform its features into key sequence 712. Query weights 710 are applied to the second sequence 704 to obtain query sequence 714. Matrix multiplication is used, for example, to compare key sequence 712 and query sequence 714 to generate attention matrix 716. In some respects, this provides attention scores representing the relevance between specific parts of the two sequences 702 and 704. Attention matrix 716 is applied to value sequence 718 to generate cross-attention feature sequence 720, which represents an aggregation of relevant features from first sequence 702 based on second sequence 704. In some respects, this can be as described above regarding... Figures 3 to 5 Modify the attention matrix 716 (or the score) as described.

[0086] Some aspects described in this paper can be implemented, at least in part, using some form of artificial intelligence (AI), such as the process of using machine learning (ML) models to infer or predict output data based on input data. Example ML models may include mathematical representations of one or more relationships between various objects to provide outputs representing one or more predictions or inferences. Once an ML model has been trained, it can be deployed to process data that is all or part of the training data and provide outputs representing one or more predictions or inferences based on the input data.

[0087] Machine learning (ML) can often be characterized by a learning type that generates a specific type of learning model that performs a particular type of task. For example, different types of machine learning include supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.

[0088] Supervised learning algorithms typically model the relationships and dependencies between input features (e.g., feature vectors) and one or more target outputs. Supervised learning uses labeled training data, which consists of data including one or more inputs and a desired output. Supervised learning can be used to train models to perform tasks such as classification (where the goal is to predict discrete values) or regression (where the goal is to predict continuous values). Some example supervised learning algorithms include nearest neighbor, Naive Bayes, decision trees, linear regression, support vector machines (SVM), and artificial neural networks (ANN).

[0089] Unsupervised learning algorithms process unlabeled input data and train models that take the input and transform it into output to solve real-world problems. Examples of unsupervised learning tasks are clustering (where the model's output can be cluster labels), dimensionality reduction (where the model's output is an output feature vector with fewer features than the input feature vector), and outlier detection (where the model's output is a value indicating how the input differs from typical examples in the dataset). An example unsupervised learning algorithm is k-means.

[0090] Semi-supervised learning algorithms process datasets containing both labeled and unlabeled examples, where the number of unlabeled examples is typically much greater than the number of labeled examples. However, the goal of semi-supervised learning is to achieve the goals of supervised learning. Typically, a semi-supervised model includes a model trained to generate pseudo-labels for the unlabeled data, which is then combined with the labeled data to train a second classifier that utilizes a larger amount of overall training data to improve task performance.

[0091] Reinforcement learning algorithms use observations gathered by an agent from its interactions with the environment to take actions that maximize reward or minimize risk. Reinforcement learning is a continuous and iterative process in which an agent learns from its experience with the environment until it has explored, for example, the full range of possible states. An example type of reinforcement learning algorithm is adversarial networks. Reinforcement learning can be particularly beneficial when used to improve or attempt to optimize the behavior of models deployed in dynamically changing environments, such as wireless communication networks.

[0092] ML models can be deployed in one or more devices (e.g., network entities such as base stations and / or user equipment) to support various wired and / or wireless communication aspects of a communication system. For example, ML models can be trained to identify patterns and relationships in data corresponding to networks, devices, air interfaces, etc. ML models can improve operations associated with one or more aspects, such as transceiver circuitry control, frequency synchronization, timing synchronization, channel state estimation, channel equalization, channel state feedback, modulation, demodulation, device location, transceiver tuning, beamforming, signal decoding / decoding, network routing, load balancing, and power saving (to name just a few). AI-enhanced transceiver circuitry control may include, for example, filter tuning, transmit power control, gain control (including automatic gain control), phase control, power management, etc.

[0093] The aspects described herein can be used to describe technical solutions for performing certain tasks and various technical problems by applying specific types of ML models (such as ANNs or CNNs). However, it should be understood that other types of ML models may be used as a complement or alternative to ANNs or CNNs. Therefore, unless explicitly stated otherwise, the topic of ML models is not necessarily intended to be limited to ANN or CNN solutions. Furthermore, it should be understood that, unless otherwise specifically stated, terms such as “AI model,” “ML model,” “AI / ML model,” and “trained ML model” are intended to be used interchangeably.

[0094] Figure 8 This is a diagram illustrating an example AI architecture 800 that can be used to implement one or more ML models. As illustrated, architecture 800 includes multiple logical entities, such as a model training host 802, a model inference host 804, a data source 806, and an agent 808. The AI ​​architecture can be used in any of the various use cases of wireless communication, such as those listed above.

[0095] In architecture 800, the model inference host 804 is configured to run an ML model based on inference data 812 provided by data source 806. The model inference host 804 may generate an output 814 (e.g., a prediction or inference, such as discrete or continuous values) based on the inference data 812, and then provide it as input to an agent 808. The type of agent 808 may depend on the type of task performed by the model inference host 804, the type of inference data 812 provided to the model inference host 804, and / or the type of output 814 generated by the model inference host 804.

[0096] Data source 806 can be configured to collect data that can be used as training data 816 for training an ML model or as inference data 812 for feeding ML model inference operations. Specifically, data source 806 can collect data from any of various entities (e.g., image data sources, text prompts, etc.), which may include the subject of action 810, and provide the collected data to model training host 802 for ML model training. In some examples, if the output 814 provided to agent 808 is inaccurate (or below an accuracy threshold), model training host 802 can determine, for example, to modify or retrain the ML model used by model inference host 804 via ML model deployment / update.

[0097] In some respects, the model training host 802 may be deployed at the same or a different entity as the entity that deploys the model inference host 804, or together with that entity. For example, to offload model training processing that might affect the performance of the model inference host 804, the model training host 802 may be deployed at a model server, as further described herein. Furthermore, in some cases, training and / or inference may be distributed across devices in a decentralized or federated manner.

[0098] Figure 9 This is an exemplary block diagram of an example artificial neural network (ANN) 900.

[0099] ANN 900 may receive input data 906, which may include one or more bits of data 902, preprocessed data (optionally) output from preprocessor 904, or some combination thereof. Here, data 902 may include training data, validation data, application-related data, etc., depending on the stage of development and / or deployment of ANN 900. In some other embodiments, preprocessor 904 may be included within ANN 900. Preprocessor 904 may, for example, process all or part of data 902, which may result in some data in data 902 being changed, replaced, deleted, etc. In some embodiments, preprocessor 904 may add additional data to data 902.

[0100] ANN 900 includes at least one first layer 908 of artificial neurons 910 to process input data 906 and provide the resulting first layer output data to at least a portion of at least one second layer 914 via edge 912. The second layer 914 processes the data received via edge 912 and provides the second layer output data to at least a portion of at least one third layer 918 via edge 916. The third layer 918 processes the data received via edge 916 and provides the third layer output data to at least a portion of a final layer 922 comprising one or more neurons via edge 920 to provide output data 924. All or part of the output data 924 may be further processed in some way by (optionally) a post-processor 926. Thus, in some examples, ANN 900 may provide output data 928 based on output data 924, post-processed data output from post-processor 926, or some combination thereof. In some other embodiments, post-processor 926 may be included within ANN 900. Post-processor 926 may, for example, process all or part of the output data 924, which may result in output data 928 being at least partially different from output data 924, for example, due to data being altered, replaced, deleted, etc. In some embodiments, post-processor 926 may be configured to add additional data to output data 924. In this example, the second layer 914 and the third layer 918 represent intermediate or hidden layers that can be arranged in a hierarchical or other similar structure. Although not explicitly shown, one or more additional intermediate layers may exist between the second layer 914 and the third layer 918.

[0101] The structure and training of the artificial neurons 910 in each layer can be customized to meet the specific requirements of the application. Within a given layer of an ANN, some or all of the neurons can be configured to process the information provided to that layer and output corresponding transformation information from that layer. For example, the transformation information from a layer can represent a weighted sum of input information associated with or based on a nonlinear activation function or other activation functions used to “activate” the artificial neurons in the next layer. Artificial neurons in such layers can be activated by or in response to weights and biases that can be adjusted during the training process. The weights of various artificial neurons can act as parameters controlling the strength of connections between layers or between artificial neurons, while the biases can act as parameters controlling the direction of connections between layers or between artificial neurons. Activation functions can select or determine whether an artificial neuron sends its output to the next layer in response to the data received by the artificial neuron. Different activation functions can be used to model different types of nonlinear relationships. By introducing nonlinearity into the ML model, activation functions allow the ML model to “learn” complex patterns and relationships in the input data. Some examples of non-exhaustive activation functions include linear functions, binary step functions, sigmoid, tanh, ReLU and its variants, exponential linear unit (ELU), Swish, Softmax, etc.

[0102] Design tools (such as computer applications and programs) can be used to select the appropriate structure of the ANN 900, multiple layers, and multiple artificial neurons in each layer, as well as to select activation functions, loss functions, training procedures, etc. Once the initial model is designed, it can be trained using training data. Training data may include one or more datasets within which the ANN 900 can detect, identify, label, or discern patterns. Training data can represent various types of information, including written, visual, audio, environmental context, operational attributes, etc. During training, the parameters of the artificial neurons 910 can be changed, such as to minimize or otherwise reduce the loss function or cost function. The training process can be repeated multiple times to fine-tune the ANN 900 in each iteration.

[0103] Various ANN model architectures are available for consideration. For example, in a feedforward ANN architecture, each artificial neuron 910 in a layer receives information from the previous layer and similarly generates information for the next layer. In a convolutional ANN architecture, some layers can be organized as filters that extract features from data (e.g., training data and / or input data). In a recursive ANN architecture, some layers may have connections that allow data to be processed across time, such as for processing information with temporal structure, such as time series data prediction.

[0104] In the autoencoder ANN architecture, compact representations of data can be processed, and models can be trained to predict or potentially reconstruct the original data from a reduced set of features. The autoencoder ANN architecture can be used for tasks related to dimensionality reduction and data compression.

[0105] Generative adversarial network (GAN) architectures can include generator ANNs and discriminator ANNs trained to compete with each other. GANs are ANN architectures that can be used for tasks related to generating synthetic data or improving the performance of other models.

[0106] Transformer ANN structures utilize attention mechanisms that enable models to process input sequences in a parallel and efficient manner. Attention mechanisms allow the model to focus on different parts of the input sequence at different times. These attention mechanisms can be implemented using a series of layers called attention layers to compute, compute, determine, or select a weighted sum of input features based on the similarity between different elements of the input sequence. A transformer ANN structure may include a series of feedforward ANN layers that learn a non-linear relationship between the input and output sequences. The output of the transformer ANN structure can be obtained by applying a linear transformation to the output of the final attention layer. Transformer ANN structures are particularly useful for tasks involving sequence modeling or other similar processing.

[0107] Another example type of ANN structure is a model with one or more invertible layers. This type of model can be inverted or "unfolded" to reveal the input data used to generate the output of the layers.

[0108] Other examples of ANN model architectures include fully connected neural networks (FCNN) and long short-term memory (LSTM) networks.

[0109] ANN 900 or other ML models can be implemented in various types of processing circuits, as well as their memory and applicable instructions, for example, as described in this paper. Figures 2 to 9 As described. For example, the model can be implemented using general-purpose hardware circuitry such as one or more central processing units (CPUs) and one or more graphics processing units (GPUs). One or more ML accelerators, such as tensor processing units (TPUs), embedded neural processing units (eNPUs), or other dedicated processors, and / or field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc., can also be used to develop ANN models. Various programming tools can be used to develop ANN models.

[0110] There are deployable ML models (such as...) Figure 9 Various model training techniques and processes used at some point before or after ANN 900.

[0111] As part of the model development process, information in a suitable form of training data may be collected or otherwise created for training the ML model accordingly. Once the ML model has been trained with the training data, its performance can be evaluated. In some scenarios, evaluation / validation tests may be conducted using a validation dataset, which may include data not present in the training data, to compare the model's performance against a baseline or other benchmark information. If the model's performance is deemed unsatisfactory, fine-tuning the model may be beneficial, for example, by changing its architecture, retraining it on the data, or using different optimization techniques. Once the model's performance is deemed satisfactory, it can be deployed accordingly. In some cases, the model may be updated in some way, for example, all or part of the model may be changed or replaced, or it may undergo further training, to name just a few examples.

[0112] As ANN (such as Figure 9 As part of the training process of an ANN (Advanced Neural Network 900), parameters affecting the function of artificial neurons and layers can be adjusted. For example, backpropagation can be used to train an ANN by iteratively adjusting the weights and / or biases of certain artificial neurons associated with the error between the model's predicted output and the expected output, which may be known or otherwise considered acceptable. Backpropagation may include forward passes, loss functions, backward passes, and parameter updates that can be performed during training iterations. This process can be repeated a certain number of iterations for each training dataset until the weights of the artificial neurons / layers are sufficiently adjusted.

[0113] Backpropagation, associated with the loss function, measures how well a model can predict the desired output for a given input. Optimization algorithms can be used during training to adjust weights and / or biases to reduce or minimize the loss function, thus improving model performance. Various optimization algorithms exist that can be used in conjunction with backpropagation or other training techniques. Some initial examples include gradient descent-based and stochastic gradient descent-based optimization algorithms. Stochastic gradient descent (or ascent) can be used to adjust weights / biases to minimize or otherwise reduce the loss function. Mini-batch gradient descent, a variant of gradient descent, involves updating weights / biases using mini-batch training data instead of the entire dataset. Momentum techniques can accelerate the optimization process by adding momentum terms to update or otherwise influence certain weights / biases.

[0114] Adaptive learning rate techniques adjust the learning rate of an optimization algorithm that is associated with one or more characteristics of the training data. Batch normalization techniques can be used to normalize the input to a model in order to stabilize the training process and potentially improve the model's performance.

[0115] The "drop-out" technique can be used to randomly discard some of the artificial neurons from the model during the training process, for example, to reduce overfitting and potentially improve the model's generalization.

[0116] The “early stopping” technique can be used to stop an ongoing training process early, such as when the performance of a model using a validation dataset begins to degrade.

[0117] Another example technique includes data augmentation, which generates additional training data by applying transformations to all or part of the training information.

[0118] Transfer learning techniques can be used, which involve using a pre-trained model as a starting point for training a new model. This can be useful when training data is limited or when there are multiple tasks that are related to each other.

[0119] Multi-task learning techniques can be used, which involve training a model to perform multiple tasks simultaneously to potentially improve the model's performance on one or more tasks. In some cases, hyperparameters can be input and applied during the training process.

[0120] Another example technique that can be useful for ML models is some form of "pruning." Pruning techniques, which can be performed during the training process or after the model has been trained, involve removing unnecessary (e.g., because they have no effect on the output), less necessary (e.g., because their effect on the output is negligible), or potentially redundant features from the model. In some cases, pruning techniques can reduce the complexity of the model or improve its efficiency without compromising its expected performance.

[0121] Pruning techniques can be particularly useful in the context of wireless communication, where available resources, such as power and bandwidth, may be limited. Some example pruning techniques include weight pruning, neuron pruning, layer pruning, structural pruning, and dynamic pruning. Pruning techniques can, for example, reduce the amount of data corresponding to a model that may need to be transmitted or stored.

[0122] Weight pruning techniques may involve removing weights from a model. Neuron pruning techniques may involve removing neurons from a model. Layer pruning techniques may involve removing layers from a model. Structural pruning techniques may involve removing connections between neurons in a model. Dynamic pruning techniques may involve adjusting the pruning strategy of a model to adapt to one or more characteristics of the data or environment. For example, in some wireless communication devices, dynamic pruning techniques may be more aggressive in pruning models used in low-power or low-bandwidth environments and less aggressive in pruning models used in high-power or high-bandwidth environments. In some aspects, pruning techniques may also be applied to training data, for example, to remove outliers. In some specific implementations, preprocessing techniques on all or part of the training dataset can improve model performance or facilitate faster model convergence. For example, training data may be preprocessed to modify or remove unnecessary, irrelevant, incorrect, or other identifiable data. Such preprocessing of training data may, for example, lead to a reduction in potential overfitting or otherwise improve the performance of the trained model.

[0123] One or more of the example training techniques presented above can be used as part of the training process. As shown above, some example training processes that can be used to train ML models include supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning techniques.

[0124] Decentralized, distributed, or shared learning, such as federated learning, enables training on data distributed across multiple devices or organizations without centralizing the data or training. Federated learning can be particularly useful in data-sensitive or privacy-constrained situations, or when centralizing data is impractical, inefficient, or costly. For example, in the context of wireless communications, federated learning can be used to improve performance by allowing ML models to be trained on data collected from a wide range of devices and environments. For instance, ML models can be trained on data collected from a large number of wireless devices in a network, such as distributed wireless communication nodes, smartphones, or Internet of Things (IoT) devices, to improve network performance and efficiency. With federated learning, a device can receive a full or partial copy of the model and perform local training on this copy using locally available training data. Such a device can then provide updates about the locally trained model (e.g., trainable parameter gradients) to one or more other devices, such as network entities or servers, where updates from other similar devices can be aggregated and used to provide updates to shared models, etc. The federated learning process can be iteratively repeated until all or part of the model achieves a satisfactory level of performance. Federation learning enables devices to protect the privacy and security of local data while supporting collaboration on training and updating shared models, in whole or in part.

[0125] Figure 10An example method 1000 for generating an output image based on text prompts is described. In one aspect, method 1000 can be derived from documents such as those concerning... Figures 1 to 6G This can be achieved using any image generation system described.

[0126] Method 1000 begins at box 1002, where a text prompt is obtained.

[0127] Method 1000 continues to box 1004, where the text prompt is encoded into multiple conditional tokens.

[0128] Method 1000 continues to box 1006 and, for each of one or more blocks in the potential image representation, performs the following operations: computes corresponding cross-attention weights for the multiple conditional terms based on the block as the query and the multiple conditional terms as the key; and modifies the maximum cross-attention weight among the corresponding multiple cross-attention weights to generate modified corresponding multiple cross-attention weights. In some aspects, modifying the maximum cross-attention weight among the corresponding multiple cross-attention weights includes decreasing the maximum cross-attention weight. In some aspects, modifying the maximum cross-attention weight among the corresponding multiple cross-attention weights includes multiplying the maximum cross-attention weight by a negative scalar value. In some aspects, the negative scalar value is received as a user-specified parameter. In some aspects, modifying the maximum cross-attention weight among the corresponding multiple cross-attention weights includes setting the maximum cross-attention weight to zero. In some aspects, the iteration of performing denoising includes: generating a first output based on a latent image representation and corresponding plurality of cross-attention weights for each of one or more blocks in the latent image representation; generating a second output based on the latent image representation and modified corresponding plurality of cross-attention weights for each of one or more blocks in the latent image representation; and subtracting the second output from the first output to obtain a modified latent image representation. In some aspects, the iteration of performing denoising includes: generating a modified latent image representation based on the latent image representation and modified corresponding plurality of cross-attention weights for each of one or more blocks in the latent image representation.

[0129] Method 1000 continues to box 1008, where a denoising iteration is performed for each of one or more blocks using the corresponding modified multiple cross-attention weights to obtain a modified potential image representation.

[0130] Method 1000 continues to box 1010, where an output image is generated based on the modified latent representation. In some respects, the output image is displayed via a display.

[0131] In some respects, method 1000 may include performing one or more initial denoising iterations with classifier-free guidance before performing denoising iterations for each of the one or more blocks using the modified corresponding multiple cross-attention weights.

[0132] In some aspects, method 1000 may include a user interface configured to receive input indicating the intensity of emphasis associated with one or more words of a text prompt, wherein the intensity of emphasis controls the amount by which the maximum cross-attention weight among a plurality of corresponding cross-attention weights is modified. In some aspects, the user interface includes one or more interface elements configured to receive input, the one or more interface elements including at least one of a slider, numeric input, or keyword highlighting.

[0133] In some aspects, method 1000 may include, for each of one or more blocks of a modified latent image representation: calculating a corresponding second plurality of cross-attention weights corresponding to a plurality of conditionalized terms based on a block as another query and a plurality of conditionalized terms as another key; and modifying another maximum cross-attention weight among the corresponding second plurality of cross-attention weights to generate another modified corresponding plurality of second cross-attention weights; and performing a second denoising iteration for each of one or more blocks of the modified latent image representation using the modified corresponding plurality of second cross-attention weights to obtain a second modified latent image representation, wherein generating an output image based on the modified latent representation includes using a decoder to decode the second modified latent image representation to generate the output image.

[0134] In some aspects, method 1000 may include modulating one or more carrier signals with data indicating an output image. In some aspects, one or more antennas are configured to transmit one or more carrier signals to the device.

[0135] It should be noted that Figure 10 This is merely one example of a method, and other methods that include fewer, additional, or alternative steps may be consistent with this disclosure.

[0136] Figure 11 An example method 1100 for generating an output image based on text prompts is described. In one aspect, method 1100 can be derived from documents such as those concerning... Figures 1 to 6G This can be achieved using any image generation system described.

[0137] Method 1100 begins at box 1102, where a text prompt is received.

[0138] Method 1100 continues to box 1104, which provides a user interface including one or more input elements associated with one or more words of the text prompt.

[0139] Method 1100 continues to box 1106, wherein input corresponding to at least one of one or more input elements is received, the input indicating the semantic importance of each of at least one of one or more words associated with the at least one of the one or more input elements. In some aspects, the first input element among the one or more input elements is a slider element configured to increase or decrease the importance of the first word among the one or more words. In some aspects, the first input element among the one or more input elements is a turntable element configured to increase or decrease the importance of the first word among the one or more words.

[0140] Method 1100 continues to box 1108, where an output image is generated based on text prompts and input. In some aspects, generating the output image involves using text prompts and input to generate an output image as input to a generative artificial intelligence (AI) model. In some aspects, the output image highlights one or more objects associated with at least one word in one or more words that is indicated to have higher semantic importance compared to other words in one or more words.

[0141] In some respects, method 1100 includes displaying a user interface at a display and displaying an output image at a display.

[0142] In some aspects, method 1100 includes modifying the appearance of at least one word among one or more words based on indicated semantic importance. In some aspects, modifying the appearance of at least one word among one or more words includes highlighting at least one word among one or more words.

[0143] In some aspects, method 1100 includes displaying a first image associated with a text prompt before receiving input; and displaying an output image after receiving input. In some aspects, method 1100 includes displaying a transition from the first image to the output image. In some aspects, the output image emphasizes one or more objects associated with at least one of one or more words, compared to one or more objects in the first image. In some aspects, the method includes using the text prompt to generate the first image as input to a generative artificial intelligence (AI) model, wherein generating the output image includes using the text prompt and the input to generate the output image as input to the generative AI model.

[0144] In some aspects, method 1100 includes: encoding text prompts into a plurality of conditionalized lexical units; for each of one or more blocks in a latent image representation: computing a plurality of corresponding cross-attention weights corresponding to the plurality of conditionalized lexical units based on the block as a query and the plurality of conditionalized lexical units as keys; modifying the maximum cross-attention weight among the plurality of corresponding cross-attention weights to generate modified plurality of corresponding cross-attention weights; and performing a denoising iteration using the modified plurality of corresponding cross-attention weights for each of the one or more blocks to generate an output image. In some aspects, at least one of the plurality of cross-attention weights is modified based on indicated semantic importance.

[0145] It should be noted that Figure 11 This is merely one example of a method, and other methods that include fewer, additional, or alternative steps may be consistent with this disclosure.

[0146] Example processing system for generating output images

[0147] Figure 12 Various aspects of the example processing system 1200 are described.

[0148] Processor 1200 may include a processing system 1202 having one or more processors 1220. The one or more processors 1220 are coupled to a computer-readable medium / memory 1230 via a bus 1206. In some aspects, the computer-readable medium / memory 1230 is configured to store instructions (e.g., computer-executable code) that, when executed by the one or more processors 1220, cause the one or more processors 1220 to perform actions related to… Figure 10 The described method 1000 or any aspect thereof, including regarding Figure 10 Any additional steps or sub-steps described. The processing system 1200 may also include one or more of a display (e.g., configured to display a user interface), a modem (e.g., configured to modulate data on a carrier, such as image data), and one or more antennas (e.g., configured to transmit and / or receive data, such as in conjunction with a modem).

[0149] In the depicted example, computer-readable medium / memory 1230 stores code 1231 (e.g., executable instructions) for obtaining text cues, code 1232 for encoding the text cues, code 1233 for calculating and modifying cross-attention weights, code 1234 for performing denoising, and code 1235 for generating an output image. Processing of codes 1231-1235 enables processing system 1200 to execute and perform actions related to... Figure 10 The method 1000 described or any aspect thereof.

[0150] One or more processors 1220 include circuitry configured to implement (e.g., execute) code stored in computer-readable medium / memory 1230, including circuitry 1221 for obtaining text cues, circuitry 1222 for encoding text cues, circuitry 1223 for calculating and modifying cross-attention weights, circuitry 1224 for performing denoising, and code 1225 for generating an output image. Processing using circuitry 1221-1225 enables processing system 1200 to execute and perform actions related to... Figure 10 The method 1000 described or any aspect thereof.

[0151] Example processing system for generating output images

[0152] Figure 13 Various aspects of the example processing system 1300 are described.

[0153] Processor 1300 may include a processing system 1302 having one or more processors 1320. The one or more processors 1320 are coupled to a computer-readable medium / memory 1330 via a bus 1306. In some aspects, the computer-readable medium / memory 1330 is configured to store instructions (e.g., computer-executable code) that, when executed by the one or more processors 1320, cause the one or more processors 1320 to perform actions related to… Figure 11 The described method 1100 or any aspect thereof, including regarding Figure 11 Any additional steps or sub-steps described. The processing system 1300 may also include one or more of a display (e.g., configured to display a user interface), a modem (e.g., configured to modulate data, such as image data, on a carrier), and one or more antennas (e.g., configured to transmit and / or receive data, such as in conjunction with a modem).

[0154] In the depicted example, computer-readable medium / memory 1330 stores code 1331 (e.g., executable instructions) for receiving text prompts, code 1332 for providing a user interface, code 1333 for receiving input, and code 1334 for generating an output image. Processing of codes 1331-1334 enables processing system 1300 to execute and perform actions related to... Figure 11 The method 1100 described or any aspect thereof.

[0155] One or more processors 1320 include circuitry configured to implement (e.g., execute) code stored in computer-readable medium / memory 1330, including circuitry 1321 for receiving text prompts, circuitry 1322 for providing a user interface, circuitry 1323 for receiving input, and circuitry 1324 for generating an output image. Processing using circuitry 1321-1324 enables processing system 1300 to execute and perform actions related to... Figure 11 The method 1100 described or any aspect thereof.

[0156] Example Terms

[0157] Specific implementation examples are described in the following numbered clauses: Clause 1: A method for generating an output image based on a text prompt, the method comprising: obtaining the text prompt; encoding the text prompt into a plurality of conditionalized lexical units; for each of one or more blocks of a latent image representation: calculating a plurality of corresponding cross-attention weights corresponding to the plurality of conditionalized lexical units based on the block as a query and the plurality of conditionalized lexical units as keys; and modifying the maximum value of the plurality of corresponding cross-attention weights to generate modified corresponding cross-attention weights; performing a denoising iteration using the modified corresponding cross-attention weights for each of the one or more blocks to obtain a modified latent image representation; and generating the output image based on the modified latent image representation.

[0158] Clause 2: The method according to Clause 1, wherein modifying the maximum cross-attention weight among the respective plurality of cross-attention weights includes reducing the maximum cross-attention weight.

[0159] Clause 3: The method according to any one of Clauses 1 to 2, wherein modifying the maximum cross-attention weight among the respective plurality of cross-attention weights comprises multiplying the maximum cross-attention weight by a negative scalar value.

[0160] Clause 4: The method described in Clause 3 further includes receiving the negative scalar value as a user-specified parameter.

[0161] Clause 5: The method according to any one of Clauses 1 to 4, wherein modifying the maximum cross-attention weight among the respective plurality of cross-attention weights comprises multiplying the maximum cross-attention weight to zero.

[0162] Clause 6: The method according to any one of Clauses 1 to 5 further includes performing one or more initial denoising iterations using classifier-free guidance before performing the denoising iterations for each of the one or more blocks using the modified corresponding plurality of cross-attention weights.

[0163] Clause 7: The method according to any one of Clauses 1 to 6, the method further comprising: displaying a user interface configured to receive input indicating an emphasis intensity associated with one or more words of the text prompt, wherein the emphasis intensity controls the amount by which the maximum cross-attention weight among the respective plurality of cross-attention weights is modified.

[0164] Clause 8: The method according to Clause 7 further comprises: displaying on the user interface one or more interface elements configured to receive the input, the one or more interface elements including at least one of a slider, a numeric input, or a keyword highlighting.

[0165] Clause 9: The method according to any one of Clauses 1 to 8, further comprising: for each of one or more blocks of the modified latent image representation: calculating a corresponding second plurality of cross-attention weights corresponding to the plurality of conditionalized terms based on the block as another query and the plurality of conditionalized terms as another key; and modifying another maximum cross-attention weight among the corresponding second plurality of cross-attention weights to generate another modified corresponding plurality of second cross-attention weights; and performing a second denoising iteration for each of the one or more blocks of the modified latent image representation using the modified corresponding plurality of second cross-attention weights to obtain a second modified latent image representation, wherein generating the output image based on the modified latent image representation includes using a decoder to decode the second modified latent image representation to generate the output image.

[0166] Clause 10: The method according to any one of Clauses 1 to 9 further includes displaying the output image at a display.

[0167] Clause 11: The method according to any one of Clauses 1 to 10 further includes modulating one or more carrier signals by a modem using data indicative of the output image.

[0168] Clause 12: The method according to Clause 11 further includes transmitting the one or more carrier signals to the device by one or more antennas.

[0169] Clause 13: The method according to any one of Clauses 1 to 12, wherein performing the denoising iteration comprises: generating a first output based on the latent image representation and the corresponding plurality of cross-attention weights for each of the one or more blocks of the latent image representation; generating a second output based on the latent image representation and the modified corresponding plurality of cross-attention weights for each of the one or more blocks of the latent image representation; and subtracting the second output from the first output to obtain the modified latent image representation.

[0170] Clause 14: The method according to any one of Clauses 1 to 13, wherein performing the denoising iteration comprises generating the modified latent image representation based on the latent image representation and the modified corresponding plurality of cross-attention weights for each of the one or more blocks of the latent image representation.

[0171] Clause 15: A method for generating an output image based on a text prompt, the method comprising: receiving the text prompt; providing a user interface including one or more input elements associated with one or more words of the text prompt; receiving input corresponding to at least one of the one or more input elements, the input indicating the semantic importance of each of the one or more words associated with the at least one of the one or more input elements; and generating the output image based on the text prompt and the input.

[0172] Clause 16: The method according to Clause 15 further includes displaying the user interface at a display; and displaying the output image at the display.

[0173] Clause 17: The method according to any one of Clauses 15 to 16, wherein generating the output image includes using the text prompt and the input to generate the output image as input to a generative artificial intelligence (AI) model.

[0174] Clause 18: The method according to any one of Clauses 15 to 17, wherein the first input element of the one or more input elements is a slider element configured to increase or decrease the importance of the first word among the one or more words.

[0175] Clause 19: The method according to any one of Clauses 15 to 18, wherein the first input element of the one or more input elements is a turntable element configured to increase or decrease the importance of the first word among the one or more words.

[0176] Clause 20: The method according to any one of Clauses 15 to 19 further includes modifying the appearance of at least one word among the one or more words based on the indicated semantic importance.

[0177] Clause 21: The method according to Clause 20, wherein modifying the appearance of at least one word among the one or more words includes highlighting at least one word among the one or more words.

[0178] Clause 22: The method according to any one of Clauses 15 to 21, wherein the output image emphasizes one or more objects associated with the at least one word among the one or more words that is indicated to have higher semantic importance than other words among the one or more words.

[0179] Clause 23: The method according to any one of Clauses 15 to 22, the method further comprising: displaying a first image associated with the text prompt on a display before the input is received; and displaying it on the display after the input is received.

[0180] Clause 24: The method according to Clause 23 further includes displaying a transition from the first image to the output image by the display.

[0181] Clause 25: The method according to Clause 23, wherein the output image emphasizes one or more objects associated with at least one word among the one or more words, compared to the one or more objects in the first image.

[0182] Clause 26: The method according to Clause 23 further includes: using the text prompt to generate the first image as input to a generative artificial intelligence (AI) model, wherein generating the output image includes using the text prompt and the input to generate the output image as input to the generative AI model.

[0183] Clause 27: The method according to any one of Clauses 15 to 26, the method further comprising: encoding the text prompt into a plurality of conditionalized lexical units; for each of one or more blocks of a latent image representation: calculating a plurality of corresponding cross-attention weights corresponding to the plurality of conditionalized lexical units based on the block as a query and the plurality of conditionalized lexical units as keys; modifying the maximum value of the plurality of corresponding cross-attention weights to generate modified plurality of corresponding cross-attention weights; and performing a denoising iteration using the modified plurality of corresponding cross-attention weights for each of the one or more blocks to generate the output image.

[0184] Clause 28: The method according to Clause 27 further includes modifying at least one of the respective plurality of cross-attention weights based on the indicated semantic importance.

[0185] Clause 29: An apparatus comprising: one or more memories (e.g., including executable instructions); and one or more processors coupled to the one or more memories, the one or more processors being configured to (e.g., execute the executable instructions and) (e.g., cause the apparatus to) perform a method according to any one of Clauses 1 to 28.

[0186] Clause 30: An apparatus comprising components for performing the method according to any one of Clauses 1 to 28.

[0187] Clause 31: A non-transitory computer-readable medium storing program code for causing a processing system to perform the steps according to any one of Clauses 1 to 28.

[0188] Clause 32: A computer program product embodied on a computer-readable storage medium, said computer-readable storage medium including code for performing a method according to any one of Clauses 1 to 28.

[0189] Additional Notes

[0190] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein do not limit the scope, applicability, or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, the function and arrangement of the elements discussed may be changed without departing from the scope of this disclosure. Various processes or components may be omitted, substituted, or added as appropriate in various examples. For example, the described methods may be performed in a different order than described, and various actions may be added, omitted, or combined. Furthermore, features described in some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement an apparatus or practice. Moreover, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that complement or replace the various aspects of this disclosure set forth herein. It should be understood that any aspect of this disclosure disclosed herein may be embodied by one or more elements of these claims.

[0191] The various exemplary logic blocks, modules, and circuits described in this disclosure can be implemented or executed using a general-purpose processor, digital signal processor (DSP), ASIC, field-programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic device, discrete hardware component, or any combination thereof designed to perform the functions described herein. While the general-purpose processor may be a microprocessor, in alternative embodiments, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors working in conjunction with a DSP core, a system-on-a-chip (SoC), or any other such configuration.

[0192] As used in this article, the phrase “at least one of the items” refers to any combination of these items, including a single member. As an example, “at least one of a, b, or c” is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbb, cc, and ccc, or any other ordering of a, b, and c).

[0193] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, operation, processing, deduction, investigation, lookup (e.g., searching in a table, database, or other data structure), assertion, and so on. Additionally, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and so on. Furthermore, "determine" can include parsing, selecting, picking, building, and so on.

[0194] As used herein, “coupling” can include “communicationally coupled,” “electrically coupled,” or “physically coupled,” and may also (or alternatively) include any combination thereof. Two devices (or components) may be coupled directly or indirectly (e.g., communicationally coupled, electrically coupled, or physically coupled) via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof). As an illustrative, non-limiting example, two electrically coupled devices (or components) may be included in the same device or in different devices and may be connected via electronics, one or more connectors, or inductive coupling. In a particular embodiment, two communicationally coupled (e.g., electrically connected) devices (or components) may transmit and receive signals (e.g., digital or analog signals) directly or indirectly via one or more wires, buses, networks, etc. As used herein, “direct coupling” can include two devices coupled (e.g., communicationally coupled, electrically coupled, or physically coupled) without intermediate components.

[0195] The methods disclosed herein include one or more actions for implementing the methods. These actions may be interchanged without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of a particular action may be modified without departing from the scope of the claims. Furthermore, the various operations of the methods described above may be performed by any suitable component capable of performing the corresponding function. This component may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors.

[0196] The following claims are not intended to be limited to the aspects shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, unless specifically stated otherwise, reference to the singular form of an element is not intended to mean “one and only one,” but rather “one or more.” Unless specifically stated otherwise, the term “some” means one or more. No element of any claim shall be interpreted in accordance with 35 USC §112(f) unless that element is explicitly stated using the phrase “for a component of.” All structural and functional equivalents of the elements throughout the various aspects described herein that are known to a person of ordinary skill in the art, or will later be known, are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be offered to the public, whether or not such disclosure is explicitly recited in the claims.

Claims

1. An apparatus configured to generate an output image based on text prompts, the apparatus comprising: One or more memories, the one or more memories being configured to store potential image representations; and One or more processors coupled to the one or more memories, the one or more processors being configured to: Obtain the text prompt; The text prompt is encoded into multiple conditional lexical units; For each of the one or more blocks represented by the potential image: Based on the block as the query and the multiple conditional terms as the key, calculate the corresponding multiple cross-attention weights for the multiple conditional terms; as well as Modify the maximum value of the corresponding multiple cross-attention weights to generate the modified corresponding multiple cross-attention weights; For each of the one or more blocks, the modified corresponding multiple cross-attention weights are used to perform denoising iterations to obtain the modified latent image representation; as well as The output image is generated based on the modified latent image representation.

2. The apparatus of claim 1, wherein modifying the maximum cross-attention weight among the respective plurality of cross-attention weights comprises reducing the maximum cross-attention weight.

3. The apparatus of claim 1, wherein modifying the maximum cross-attention weight among the respective plurality of cross-attention weights comprises multiplying the maximum cross-attention weight by a negative scalar value.

4. The apparatus of claim 3, wherein the one or more processors are configured to receive the negative scalar value as a user-specified parameter.

5. The apparatus of claim 1, wherein modifying the maximum cross-attention weight among the respective plurality of cross-attention weights comprises setting the maximum cross-attention weight to zero.

6. The apparatus of claim 1, wherein the one or more processors are configured to perform one or more initial denoising iterations using classifier-free guidance prior to performing the denoising iterations for each of the one or more blocks using the modified corresponding plurality of cross-attention weights.

7. The apparatus of claim 1, further comprising a display coupled to the one or more processors, the display being configured to display a user interface configured to receive input indicating an emphasis intensity associated with one or more words of the text prompt, wherein the emphasis intensity controls the amount by which the maximum cross-attention weight among the respective plurality of cross-attention weights is modified.

8. The apparatus of claim 7, wherein the user interface includes one or more interface elements configured to receive the input, the one or more interface elements including at least one of a slider, a numeric input, or a keyword highlighting.

9. The apparatus of claim 1, wherein the one or more processors are configured to: For each of one or more blocks in the modified potential image representation: The corresponding second plurality of cross-attention weights are calculated based on the block as another query and the plurality of conditionalized terms as another key; as well as Modify another maximum value of the corresponding second plurality of cross-attention weights to generate another modified corresponding plurality of second cross-attention weights; as well as For each of the one or more blocks of the modified latent image representation, a second denoising iteration is performed using the corresponding modified plurality of second cross-attention weights to obtain a second modified latent image representation, wherein generating the output image based on the modified latent image representation includes using a decoder to decode the second modified latent image representation to generate the output image.

10. The apparatus of claim 1, further comprising a display coupled to the one or more processors, the display being configured to display the output image.

11. The apparatus according to claim 1, further comprising: A modem coupled to the one or more processors, the modem being configured to modulate one or more carrier signals with data indicative of the output image.

12. The apparatus of claim 11, further comprising: One or more antennas coupled to the modem, the one or more antennas being configured to transmit the one or more carrier signals to the device.

13. The apparatus of claim 1, wherein, in order to perform the denoising iteration, the one or more processors are configured to: A first output is generated based on the latent image representation and the corresponding plurality of cross-attention weights for each of the one or more blocks in the latent image representation; A second output is generated based on the latent image representation and the modified corresponding plurality of cross-attention weights for each of the one or more blocks of the latent image representation; and Subtract the second output from the first output to obtain the modified potential image representation.

14. The apparatus of claim 1, wherein, in order to perform the denoising iteration, the one or more processors are configured to: The modified latent image representation is generated based on the latent image representation and the modified corresponding multiple cross-attention weights for each of the one or more blocks used in the latent image representation.

15. An apparatus configured to generate an output image based on text prompts, the apparatus comprising: One or more memories, the one or more memories being configured to store the output image; One or more processors coupled to the one or more memories, the one or more processors being configured to: Receive the text prompt; Provide a user interface that includes one or more input elements associated with one or more words in the text prompt; Receive input corresponding to at least one of the one or more input elements, the input indicating the semantic importance of each of the one or more words associated with the at least one of the one or more input elements; as well as The output image is generated based on the text prompt and the input.

16. The apparatus of claim 15, further comprising a display coupled to the one or more processors, the display being configured to: Displaying the user interface; and The output image is displayed.

17. The apparatus of claim 15, wherein, in order to generate the output image, the one or more processors are configured to use the text prompt and the input to generate the output image as input to a generative artificial intelligence (AI) model.

18. The apparatus of claim 15, wherein the first input element of the one or more input elements is a slider element configured to increase or decrease the importance of the first word among the one or more words.

19. The apparatus of claim 15, wherein the first input element of the one or more input elements is a turntable element configured to increase or decrease the importance of the first word among the one or more words.

20. The apparatus of claim 15, wherein the one or more processors are configured to modify the appearance of at least one word among the one or more words based on indicated semantic importance.

21. The apparatus of claim 20, wherein modifying the appearance of at least one word among the one or more words comprises highlighting at least one word among the one or more words.

22. The apparatus of claim 15, wherein the output image emphasizes one or more objects associated with the at least one word among the one or more words that is indicated to have higher semantic importance than other words among the one or more words.

23. The apparatus of claim 15, further comprising a display coupled to the one or more processors, the display being configured to: Before the one or more processors receive the input, a first image associated with the text prompt is displayed; and After the one or more processors receive the input, the output image is displayed.

24. The apparatus of claim 23, wherein the display is configured to: This displays the transformation from the first image to the output image.

25. The apparatus of claim 23, wherein the output image emphasizes one or more objects associated with at least one word among the one or more words, compared to the one or more objects in the first image.

26. The apparatus according to claim 23, wherein: The one or more processors are configured to use the text prompt to generate the first image as input to a generative artificial intelligence (AI) model; and In order to generate the output image, the one or more processors are configured to use the text prompt and the input to generate the output image as input to the generative AI model.

27. The apparatus of claim 15, wherein the one or more processors are configured to: The text prompt is encoded into multiple conditional lexical units; For each of one or more blocks in the potential image representation: Based on the block as the query and the multiple conditional terms as the key, calculate the corresponding multiple cross-attention weights for the multiple conditional terms; as well as Modify the maximum value of the corresponding multiple cross-attention weights to generate the modified corresponding multiple cross-attention weights; as well as For each of the one or more blocks, the modified corresponding multiple cross-attention weights are used to perform denoising iterations to generate the output image.

28. The apparatus of claim 27, wherein at least one of the respective plurality of cross-attention weights is modified based on the indicated semantic importance.

29. A method comprising: Get a text prompt; The text prompt is encoded into multiple conditional lexical units; For each of one or more blocks in the potential image representation: Based on the block as the query and the multiple conditional terms as the key, calculate the corresponding multiple cross-attention weights for the multiple conditional terms; as well as Modify the maximum value of the corresponding multiple cross-attention weights to generate the modified corresponding multiple cross-attention weights; For each of the one or more blocks, the modified corresponding multiple cross-attention weights are used to perform denoising iterations to obtain the modified latent image representation; as well as The output image is generated based on the modified latent representation.

30. A method comprising: Receive text prompts; Provide a user interface that includes one or more input elements associated with one or more words in the text prompt; Receive input corresponding to at least one of the one or more input elements, the input indicating the semantic importance of each of the one or more words associated with the at least one of the one or more input elements; as well as The output image is generated based on the text prompt and the input.

Images