Image generation method

Abstract

To provide an image generation method.SOLUTION: A method includes: receiving input text and layout information; decomposing the input text into one or more global features and one or more local features; coding the one or more global features into a global vector, and coding the one or more local features into a local vector; predicting a global noise from the global vector and an initial image representation using a trained neutral network (NN); predicting an initial local noise from the one or more local vectors and the initial image representation for each local feature by using the trained NN; determining a final local noise for each local feature using the initial local noise and the global noise; synthesizing the predicted global noise and the final local noise into a synthesized noise using the layout information; and removing noise in the initial image representation by using the synthesized noise and providing a next image representation.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to a method for generating an image from an input text and layout information using a neural network, and further to an apparatus for generating an image from an input text and layout information using a neural network. [Background technology]

[0002] Many companies across a variety of industries need specific images for their business to use in advertising, presentations, web content, etc. In the field of computer vision, image synthesis has seen incredible development in recent years, but not without significant computational demands. In particular, neural networks have become popular to generate images from a given text description.

[0003] Users want to control the layout of objects in the generated images and the visual attributes of the objects (color, shape, pose, motion) to suit their specific objectives. Likelihood-based models are often used to create high-resolution, complex scenes, but can require billions of parameters in autoregressive (AR) transformers. Promising results have also been obtained using Generative Adversarial Networks (GANs). However, GANs' adversarial learning procedures do not easily scale to modeling complex multi-model distributions. Furthermore, both AR models and GANs require a lot of trial and error to arrive at the desired image.

[0004] Recently, diffusion models have been shown to generate high-quality images for text input. Popular diffusion models include OpenAI's Dall-E2, Google's Imagen, and StabilityAI's StableDiffusion. Diffusion models can generate complex distributions for natural images without requiring billions of parameters like AR models, and do not suffer from the mode collapse problem exhibited by GANs. However, diffusion models are still computationally intensive, as they tend to use excessive processing to model imperceptible details in the image data.

[0005] Furthermore, while diffusion models have shown promising high-quality image synthesis, the inventors have found that the models often struggle to correctly interpret text inputs. Conventional methods may fail to consider relationships between multiple text descriptions and fail to match local features (visual attributes) to the layout. For example, conventional methods construct multiple descriptions about an image as requirements, use them to guide image generation, and use layout loss to guide the location of objects in the image to their desired locations. However, using these methods, when two of the same objects are given as input, diffusion models often struggle to assign the correct description to each object, and may even completely ignore one of the objects in the output image. Furthermore, when two different objects are given as input, diffusion models have been shown to improperly combine the two objects into one hybrid object.

[0006] Therefore, it is desirable to improve the accuracy of neural networks, such as diffusion models, in generating desired images from inputs. It is also desirable to reduce the trial and error costs of neural networks. Summary of the Invention

[0007] The invention is defined in the independent claims, to which reference should be made here. Further characteristics are set out in the dependent claims.

[0008] According to an aspect of the present invention, there is provided a computer-implemented method for generating an image, the method comprising: receiving input text and layout information; decomposing the input text into one or more global features and one or more local features; encoding the one or more global features into a global vector and the one or more local features into respective local vectors; initializing an initial image representation; predicting a global noise from the global vector and the initial image representation using a trained neural network; predicting, for each of the one or more local features, an initial local noise from the one or more local vectors and the initial image representation using a trained neural network; determining a final local noise for each of the one or more local features using the initial local noise and the (predicted) global noise; using said layout information to combine said predicted global noise and said final local noise into a combined noise (possibly using a refinement vector of said predicted global noise for this combination, e.g. using unconditional noise, where the final global noise is determined from the predicted global noise in the previous step); denoising the initial image representation using the synthetic noise to provide a next image representation for generating the image; Includes. [Brief description of the drawings]

[0009] Reference is made, by way of example only, to the accompanying drawings in which: [Figure 1] 1 is an example of a known configuration used for image generation. [Diagram 2] This is an example of a known diffusion model. [Diagram 3] 1 is an example of a known image generation method using a diffusion model. [Figure 4A] An example of a failure of the image generation method in FIG. [Figure 4B] An example of a failure of the image generation method in FIG. [Diagram 5] 4 illustrates the denoising process performed by the diffusion model. [Figure 6] 2 is a second example of a known image generation method using a diffusion model. [Figure 7] An example of a failure of the image generation method of FIG. [Figure 8] 1 is a third example of a known image generation method using a diffusion model. [Figure 9A] An example of a failure of the image generation method of FIG. 8 is shown. [Figure 9B] 4 is a fourth example of a known image generation method using a diffusion model. [Figure 10] 4 illustrates method steps for generating an image from text input and layout information according to the methods disclosed herein. [Figure 11] FIG. 11 is a functional block diagram of a unit capable of executing the method according to FIG. 10; [Figure 12] 4 illustrates another example of generating an image from an input scene graph according to the methods disclosed herein. [Figure 13] 1 is an example of a scene graph decomposition method. [Figure 14] An example is given of converting an input scene graph into a text output that describes the global and local graphs of the scene graph. [Figure 15] FIG. 1 is an example of a functional diagram for predicting noise from global and local text inputs. [Figure 16] FIG. 1 is an example of a functional diagram for predicting noise from global graph inputs and local graph inputs. [Figure 17] 13 is an example of combining predicted global and local noise into a combined noise. [Figure 18]13 is an embodiment of the method according to FIG. 12 for generating an image. [Figure 19] 1 shows an example of an image generated using the methods disclosed herein. [Figure 20] 4 is a further example of an image generation method using input text, layout information, and layers. [Figure 21] 1 shows an example of input text, layout information, and layers. [Figure 22] An example of determining noise from a hierarchical input is shown. [Figure 23] We show the effect of changing hyper-parameters on the generated images. [Figure 24] FIG. 1 is a block diagram of an information processing device or computer system. [Diagram 25] 1 is an example of a graphical user interface for inputting text and layout information. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] FIG. 1 shows a known configuration of a simple image generation method 100. The method illustrates the inference or implementation stage of image generation. A user inputs a text description 110 into an image generation model 120. For example, the user may want a "picture of a squirrel eating a hamburger". The image generation model 120 may be, for example, a neural network such as a diffusion model. In particular, the diffusion model may be a diffusion probabilistic model such as a latent diffusion model.

[0011] Diffusion models are often trained using datasets that contain image and caption pairs. For example, LAION-5B, a public dataset derived from general crawl data, is a popular training set. The dataset is often preprocessed by normalization and scaled to an appropriate range. Diffusion models are usually trained using a loss function derived from maximum likelihood estimation (MLE) or using score matching. Advanced optimization techniques such as stochastic gradient descent (SGD) or Adam can be used to efficiently train the model. Further learning rate schedules or warm-up strategies can be used to stabilize the model performance. Further mechanisms such as cross-attention and latent space have been implemented in diffusion model architectures to improve the quality of the generated images. However, image generation using known techniques still has deficiencies, and the model may not be able to generate the desired images.

[0012] In the inference stage, an input vector 130 is also provided to the image generation model along with the text description. For example, the input vector may be a vector of an initial image representation. In the inference stage of a diffusion model, the initial image representation is, for example, a random noise image generated from a random seed. The input vector is, for example, a k-dimensional vector generated from a random seed.

[0013] For example, if the image generation model is a generative adversarial network (GAN), the input vector is a noise vector. The noise vector may be a 100-dimensional noise vector generated by sampling a normal distribution.

[0014] The image generation model may work in an iterative process to generate the final image 140. For example, the image generation model may iteratively denoise the input vectors to arrive at a final image with a high probability that the image matches the text input. A method for using a diffusion model to arrive at a desired image is described below in relation to FIG. 2. Once the final representation is generated by the image generation model, the representation may be input to an image decoder or upsampler (not shown) to generate the final image.

[0015] FIG. 2 illustrates an example of a diffusion model 200 that can be used in the image generation method 100 described above (and that can be implemented in the methods described herein). In this example, an input text 210, "A black dog standing, and a dog is white and walking," is input to a tokenizer 220. The tokenizer splits the input text into tokens and converts each token into an ID. An example tokenization 225 is shown in FIG. 2. Each word in the input text is split and assigned an ID. For example, the word "dog" is assigned ID

[33] and the word "black" is assigned ID [2]. Any suitable tokenizer can be used. Each tokenizer can have its own dictionary that matches words with unique IDs. For example, the inventors used HuggingFace's CLIP tokenizer, available at https: / / huggingface.co / docs / transformers / v4.31.0 / en / model_doc / clip#transformers.CLIPTokenizer, as described with respect to FIG. 15. Of course, other versions of the CLIP tokenizer can be used, such as CLIP 4.33.0 or earlier versions.

[0016] Once the input text has been tokenized and assigned IDs, a text encoder 230 converts each ID into an input vector 240. For example, the input text sentence shown here has 11 words, each of which is assigned an individual ID. Thus, the text encoder converts the IDs into a [11xD] vector, where D represents the dimension of the vector, which in this example is equal to 768. Any suitable text encoder can be used. For example, the inventors used HuggingFace's CLIP text encoder, available at https: / / huggingface.co / docs / transformers / model_doc / clip, as described with respect to FIG. 15.

[0017] The tokenizer and text encoder can be considered as separate units. Alternatively, in some examples, the tokenizer can be considered as part of the text encoder. In another example, the tokenized words can be embedded, for example, using OpenAi's ViT-L / 14 CLIP model. The embeddings are 768-value vectors, with each token having its own unique embedding vector. The embeddings may need to be further processed using a text transformer before being fed into the diffusion model. For example, the CLIP encoder includes a process to convert IDs to embeddings. The encoder can transform the embeddings by considering other embeddings of words in the text.

[0018] An attention score can be calculated for the input vector. For example, if the diffusion model uses a U-net architecture, both self-attention and cross-attention mechanisms can be used. The self-attention layer can be introduced at the coarse resolution of the U-net. For a given representation, in this example a 5x7 representation, a self-attention score is calculated for each element in the input vector. For each word or element in the vector, the self-attention score is used to determine the importance of all other words in the vector with respect to that word. The self-attention score is calculated using known methods. For example, in a self-attention mechanism, every input vector is assigned three representations: key, query, and value. The weights of the key, query, and value representations can be randomly initialized using a suitable random distribution such as a Gaussian, Xavier, or Chiming distribution. The initialization can be done once before training the self-attention mechanism.

[0019] Self-attention scores are calculated for the input vectors using standard methods. For example, dot-product attention is calculated as the score function. A softmax is taken for each attention score. A weight value can be calculated by multiplying the softmax attention score with a value representation. The weight values ​​can then be summed element-wise to produce the output. If there are multiple input vectors, the process is repeated with each input vector to produce the output.

[0020] In this example, an attention score is calculated for each word in each patch of the 5x7 representation, so an overall 11x35 attention score matrix is ​​calculated - one score for each token in the input vector, and for each 5x7 patch, or position, or square in the representation.

[0021] For example, the diffusion model may be a latent diffusion model, such as the stable diffusion model described in (Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. 2022, High-resolution image synthesis with latent diffusion models, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition) by StabilityAi. For example, instead of training the diffusion model using a 512x512 RGB image, which results in a 786,432-dimensional landscape, the input image is compressed into a much smaller image in the latent space. For example, a 512x512 image may be compressed into a 4x64x64 representation in the latent space. In the above example, for example, the 5x7 representation is a representation in the latent space. Thus, instead of operating on the raw input image, the stable diffusion model uses the latent space. Using the latent space significantly increases the processing speed because the image size is much smaller than the input image. An input image, for example a 512x512 image, is compressed into a small latent space using an encoder, processed, and transformed into pixel space using a decoder network.

[0022] As shown in Figure 2, a self-attention mechanism can be used to indicate lexical attributes as well as spatial layout. Thus, in the example sentence "a black dog standing, and a dog is white and walking", the attention mechanism can identify that the description "black" is assigned to the word "dog" and not to the background of the image, etc. The diffusion model then uses the attention score to predict the noise 260 of the representation. For example, the noise is predicted by a u-net architecture. If the representation is specified in a 4x64x64 tensor, the predicted noise 260 will also be a 4x64x64 tensor. The input vector is used multiple times by the u-net architecture to tune the diffusion model to generate a representation that represents the input text. That is, the diffusion model uses cross-attention between the prompt and the image representation. The attention mechanism cross-contextualizes the input text with the coordinate-aware representation of the image and generates an output score for each token of the input vector and each patch of the representation.

[0023] The inventors have found that diffusion models, particularly the stable diffusion model described above, often have misconnections 270 produced by cross-attention models when used in the prior art. An example of a misconnection produced by a latent diffusion model is shown in FIG. 2. The attribute "standing" is misassigned to "white dog" when it should only be assigned to "black dog". Similarly, the label "black" is misassigned to a patch that was assigned to a white dog. Thus, the generated image is likely to be an incorrect representation of the input text. See FIG. 4 for an example of a failure to use this method.

[0024] FIG. 3 shows a known image generation method 300 that uses a diffusion model 320 as the image generation model 120. In this example of the image generation method, the diffusion model uses the stable diffusion model described in connection with FIG. 2. The image generation method 300 is described, for example, in Rombach, R., Blattmann, A., Lorenz, D., Esser, P. and Ommer, B., 2022. High-resolution image synthesis with latent diffusion models, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 10684-10695). An initial text description 310 is input to a text encoder 315. The input text may be tokenized using known tokenization methods, for example the CLIP tokenizer, before being input to the text encoder.

[0025] Additionally, an initial image representation, such as an input vector 130 (not shown), is input to the diffusion model along with the text description. The initial image representation may be a K-dimensional vector as described above. That is, the initial representation may be a random K-dimensional vector generated from a random seed. Those skilled in the art will appreciate that although the term vector has been used here and elsewhere to describe the initial image representation, this representation may be a multi-dimensional vector and therefore may be referred to as a tensor. The terms "vector" and "tensor" are used interchangeably herein. FIG. 3 illustrates a current representation 330 that is input to the diffusion model. The method 300 may be an iterative process and may be repeated multiple times, updating the current representation each time. The first iteration of the method uses the initial image representation as the current representation.

[0026] A pre-trained diffusion model 320 is used to predict noise from the input text description. In this example, the pre-trained diffusion model uses a U-net architecture. The model can be trained by first adding Gaussian noise to each image in the training set, and in the reverse process, the model can learn a data distribution that denoises the noisy images back to the initial input. Both the forward process of adding noise and the inverse denoising process are modeled as a product of Markov transition probabilities. Details of the denoising process are described in connection with FIG. 5.

[0027] Starting with an initial image representation and a text description, the diffusion model uses the learned data distribution to predict the noise associated with the image representation, predicted noise 325. The predicted noise is, for example, a tensor with the same size / resolution as the input image. In another example, the input image is transformed into a latent space and the noise is a tensor with the same size as the latent representation. The predicted noise and the current representation are input to a denoising unit 335. The denoising unit denoises the current representation with the predicted noise to obtain a next image representation 340. The next image representation is then set as the current representation and the image generation process is repeated.

[0028] The dashed box 345 indicates the steps of image generation that are repeated in an iterative process to generate a final representation 350. The next representation (either the initial image representation or a representation generated by processing the previous representation through a diffusion model and a denoiser) is input to the diffusion model as the next current representation, and the diffusion model predicts the noise of that representation. The current representation is then denoised using the predicted noise to generate the next representation. This process is repeated N times, where N may be determined experimentally, for example N=50 may be used. Once the denoising process has been repeated N times, the final image representation 350 is generated. The final image representation is input to an image decoder 355 to generate the final generated image 360.

[0029] 4A and 4B show examples of failures 400A and 400B using the above-mentioned image generation method. The inventors have found that the above model has difficulty in reflecting all the visual details described in the input text in the generated image. For example, the input text describes two people, but the generated image contains three people, or a red box is above a blue box, but these are reversed in the generated image.

[0030] 4A shows three examples where the diffusion model failed to generate the desired image: The first text input 405a shows an example of the description "a cup above a clock", but the first image output 410a fails to show the cup.

[0031] The second text input 415a shows an example description: "two people standing on a sandy beach flying a kite." However, the second image output 420a shows three people, and the image generation fails to generate and connect the body parts correctly. The person flying the kite is not clearly shown.

[0032] A third text input 425a shows an example description: "a sheep by another sheep standing on the grass with sky above and the ocean by a tree and a boat on the grass." A third image output 430a shows that the diffusion model was unable to generate a second sheep.

[0033] FIG. 4B illustrates an exemplary prompt 405b, "a sheep sitting on the grass and another black sheep is standing," input into the well-known DALL-E2 image generation model. DALL-E2 is another example of an image generation model using a diffusion model. FIG. 410b illustrates an example where an image is successfully generated with one black sheep standing and another sheep sitting. FIG. 420b, FIG. 430b, and FIG. 440b illustrate further examples where the model failed to generate a correct image following the prompt. Thus, in this example, the model was only 25% successful in generating a correct image using the given prompt. Thus, the inventors have found it desirable to improve the ability of the diffusion model to generate accurate images.

[0034] Figure 5 shows an example result of the diffusion model denoising process 500. As described above in relation to Figure 3, the diffusion model can be trained in a forward direction by adding a known Gaussian noise to each image in the training set, and in a reverse direction, the model can learn a data distribution that denoises the noisy images back to the initial input. That is, the model uses the known Gaussian and ground truth of the training images to train the diffusion model.

[0035] An initial representation 510 is input to the diffusion model. The initial representation is, for example, a k-dimensional input vector generated from a random seed. In this example, representations 510, 520, 530, and 540 are shown. The visual representations show example outputs when images are generated during each iteration of the denoising process. However, those skilled in the art will appreciate that the denoising in the stable diffusion model described above occurs in latent space, and thus visual images are not necessarily generated at every iteration. A visual image may be generated if the latent space image is decoded at each step. However, decoding a representation during a denoising cycle may add extra computation. A first intermediate visual representation 520 shows a first denoising step performed on the initial representation 510. In the denoising step, a predicted noise is subtracted from the previous image representation. Thus, in this first denoising step, a predicted noise is subtracted from the initial representation to generate a first intermediate visual representation. This process is repeated with the diffusion model taking the first intermediate visual representation as input and generating a predicted noise. The predicted noise is then subtracted from the first intermediate image representation to generate a second intermediate image representation 530. The first intermediate image representation is said to be denoised by the predicted noise. This process is repeated again to generate a final image representation 540. With reference to Figure 3, the final image representation may form the input of an image decoder for generating the final generated image.

[0036] In this example, the input text description may be "a cat lying down." This is an example of a conditional diffusion model because a text description is given as an input. In other examples, the diffusion model may be an unconditional diffusion model, where the model is trained on a dataset and, given a random seed, generates images without a text prompt or generates prompts of any modality based solely on its training data.

[0037] 6 shows a second known image generation method 600 that uses diffusion models 620a, 620b as the image generation model 120. In this image generation method, the diffusion model may be the stable diffusion model described in connection with FIG. 3 or other diffusion models such as the well-known GLIDE model. The image generation method 600 is described, for example, in "Compositional Visual Generation with Composable Diffusion Models."

[0038] The image generation method aims to improve the generation of accurate representations of input text. To generate images, the method constructs a set of diffusion models, i.e., diffusion models 620a, 620b shown in Fig. 6, each modeling a specific component of the image. To do this, the diffusion models were interpreted as energy-based models that can explicitly combine data distributions defined by energy functions.

[0039] In this method, the input text is split into two input sentences. The first input sentence 610a contains information about a first concept, in this example "Road leading into the mountains", and the second input sentence 610b contains text about a second concept, "Yellow trees on the side of the road". The sentences are each input to a text encoder 615a, 615b. In the diagram, the text encoders are shown as separate units. However, a single text encoder may be used. The diagram shows that the first and second input sentences are encoded separately.

[0040] Similar to the method described in connection with FIG. 3, an initial representation, such as the input vector 130, may be input to the diffusion model along with the text description. The initial representation may be a K-dimensional vector as described above. That is, the initial representation may be a random K-dimensional vector generated from a random seed. FIG. 6 shows a current representation 630 that is input to the diffusion model for simplicity. The method 600 is an iterative process and may be repeated multiple times, updating the current representation each time as described above. The dashed box 645 indicates the steps that are repeated. The first iteration of the method uses the initial image representation as the current representation.

[0041] The encoded text is input to the diffusion models 620a, 620b. As with the encoder, two diffusion models are shown in the figure. However, the same diffusion model can be used in each case. For example, the authors of this paper mainly used the GLIDE diffusion model. The diffusion models are shown as separate units to show that the prediction noise is calculated separately for each input text.

[0042] The predicted noise from each diffusion model is input to the combiner 625, where the noise is combined. As mentioned above, the authors of this paper interpreted the diffusion models as energy-based models that can explicitly combine data distributions defined by energy functions. The predicted noises can be combined using a conjunction concept, which aims to obtain an image containing certain defined attributes, or a negation concept, which aims to generate an image without certain factors.

[0043] The synthetic noise is then used to denoise 635 the current representation 630 to generate the next representation. The denoising process uses known methods such as subtracting noise from the current representation.

[0044] Figure 7 shows examples of failures 700a and 700b using the second image generation method described above. Several problems have been identified with the second image generation method. First, the model cannot determine whether two input sentences describe the same object. Second, the model cannot determine which phrases correspond to which objects.

[0045] In the first example, the input sentences 705a, "A bird" and "a flower", are fed into the image generation model. The output of the model is a generated image in which the bird and the flower are merged into a single object.

[0046] Similarly, in the second example, the input sentences "A couch" and "A dog sitting in the living room" are fed into an image generation model, which outputs an image containing a hybrid of a dog and a couch.

[0047] Accordingly, the present inventors have discovered that it would be desirable to improve the ability of the diffusion model to generate accurate images given user input.

[0048] FIG. 8 shows a third example of a known image generation method 800 using a diffusion model 820. The image generation method 800 is described, for example, in Chen, M., Laina, I. and Vedaldi, A., 2023. Training-free layout control with cross-attention guidance. arXiv preprint arXiv:2304.03373. In this image generation method, the diffusion model used was the stable diffusion model described in Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, 2022. In particular, the method used Stable-Diffusion (SD) V-1.5 trained on the LAION-5B dataset. This diffusion model and training set can also be used with the methods disclosed herein.

[0049] The image generation method 800 aims to improve the generation of an accurate representation of an input text by supplementing the text input with a layout input. To generate an image, the method takes as input a text description 810a, in this example, "A man with white tie and a dog is black and running", along with a layout input 810b with bounding boxes that specify the objects in the text description. In this example, the layout shows a "dog" bounding box and a "man" bounding box.

[0050] The Training Free Layout Control with Cross Attention Guidance paper presents two methods that can be used to generate images from input text descriptions and bounding boxes. The first method uses so-called forward guidance. The text description and the current representation 830 are input to a diffusion model 820a, and an attention map 825 is extracted from the diffusion model. The input bounding box is represented as a smooth window function that is equal to a constant greater than zero inside the box and quickly becomes zero outside the box. The window function is rescaled and the attention map is biased by the bounding box. The biased attention map is injected into the diffusion model at a selected subset of layers during each application of the model. Thus, the updated 845 activation maps computed by each of the selected layers are replaced by the biased attention map. In this example, the regions in the layout where "dog" is present and the regions where "man" is present are shown in the dashed box 850. The attention maps of "dog" and "man" are biased, but their bounding boxes are biased.

[0051] The second method uses so-called backward guidance. In the backward guidance method, the attention map 825 is biased by introducing an energy function. This method is similar to classifier guidance known in the art. At each application of the diffusion model, when a layer in the model is evaluated, the gradient of the loss, i.e., the energy function, is calculated via backward propagation to update the potential at that (time) step of the diffusion method. In this method, the loss on the cross-attention map was calculated for the middle block and the first block of the upsampling branch of the U-net architecture in the stable diffusion model. As shown in FIG. 8, the next representation 840 is generated by alternating between updating the gradient, i.e., update 845, and a denoising step, where the diffusion model 820b is used to predict noise and the updated representation is denoised using a denoiser 835.

[0052] Both the first and second methods rely on attention maps extracted from a stable diffusion model and use cross-attention techniques to inject the updated attention maps into the model.

[0053] FIG. 9A shows a failure example 900A using the third image generation method 800 described above. The inventors have confirmed that the third method still fails to correctly represent all input objects. As shown in FIG. 9, an input text description 905a of "a sheep standing on the grass and another black sheep is sitting" was given along with bounding boxes 910a of two sheep next to each other. However, the third image generation method failed to generate two sheep in the output image 920a. Although the model captured one sheep, the sheep in the image should be sitting as it is a "black sheep", and no standing sheep was generated.

[0054] FIG. 9B shows a fourth known image generation method 900B that uses a diffusion model 920b. The image generation method 900 is described in "Bar-Tal, O., Yariv, L., Lipman, Y. and Dekel, T., 2023. Multidiffusion: Fusing diffusion paths for controlled image generation". In this image generation method, the diffusion model used was the stable diffusion model described above. Specifically, this method used the stable diffusion described in "von Platen, P., Patil, S., Lozhkov, A., Cuenca, P., Lambert, N., Rasul, K., Davaadorj, M., and Wolf, T. Diffusers: State-of-the-art diffusion models. https: / / github.com / huggingface / diffusers, 2022".

[0055] Similar to the third known image generation method, this method 900B aims to improve the generation of a representation of an input text by supplementing the text input with a layout input. To generate an image, the method takes as input a number of text descriptions 905b, in this example "A photo", "a pink cloud", "a yellow balloon", and "a tower", along with a layout input 910b with region masks that specify objects in the text descriptions. https: / / github.com / huggingface / diffusers, 2022

[0056] The diffusion model 920b predicts the noise for each text input defined by their masks, synthesizes the noise for each object into a single noise, and then updates the image representation with the synthesized noise (e.g., by denoising the current representation) to generate the next representation.

[0057] In this method, the noise for each text input is determined separately and simply combined into a single noise. That is, this method does not consider the relationship between the text inputs. A user can input, for example, "a black dog" and "a pink ball", but cannot give a relationship to the text, such as "a black dog is playing with a pink ball". Thus, a user may not be able to generate a desired image from the input text using this method.

[0058] FIG. 10 illustrates steps of a method 1000 for generating an image from input text and layout information according to the methods disclosed herein. In a receiving step s1010, the input text and layout information may be received (or loaded). For example, the input text and layout information may be input by user input on a graphical user interface (GUI), where the layout information is input as a bounding box on a canvas for each object, and the input text is input as one or more labels added to each bounding box to name the object. That is, the method may receive the input text and layout information from a user input on a GUI. The layout information may, for example, specify position and / or size information for one or more objects described in the input text. The input text may further be input as a label along with links to form a predicate for one or more objects. The links may, for example, be drawn on a canvas or added using a menu function. The predicate may include, for example, a labeled link between two objects giving a relationship between the objects, or a labeled link to an attribute of the object.

[0059] The input text may include global features and local features. Global features include global predicates or objects that connect two objects, where a global predicate defines a relationship between two objects, and local features include local predicates that define attributes of a single object. Those skilled in the art will appreciate that an "object" may include a group of objects, e.g., a collection of similar objects such as a "group of dogs." Thus, a single object in this sense includes multiple related objects.

[0060] The input text and links may form a scene graph; that is, the input text and links may be input by a user as a scene graph. The method may receive a scene graph from a user, for example by input to a GUI. A scene graph is a graph that shows a particular combination of at least one object and at least one relationship, action, or property (or 'predicate'). A predicate may define the details of a single object, such as a 'black' dog, where 'black' is the predicate and 'dog' is the object, and a predicate may define an action or relationship between objects, such as a man 'looking at' a dog, where 'looking at' is the predicate. Object nodes and predicates that link two or more objects may be considered as global features of the scene graph, and actions or descriptions of single objects may be considered as local features. A predicate that links two or more objects may be called a global predicate, and a predicate that links a single object may be called a local predicate.

[0061] As mentioned above, a scene graph can represent objects in an image and their interrelationships. That is, a scene graph can correspond to a logical / spatial representation of a graphical scene. Details of a scene graph are provided in connection with FIG. 13.

[0062] In the decomposition step s1015, the input text is decomposed into one or more global features and one or more local features. As described above, the input text may include global features and local features. For example, the input text may describe one or more objects, and two objects may be linked or connected by a predicate. Furthermore, each of the one or more objects may have a labeled link to an attribute of that object. The input text may be decomposed by extracting all objects (and links between objects, if any) in the text input as global features, and extracting each object (and attributes of that object, if any) as a local feature.

[0063] Additionally, the input text can be resolved into a global background. For example, the input text can describe a scene, a background, or a style. Input text describing a background can be resolved into additional global background. Details of resolving the input text into a background are provided in connection with FIG. 21 below.

[0064] In the example where the input text and links form a scene graph, the scene graph can be decomposed into global and local features using graph decomposition. For example, the inventors have found that the scene graph can be decomposed into global and local features by extracting all object nodes and their connections, i.e., their global predicates, as a global graph, and extracting each object node and the attributes connected to that object node, i.e., the object local predicates, as a local graph. A local graph can be created for each object and its associated attributes. Details of the graph decomposition method are provided in connection with FIG. 13.

[0065] In the encoding step s1020, one or more global features are encoded into a global vector and one or more local features are each encoded into a local vector. The global background can also be encoded into a global background vector. The inventors have found that two different methods can be used to encode the features into global and local vectors (and global background vector, if included). In a first method, the inventors have encoded the global and local features into global and local vectors using a text encoder, such as the text encoder described in connection with FIG. 2. That is, the inventors have used a CLIP text encoder, but of course any suitable text encoder can be used. Those skilled in the art will appreciate that the encoded text vectors may be embedded as part of the encoding step. In an example where the input text and links form a scene graph, to use the text encoder, the global and local features are first converted to text, for example using a graph-to-text converter. The graph-to-text encoder can convert a graph, for example in the form of a scene graph, into text using a rule-based converter. More details of the conversion process s1017 (not shown) are provided in connection with FIG. 14.

[0066] In the second method, the global and local features are encoded using a graph encoder. The global feature or global map is input to the graph encoder and the output is a global vector. Similarly, each local feature or local feature map is input to the graph encoder and the output is a local vector for each input. Any suitable graph encoder can be used, such as a Graph Neural Network (GNN) or Graphormer. For more details on the graph encoding process, see FIG. 16.

[0067] In an initialization step s1025, an initial image representation is initialized. For example, the initialization may be of the same type as the initialization described in relation to FIG. 3. That is, the initialization may be an initial image representation. The representation may be a k-dimensional vector representing random playback noise. The representation is initialized, for example, by generating random noise from a random seed. The initialization step may be referred to as a generation step, for example, since an initial image representation is generated.

[0068] In a prediction step 1030, the global noise is predicted from one or more global features and the initial image representation using a trained neural network. The trained neural network may be a diffusion model. For example, the trained neural network may be a latent diffusion model. The diffusion model may be called a denoising diffusion probabilistic model (DDPM). The inventors used the stable diffusion 2.0 model by stabilityAI, available at https: / / stability.ai / blog / stable-diffusion-v2-release. The stable diffusion model by StabilityAI is an example of a latent diffusion model. The diffusion model takes as input the image representation and a global vector and predicts the global noise. The diffusion model predicts the global noise using standard methods. For example, the stable diffusion model works in latent space. The latent space representation is input to the diffusion model and the predicted noise is output as a tensor. The tensor is the same size as the latent representation input.

[0069] As mentioned above, the diffusion model may be a pre-trained latent diffusion model. For example, the inventors used a diffusion model trained on (a subset of) the LAION-5B training set available from: Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M. and Schramowski, P., 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35, pp. 25278-25294. Of course, other training sets may be used. In general, diffusion models are probabilistic models trained to learn the data distribution by gradually denoising Gaussian or normally distributed variables. This corresponds to learning the inverse process of a fixed Markov chain. Successful diffusion models use a reweighted variant of the lower bound of the variance of the data distribution. A diffusion model is effectively a uniformly weighted sequence of denoising autoencoders trained to predict the noise in the input, which is a noisy version of the initial training images of the Markov chain.

[0070] The diffusion model we used was trained using a cross-attention mechanism for conditioning (other suitable training mechanisms can also be used). The cross-attention mechanism is similar to the self-attention mechanism described in connection with Figure 2. However, instead of using one embedding sequence as input, the cross-attention used in the Transformer mixes two different embedding sequences. The two sequences may be of different modalities, such as text, image, or audio. One of the modalities is used as the query matrix in the cross-attention computation and therefore defines the dimensionality and length of the output.

[0071] The diffusion model we used includes a conditional denoising encoder to model the conditional distribution. The diffusion model implements the U-net architecture commonly used in such models. A cross-attention mechanism was used between the images generated by the U-Net model and the conditional text prompts (i.e., the input text prompts). The U-net backbone underlying the diffusion model is effectively extended by the cross-attention mechanism. To preprocess the input prompts from various modalities, the diffusion model introduces a domain-specific encoder that projects the prompts into an intermediate representation. This representation is then mapped to the intermediate layers of the U-Net through a cross-attention layer.

[0072] Similar to the self-attention mechanism described in connection with FIG. 2, the cross-attention mechanism uses key (K), query (Q), and value (V) matrices. However, unlike the self-attention mechanism where K, Q, and V are derived from a single sequence, the cross-attention mechanism uses multiple sequences. For example, in the diffusion model used by the inventors, the spatial features of the generated image (the intermediate noise image of the Markov chain) are projected onto a query (Q) matrix, and the text embedding of the prompt is projected onto a key (K) matrix and a value (V) matrix. The attention score can then be calculated by applying a softmax function to Q, K, and V using a normalization constant.

[0073] Although the inventors have used a latent diffusion model with a cross-attention mechanism, those skilled in the art will appreciate that any suitable diffusion model may be used.

[0074] In a prediction step s1035, an initial local noise is predicted from each local feature and the image representation using a trained neural network. The trained neural network may be the diffusion model described above. As an example, one of the local features may be a 'black' dog and a second local feature may be a 'white' cat. In this example, the global feature is 'the dog and the cat'. For each local feature, the diffusion model predicts an initial noise associated with that feature and the image representation. Thus, in this example with two local features and their object node, the diffusion model computes two initial local noises. The initial local noises are, for example, predicted noises generated by the diffusion model. The initial local noises can be considered as raw (or pre-processed) noises directly output from the neural network.

[0075] The diffusion model predicts the noise for each vector given as input separately, i.e., it predicts a global noise for the global vector, an initial local noise 1 for local vector 1, and an initial local noise 2 for local vector 2. Those skilled in the art will appreciate that the term initial local noise refers to the noise of the representation computed using the local features, and that in this step of the method, the initial local noise is a noise prediction over the entire representation, not just the region of the representation assigned to the object node of that local feature. The diffusion model can further predict a global background noise if a global background is loaded.

[0076] The global features and the local features may each occupy a separate layer in the input. For example, the global features and the local features may each occupy conceptually successive layers. The global features may, for example, occupy a global layer and the local features may occupy a local layer, or the local features may occupy different local layers. Further features of a local feature (features related to or overlapping with the local feature) may occupy additional layers superimposed on the local feature's conceptual layer. The additional layers may use noise in the local feature's layer in determining the noise on the additional layers. Details of the layer configurations and further features are provided in relation to Figures 20-23 below.

[0077] In a determining step s1037, a final local noise for each of the one or more local features is determined using the initial local noise and the global noise. That is, the final local noise for a local feature is determined using the predicted noise for that local feature and the global noise. The final local noise for each local feature is determined, for example, by subtracting the predicted noise for the global vector from the predicted noise for the local vector, preferably then multiplying by a local noise hyperparameter, and extracting the final local noise using layout information for the object associated with the local feature.

[0078] In a combining step s1040, the global and final local noises are combined into a combined noise. For example, the global noise and the final local noises may be summed to combine the noises into a combined noise. That is, one or more final local noises are added to the global noise. If the noises are represented as tensors, the tensors are summed. The noise tensor may be, for example, a tensor in the latent image space where the size of the noise tensor is the same as the latent image representation. The layout information is used to construct the combined noise. For example, the layout information may be used to extract local noises within a region of the initial image representation defined by the layout information. Details of the noise construction are provided in connection with FIG. 17.

[0079] In examples where global background and / or additional features are included, the synthesis step also includes global background noise and / or additional feature noise, see e.g., FIG.

[0080] In the update step s1045, the initial image representation is updated by denoising it with synthetic noise. The initial image representation may be denoised using known denoising methods. For example, the denoising method may follow a sampling method similar to Langevin dynamics with a predicted parameterisation ε as a learned gradient of the data density. That is, ε may be similar to Langevin dynamics and simplify the variational bounds of the diffusion model to be similar to denoising score matching. The initial image representation may be denoised by multiplying the predicted noise by a coefficient to generate a "tuned" noise and then subtracting the tuned noise from the image representation. Gaussian noise may be added to generate the next image. If the image representation is given by a tensor, e.g., if the tensor is a latent space, the combined noise is also given as a tensor of the same dimension as the representation tensor.

[0081] The dashed box s1060 indicates a further optional step of the image generation method. In the iteration step s1050, the prediction, synthesis and update steps may be repeated N times. For example, these method steps may be repeated 3-5N times, for example, the method may be repeated a total of 50 times. The inventors have found that repeating the steps 50 times produces favorable results. By repeating these method steps N times, the image quality of the final representation may be improved. In each iteration, the current image representation is updated by the denoised representation from the previous step. This sequence of method steps may be performed iteratively, with more features of the denoised image recovered after each iteration. In the first iteration, the initial image representation is denoised, and the denoised representation may be referred to as the next representation. The next representation, or current representation, replaces the initial image representation in subsequent iterations.

[0082] In the generating step s1055, the final image is generated. The final image can be generated using an image decoder known to those skilled in the art. For example, if the neural network operates on the representation in the latent space, the image decoder can convert the representation to pixel space. The generated image can then be displayed to the user. The generated image can be a 3x640x640 or 3x512x512 generated image (RGBxheightxwidth).

[0083] Thus, the method described herein can control the features of the image to be generated by separately guiding the global features (i.e., layout of objects and their relationships) and local features (attributes of each object) provided by the input text and layout information (e.g., in the form of an input scene graph having a layout). The inventors have found that by controlling the global (e.g., layout) and local (e.g., visual attribute) features for image generation, the method disclosed herein reduces the trial and error cost for obtaining a desired image compared to conventional methods.

[0084] FIG. 11 shows a functional block diagram 1100 of units or modules capable of carrying out the above method. In this example, the input text and links are represented as a scene graph 1110, but they can be input as a complete scene graph or by inputting the information that constitutes the scene graph in pieces. The scene graph may include layout information or this may be input separately. The scene graph may include an option to input a background (not shown, see FIGS. 20-23). ​​The scene graph is input, for example, to a graph decomposition unit 1120. The graph decomposition unit decomposes the scene graph into global features 1122 and local features 1124. For example, the graph decomposition unit generates a global graph with all object nodes and all predicate nodes (or predicate edges) connected to two or more object nodes. That is, the global graph is a representation of the global features of the scene graph. The graph decomposition unit may also generate a local graph for each object and predicate associated only with that object. That is, the local graph includes only one object node and all predicate nodes connected only to that object node. A local graph may be generated for each object in the scene graph.

[0085] The global features are input to an encoder unit 1125a. The encoder unit encodes one or more global features into a global vector. The encoder unit may be a text encoder, such as a CLIP text encoder, or a graph encoder, such as a GNN. For example, in the case of a GNN, the nodes of the input graph are represented by vectors and the edges / connections are represented by scalars with larger values ​​representing "stronger" connections. Node information is propagated between nodes depending on the edge values, and each node vector is modified / updated by each other. If the text input and links are represented as a scene graph and a text encoder is used, the global features or global graph may first be converted into text sentences before being input to the text encoder. A graph-to-text unit (not shown) may be used to convert the global features into text before being input to the text encoder, for example using a rule-based method.

[0086] Similarly, the local features are input to the encoder unit 1125b. The encoder unit encodes one or more global features each into a global vector. The encoder unit may be a text encoder and / or a graph encoder as described above. Although the encoder units 1125a, 1125b are shown as separate units in the figure, one skilled in the art will understand that these units may belong to the same text encoding unit. Alternatively, the encoders may be different text encoders. For example, the encoder of the global features may be a graph encoder and the encoder of the local features may be a text encoder. The reverse is also true. For example, different encoders may be used if the encoding vectors (i.e. the outputs of the encoders) share (or are within) the same latent space. That is, given the same text (e.g. cars), it is preferable to use different encoders to generate the same / very close vectors from different encoders. Furthermore, if there are multiple local features, the same or different encoders can be used for each local feature.

[0087] Each encoder can output a vector of the same size for use by a trained neural network, in this example a diffusion model. Then, multiple vectors can be computed as a batch.

[0088] A representation initialization unit 1115 initializes an initial image representation 1117. The image representation may be, for example, the image representation described in connection with FIG. 3. That is, the image representation may be a k-dimensional vector, where k is of any size. The representation initialization unit may initialize the image representation as a random regenerated noise generated by a random seed. The representation may be, for example, a latent space of dimension 4, height and width 64x64. That is, the representation may be a (4x64x64) image representation.

[0089] The global noise prediction unit 130 is a unit that is part of a trained neural network. For example, the global noise prediction unit may be part of a diffusion model, such as the stable diffusion model described above. The global noise prediction unit 1130 takes as input a global vector and an image representation and predicts a global noise. The global noise may be a tensor and may have the same dimension (or size) as the input image representation and / or global vector.

[0090] Similarly, the local noise prediction unit 1135 is a unit that is part of the trained neural network. The local noise prediction unit may be part of a diffusion model, such as the stable diffusion model described above. For each of the local features, the local noise prediction unit receives as input a local vector and an image representation and predicts a local noise. The local noise may be a tensor and may have the same dimensions as the input image representation and / or the global vector.

[0091] The global noise prediction unit and the local noise prediction unit may be part of the same trained neural network. For example, both units may be part of the same diffusion model. One exemplary implementation uses a stable diffusion model to predict global and local noise. However, it will be appreciated by those skilled in the art that any trained neural network capable of predicting noise from an input vector and an input representation may be used. Furthermore, different types of diffusion models may be used for global noise prediction and each local noise prediction. That is, a diffusion model operating in pixel space may be used to predict global noise and a diffusion model in latent space may be used to predict local noise. One diffusion model may use a cross-attention mechanism as a conditional mechanism, while another diffusion model may use a different mechanism.

[0092] The predicted global noise and the predicted local noise are input to a noise synthesis unit 1140. The synthesis unit synthesizes the noises into a single synthetic noise. Details of an exemplary noise synthesis process are provided in relation to FIG. 17. The local noise and the global noise are, for example, synthesized by a sum of their tensor representations. The synthesis unit can synthesize the synthetic noise using layout information 1137. For example, the layout information can be used to extract local noises within a region of the initial image representation defined by the layout information. The extracted local noises or the final noise can then be used in a summation to calculate the synthetic noise.

[0093] The synthetic noise is input to the representation update unit 1145 along with the image representation. The representation update unit denoises the image representation using the combined or synthetic noise. For example, the synthetic noise can be used to denoise the image representation using known methods. For example, the image representation is a tensor in the latent space and the synthetic noise is a tensor of the same dimensions as the image representation.

[0094] The dashed box 1155 indicates steps that may be repeated in an iterative process to generate a next image representation. The steps may be repeated N times, for example, the steps may be repeated 30, 40, or 50 times. That is, after the representation update unit generates an updated representation, the updated or next representation may be input to the global and local prediction units along with the global and local vectors, respectively, to predict new noise. The dashed arrows 1157, 1159 indicate the updated representation being input to the noise prediction unit. The next representation, when input to the noise prediction unit, may be referred to as the current representation. That is, the noise prediction unit predicts the noise of the current representation.

[0095] A second dashed box 1160 indicates further units capable of performing the steps of the method described above. Once the representation has been iterated, optionally multiple times, a final updated representation 1147 is generated as output from the representation update unit. The final updated representation is input to an image decoder unit 1150 which decodes the output to generate a generated image 1152. For example, the final updated representation may have dimensions (4x64x64) and the image decoder may decode the representation to generate a generated image having dimensions (3x640x640). The generated image may be output, for example, on a computer monitor or screen and displayed to a user.

[0096] FIG. 12 shows an example 1200 of generating an image from input text and layout information. In this example, the input text and links are represented as a scene graph 1210 for input, but can be input as a complete scene graph or by inputting the information that makes up the scene graph in pieces. In this example, the scene graph also includes layout information. That is, the scene graph is extended with layout information. The scene graph is provided as input (or loaded) along with an initial image representation 1217. The input scene graph may be a user-generated scene graph (also including layout information). Alternatively, the scene graph may be extracted from an image captured by, for example, a camera using a scene graph generative model, or may be a combination of a user-generated scene graph and a camera-generated scene graph. For example, a user may define the layout and bounding box sizes of defined objects on a digital canvas (a drawing space on a screen) along with predicates that describe the relationships and / or interactions between the objects and predicates that describe the objects. The initial image representation may be a representation formed from random noise generated from a random seed.

[0097] The digital canvas may be represented in the form of a graphical user interface. Input text and layout information are entered by user input on a graphical user interface (GUI), with the layout information being entered as bounding boxes on the canvas for each object, and the input text being entered as one or more labels added to each bounding box to name the object. In this example, links between objects are also added to form a scene graph.

[0098] The scene graph is decomposed into global and local features by graph decomposition, which extracts all object nodes and predicates connecting two or more objects into a global graph, and extracts each object node and the predicates or attributes connected only to that object node into a local graph.

[0099] An example of scene graph decomposition is shown in Figure 13. The input scene graph contains the objects dog, man, and another dog. dog and man are connected by a global predicate 'looking'. Also, dog has a local predicate 'sitting'. The other dog is not connected to dog or man, but has local predicates or attributes 'running' and 'black'. Thus, the scene graph is decomposed into global features: 'dog', 'man', and '(other) dog', and dog and man are connected by the predicate looking. In this example, the graph decomposition decomposes the scene graph into three local features. The first local feature 1224a is dog with attributes or predicates black and running, the second local feature 1224b is other dog with attribute sitting, and the third local feature 1224c is man. In this example, the object node 'man' does not have any local attributes that are assigned only to that object. In some examples, object nodes such as "man" that do not have local attributes or local predicates are not included as local features. That is, the object node "man" will only appear in global features. The dashed box 1221 shows an example of graph decomposition. The graph decomposition method is described in more detail in connection with FIG. 13.

[0100] The global features and local features are input to encoders 1225a, 1225b, 1225c, 1225d, respectively. As described in connection with Figures 10 and 11, the encoders may be text encoders, such as the CLIP text encoder, or graph encoders, such as the GNN. As previously described, if a text encoder is used, the local features may first be converted to text sentences using a graph-to-text converter (not shown), such as a rule-based converter. An encoder receives an input and encodes the input into a vector. Thus, in this example, encoder 1225a receives the global features as input and generates a global vector as output. Similarly, encoders 1225b, 1225c, 1225d encode the local features to generate a local vector for each local feature.

[0101] The global vectors are input to a trained neural network. In this example, the trained neural network is a diffusion model. Those skilled in the art will appreciate that any suitable neural network capable of predicting noise can be used. The diffusion model takes as input the global vectors and the current image representation 1217. The diffusion model predicts the global noise for each vector using the representation. The diffusion model similarly takes as input the local vectors and the current representation and predicts the local noise for that local feature. That is, the diffusion model predicts global noise for the global vector, local noise 1 for local vector 1, and local noise 2 for local vector 2. Those skilled in the art will appreciate that the term local noise refers to the noise of the representation computed using the local features, and that in this step of the method the local noise may be a noise prediction across the entire representation, not just the representation assigned to the object node of that local feature.

[0102] In a combining step 1240, the global noise and the local noise are combined into a combined noise. For example, the global noise and the local noise can be summed to combine the noise into a combined noise. That is, the global noise is added to one or more local noises. If the noise is represented as a tensor, the tensors are summed. The noise tensor may be, for example, a tensor in the latent image space where the size of the noise tensor is the same as the latent image representation. Details of the noise construction are provided in relation to FIG. 17.

[0103] The synthetic noise is input to the denoiser 1245 along with the current image representation. The denoiser uses the combined noise to denoise the image representation. For example, the synthetic noise can be subtracted from the image representation. That is, the image representation is a tensor in the latent space, a tensor of the same dimension as the combined noise combined representation. The noise tensor can be subtracted from the representation tensor.

[0104] The current image representation is denoised using the synthetic noise to generate a next image representation. The dashed box 1250 indicates steps that can be repeated iteratively to generate an updated or next image representation. During each iteration, the updated image representation is used at input as the current image representation 1217. Thus, the diffusion model predicts a new noise for each current representation. The steps can be repeated N times, for example 3-5 times. After the fifth iteration, the next image representation can be input to an image decoder (not shown), such as the image decoder described in connection with Figures 10 and 11, to generate the final image.

[0105] Figure 13 shows an example of a graph decomposition method 1300 for use, for example, in the graph decomposition described in relation to Figures 10 to 12. The inventors have discovered that a scene graph 1310 can be decomposed into a global graph, i.e., one or more global features, and one or more local graphs having one or more local features.

[0106] A scene graph is, for example, a user-generated scene graph that includes layout information. A user can define bounding boxes within the layout that define the size and location of desired objects. For example, a user may be presented with a layout or canvas with a working area that is 512x512 pixels in size. Of course, the canvas can be of any size, and preferably has the same aspect ratio as the generated image (e.g., 1:1 or 16:9). For example, from a box drawn on the canvas, the x and y ratios of the top left and bottom right of the box can be calculated. For example, if the canvas size is 100x200, and the top left of the box is at (50,50) and the size is 20x10, then the ratio of the top left is (0.5,0.25) and the bottom right is (0.7,0.3). From these ratios, the location of the box within the image representation (64x64) can be calculated.

[0107] The user can draw a bounding box with dimensions 90x120 on the left side of the layout. The user can label the box with "dog" as the input text. The user can connect or link the bounding boxes of dog using the predicate or description "sitting". That is, the attribute or property "sitting" is provided as a labeled link. In the center of the canvas, the user can draw another bounding box with dimensions 200x200 and label the bounding box "man". The user can link or connect the object nodes "dog" and "man" using the predicate "looking". The labeled link "looking" defines the relationship between the two objects. Furthermore, the user can draw a third bounding box with dimensions 100x80 and label "dog" on the right side of the canvas. This object node can be assigned the predicates "black" and "running". Of course, the bounding boxes can also be specified automatically based on the objects and their relationships. For example, the user can provide the input text "dog" and the bounding boxes are generated automatically.

[0108] The above example can be realised in Figure 13 as follows: The dog on the left can be labelled O-0, the man in the middle O-1 and the dog on the right O-2. The global predicate describing or linking the interaction between the dog and the man is given the labelled link P-0 ("looking at"). The local predicate or attribute describing the dog on the left as "sitting" is given the labelled link P-1. The local predicate describing the dog on the right as "running" is labelled P-2 and the local predicate labelling the dog on the right as "black" is labelled P-3.

[0109] In general, a scene graph is a graph that represents a particular combination of at least one object and at least one relationship and / or action (or "predicate"). For example, a scene graph may represent objects in an image and their relationships to each other. That is, a scene graph may correspond to a logical / spatial representation of a graphical scene. Objects in a scene, such as dogs, people (i.e., common nouns), are formed from object nodes, and relationships, actions or descriptors between objects, e.g., predicates such as "looking", "sitting", "playing", "black", "tall", etc., form predicate nodes that connect the object nodes or are connected to a single object node. In some scene graphs, nodes are assigned only to objects, and predicates are labels along edges that connect the objects. Those skilled in the art will appreciate that any suitable scene graph can be used with the image generation methods disclosed herein.

[0110] A scene graph may be a directed graph where nodes represent objects and edges represent pairwise relationships between the objects. An example of two objects and a pairwise relationship is "a man playing with a dog", where man and dog are object nodes and the pairwise relationship or predicate is "playing with". Another example is "a man looking at a dog", where the predicate is the term "looking at". To indicate this relationship in a scene graph, an arrow can be drawn from the man to the dog (or in any other manner used in other notations) to indicate that the man is doing the "looking" and not the other way around.

[0111] Thus, a scene graph is a structural representation that captures detailed semantics that model objects ("man", "dog", "car"), object attributes or local predicates ("man is tall", "car is red"), and relationships between paired objects or global predicates ("man is sitting in car"). The elements of a scene graph are thus object nodes and predicate nodes (or in some cases predicate edges).

[0112] Objects or subjects of a scene graph can be enclosed by bounding boxes that delimit object nodes. Each object can have zero, one or more attributes or local predicates such as color, state (e.g., sitting, looking), material (e.g., metallic, plastic) and / or emotion (e.g., smiling, crying). Additionally or alternatively, objects may have relationships such as "jumping over" or "sitting on" and these relationships may be spatial ("in front of", "behind"), the preposition "with", the comparative "taller than", the descriptive verb "strolled", etc. Thus, a scene graph can be thought of as a set of visual relation triplets of the form "object, relation, object" and / or "object, is, predicate (or attribute)". In the methods disclosed herein, attributes and relations can be provided as labeled links between objects, or links connected to or linking objects.

[0113] In the methods disclosed herein, it is understood that a scene graph may also include layout information, i.e., a scene graph may define object nodes and labeled links (or predicate nodes), as well as the layout of objects, for example, within a canvas. Thus, a scene graph may be referred to as a scene graph including layout information.

[0114] Additionally, a scene graph generative model can be used, which can be trained on a large number of scene graph and image pairs. The model can be used, for example, to predict the relationship between two input objects or bounding boxes. Additionally or alternatively, for example, if a box with a label of dog is input and the predicate "near by" is added, the model can predict candidates from the objects most likely to be near the dog, e.g., person, stick, or frisbee. If two objects are input (e.g., dog and man) and the relationship between the objects is given, the model can predict the location and layout of the objects. Given a predicate, a large language model can be used to predict suitable candidates (i.e., objects) for the predicate.

[0115] The scene graph is decomposed into a global graph 1322 and one or more local graphs 1324a, 1324b, 1324c. A global graph is extracted 1320a, for example, by first detecting whether each node is an object or a predicate. For example, a user interacting with the GUI can explicitly select whether they want to add an object or a predicate, or from text input, a neural network such as a language model can be used to classify each "node" as an object or a predicate. Thus, the global graph can be extracted by detecting or determining all object nodes and connections (or links or predicates) between the object nodes. Similarly, one or more local graphs are extracted 1320b, for example, by detecting (or finding) the object nodes and all predicates connected to a (single) node from the scene graph and setting the object node as, for example, the root node of the new local graph. The global background can also be extracted from the scene graph or included as additional input text together with the scene graph.

[0116] As mentioned above, the global feature or global map includes all object nodes (i.e., O-0, O-1, O-2) and all predicate nodes (i.e., P-0) connected to two or more object nodes. One or more local features or local feature maps each include only one (object) node, e.g., O-1, and all predicate nodes (i.e., P-1) connected to the object node. As shown in FIG. 13, the global map can hold the layout of the object nodes on the canvas. That is, the global map encodes both the object nodes, their relationships and attributes between the object nodes, and the layout of the objects. As described above in relation to FIG. 12 and the “man” object node 1224c, the local graph 1 can be removed from the local features because it does not have a local predicate, i.e., an attribute associated only with that object. The object node O-1 in the local graph, and the position and layout of the object nodes have already been considered globally in the global graph 1.

[0117] As mentioned above, all object nodes in the local graph are included in the global graph. The global graph consists of all object nodes and the labeled links or connections between them. The global graph is used to guide the image so that the objects and the relationships between them are included in the generated image. The local graph is used to guide the image so that each object in the image has its specified attributes.

[0118] Fig. 14 shows an example 1400 of converting an input scene graph 1410 into a textual output describing a global graph and a local graph. The scene graph is first decomposed 1420 into a global graph and a local graph, for example using the method described in relation to Fig. 13. In the graph decomposition step, a global graph 1422 is extracted, which contains, for example, information about objects, predicates linking two or more objects, and the layout of the objects. As described above, the graph decomposition generates a local graph 1423a, 1424b for each of the local features. The graph decomposition also generates a layout or layout map 1426a, 1426b, layout 0 and layout 2 in the figure, for each of the object nodes.

[0119] The layout map defines the location of object nodes on the canvas. An exemplary bounding box 1428 is shown in the figure. The location of the bounding box may be defined using a coordinate system, such as a Cartesian coordinate system. For example, the zero point or origin, i.e., the point with coordinates (0,0), may be defined as the location of the bottom left corner of the canvas 1430. The location or layout of the bounding box may be defined using the top left and bottom right locations of the box, such as ((x1,y1)(x2,y2)). Additionally or alternatively, the layout of the bounding box may be defined using the location of the top left corner of the box and the size of the box. Those skilled in the art will appreciate that any suitable method of using a coordinate system may be used to define the location of object bounding boxes on the canvas. A scene graph may include bounding boxes, which may be input by a user and are described above. That is, a user may draw a bounding box and input text for the box as a node. The text may be a label for the bounding box, for example, giving the bounding box the label "man".

[0120] The scene graph can be converted into text sentences, for example for use in a text encoder. That is, the text generated from the global graph and the local graph can be input to the text encoder described in connection with Figures 10 to 12. The global graph and the local graph can be converted into text using graph-to-text converters 1432a, 1432b, 1432c. For example, the graph-to-text converters described above can be used. In this example, the global graph 1422 is converted into global text 1434a "a dog is playing with a ball and a man". Object node O-0 represents a dog, object node O-1 represents a ball, and object node O-2 represents a man. Predicate P-0 is the action "playing". The first local feature is decomposed into local graph 0 1424a and layout 0 1426a. The coordinates of the layout can be defined as above, i.e. ((x1, y1)(x2, y2)). Local graph 0 is converted into text 1432b using graph-to-text conversion. In this example, the local feature is dog with the attribute "black", so the local graph is transformed into the local text 0 1434b "a dog is black".

[0121] Similarly, the second local feature is decomposed into local graph 2 1424b and layout 2 1426a. The layout coordinates can be defined as above, i.e. ((x1, y1)(x2, y2)). Local graph 2 is converted to text using graph-to-text transformation 1432c. In this example, the local feature is a man with attributes "smiling" and "standing". Thus, local graph 2 is converted to local text 2 1434c "a man is smiling and standing".

[0122] In this example, a local graph is not generated for ball because "ball" has only a global predicate associated with it, i.e., ball is linked only to dog by the action "playing". In some examples, a local graph may be generated for ball. In these examples, the text sentence to be generated is "ball".

[0123] From the text input, the diffusion model knows, for example, how many dogs there should be in the image, and if a particular region in the image is guided by local noise to be a dog, the diffusion model takes that into account and predicts the global noise. The global and local noises are connected based on the layout. If the global text is "a dog is playing with another dog", the global noise does not consider the individual attributes of each dog, but instead considers the region of each dog (i.e., one dog is in region A and another dog is in region B). If the local text is "a dog is black" for region A and "a dog is running" for region B, the local noise guides the details of each dog (i.e., the dog in region A should be black and the dog in region B should be running).

[0124] FIG. 15 is an example of a functional diagram 1500 for predicting noise from global and local text inputs. A blank text 1510a may be input to a text encoder 1525. The text encoder may be, for example, the encoder of FIG. 10 to FIG. 12. The text encoder generates an unconditional vector 1515a from the blank text. For example, the blank text input may be converted into a vector consisting entirely of padding. The padding may be assigned a token by a tokenizer before being encoded by the text encoder. For example, the maximum length of the input text may be set as 5 words, and each word may be assigned its own ID. For example, when the text "a dog is black" is input, the text encoder may convert this into a vector with each word ID [0, 1, 2, 3, 100], where 100 is the ID of the pad. Thus, the blank text may have the vector [100, 100, 100, 100, 100]. The ID of the pad can typically be set by, for example, a text tokenizer, where the ID is 0 or 100, or any other number set in the encoding.

[0125] As mentioned above, the diffusion model used by the inventors was a conditional diffusion model (although of course other suitable neural networks may be used). Thus, in this example, the data distribution of the trained model may be conditional on the class identifier or input. A null vector is input to a diffusion model 1530, e.g., a conditional diffusion model, to generate an unconditional noise 1520a. The unconditional noise can be used to enhance the conditional noise 1520b, 1520c, 1520d (computed from the text inputs, i.e., global text, local text0, and local text2) for denoising the current representation. Using unconditional noise to enhance the conditional noise is a technique known as classifier-free guidance. An example of an unconditional representation 1540 is shown in FIG. 15. The unconditional representation is an image representation of the unconditional noise nc, which shows the predicted noise generated by the diffusion model without the conditional modality.

[0126] The conditional diffusion model uses the conditional data distribution as follows:

number

number

number

number

number

number

number

[0127] In this example, global text 1510b is input to the text encoder to generate a global vector 1515b. That is, one or more global features are encoded into the global vector. Similarly, local text 0, 1510c is input to the text encoder to generate a local vector 0, 1510c. Local text 2, 1510d is input to the text encoder to generate a local vector 1515d. That is, one or more local features are encoded into the local vector. The text encoder may be, for example, the text encoder described in connection with FIG. 2. That is, the text encoder may be a CLIP text encoder. Those skilled in the art will appreciate that before inputting the text into the text encoder, the text may be tokenized, for example, using a CLIP tokenizer. Furthermore, those skilled in the art will appreciate that the text vector may have word embeddings. For example, the dimensionality of the text encoder, encoder, and pooler layer may be 768, and the maximum token input may be 77. For example, the maximum embedding size of the ViT used by CLIP is 768. Thus, the maximum embedding dimensionality may be 768×77.

[0128] The unconditional, global and local vectors are input to a diffusion model 1530 along with the current representation 1535. The diffusion model then predicts noise for each of the input vectors. For example, the diffusion model predicts unconditional noise 1520a for unconditional vector 1515a, global noise 1520b for global vector 1515b, local noise 0 1520c for local vector 0 1510c and local noise 2 1520d for local vector 2 1515d. An example of the unconditional representation 1540 is shown in Figure 16. The unconditional representation is an image representation of the unconditional noise nc and shows the predicted noise generated by the diffusion model without the conditioned modality.

[0129] Unconditional noise drives the image representation towards random images (random in the sense that the user does not define the output image), global noise drives images to represent global text, e.g., "a dog and a man", and each local noise drives images to represent text associated with its local text, e.g., "a dog is black" and / or "a man is sitting". Thus, the method allows control of both local features (visual attributes) and global features (relationships between objects, object positions).

[0130] Although all vectors are shown as being input to the diffusion model together, one skilled in the art will appreciate that the diffusion model processes the vectors separately to predict the noise for each vector.

[0131] FIG. 16 is an example of a functional diagram 1600 for predicting noise from global and local graph inputs. A whitespace graph 1610a may be input to a graph encoder 1625. The graph encoder may be, for example, the encoder of FIG. 10 to FIG. 12. The graph encoder generates an unconditional vector 1615a from the whitespace graph. For example, the whitespace graph input is converted to a vector entirely composed of padding. The padding can be assigned tokens by a tokenizer before being encoded by the text encoder. Similar to generating vectors from whitespace text above, an unconditional vector can be generated from the whitespace graph. The whitespace graph can be generated with the blank nodes converted to special IDs. The whitespace graph can be encoded using a graph encoder to generate an unconditional vector.

[0132] As mentioned above, the diffusion model we use may be a conditional diffusion model that operates in the latent space, such as the stable diffusion model by stabilityAi. The diffusion model of FIG. 16 may be identical or substantially similar to the diffusion model described above in relation to FIG. 15. The vector output generated by the graph encoder may be substantially identical to the vector output generated by the text encoder. Thus, for example, the unconditional vectors may be generated using a text encoder, and a graph encoder may be used for the remaining inputs.

[0133] In this example, global graph 1610b is input to a graph encoder to generate global vector 1615b, i.e., one or more global features are encoded into a global vector. Similarly, local graph 0, 1610c is input to a graph encoder to generate local vector 0, 1610c. Local graph 2, 1610d is input to a text encoder to generate local vector 1615d, i.e., one or more local features are each encoded into a local vector. Graphical Neural can be used as a graph encoder. Of course, any graph encoder can be used, such as a text graph transformer.

[0134] Those skilled in the art will appreciate that graph nodes and predicates can be tokenized, for example, using the CLIP tokenizer, before inputting the graph to the graph encoder. Furthermore, those skilled in the art will appreciate that the graph vectors may have embeddings. For example, the dimensionality of the graph encoder, encoder, and pooler layers may be 768, and the maximum token input may be 77. For example, the maximum embedding size of ViT used by CLIP is 768. Thus, the maximum embedding dimensionality may be 77×768.

[0135] In this example, the unconditional vector, the global vector, and the local vector are input to a diffusion model 1630 along with a current representation 1635. Although all vectors are shown as being input to the diffusion model together, one skilled in the art will understand that the diffusion model processes the vectors separately to estimate the noise for each vector.

[0136] The unconditional, global, and local vectors are input to a diffusion model 1630 along with a current representation 1635. The diffusion model then predicts noise for each of the input vectors. For example, the diffusion model predicts an unconditional noise (or unconditioned noise) 1620a for the unconditional vector 1615a, a global noise 1620b for the global vector 1615b, a local noise 0 1620c for local vector 0 1610c, and a local noise 2 1620d for local vector 2 1615d. An example of an unconditional representation 1640 is shown in Figure 16. The unconditional representation is an image representation of the unconditional noise, nc, and shows the predicted noise generated by the diffusion model without the conditioned modality.

[0137] As illustrated in the text encoder example of Figure 15, unconditional noise leads the image representation to be a random image (random in the sense that the user has not defined the output image), global noise leads the image to represent a global graph, and local noise leads the image to represent the text associated with that local graph.

[0138] Although all vectors are shown as being input to the diffusion model together, one skilled in the art will appreciate that the diffusion model can process the vectors separately to predict the noise for each vector.

[0139] 17 is an example 1700 of combining predicted global and local noises into a combined noise. This example of combining noises may be performed, for example, in step s1040 of FIG. 10, in the noise combination unit 1140 illustrated in FIG. 11, and / or in step 1240 of FIG. 12.

[0140] The unconditional noise, global noise, and / or local noise may be generated using the steps described in relation to Figure 15 and / or Figure 16. That is, the noise may be generated by a conditional diffusion model.

[0141] Step 1715a shows how the global guide gg is computed. The global guide is the difference between the global (conditional) noise ng and the unconditional noise nc, and is used in a classifier free computation of noise. The global guide may be multiplied by some factor cg to emphasize the condition of the global guide. The factor may be a hyperparameter. The hyperparameter may be user defined. The best practice hyperparameter may be 7.5. The inventors used the best practice hyperparameter. Of course, other values ​​of the hyperparameter may be used. See, for example, FIG. 23.

[0142] The enhanced global guides can be added to the unconditioned noise to compute the final global noise 1720a. A representation of the final global noise 1722a is shown in Figure 16. In this example, the representation is a 6x6 representation with 36 squares or patches. Each square represents the predicted global noise for that segment.

[0143] Step 1715b shows how to determine (or calculate) the final local noise or local guide gl0. The local guide is the difference between the initial local noise 0 1710c and the global noise ng, which may be multiplied by a coefficient cl0. The coefficient cl0 may be another hyperparameter. This hyperparameter may be user-defined. The inventors used best practice hyperparameters from 3 to 8. Thus, the final local noise for each of one or more local features may be determined using the initial local noise (the local noise predicted by the diffusion model) and the global noise.

[0144] Layout 0 1712 (i.e., layout information) may be used to extract final local noise 0 1720b. Layout 0 is, for example, the layout of one of the one or more local features. That is, layout 0 defines the location of one of the object nodes mentioned above. For example, the layout of the object nodes is defined with reference to FIG. 14. That is, for example, the location of the object nodes is defined by a bounding box. The location of the bounding box may be defined using a coordinate system such as a Cartesian coordinate system. For example, the zero point or origin, i.e., the point with coordinates (0,0), may be defined as the location of the bottom left corner of the canvas 1430. The location or layout of the bounding box may be defined using the top left and bottom right locations of the box, such as ((x1,y1)(x2,y2)). Those skilled in the art will appreciate that any suitable method of using a coordinate system may be used to define the location of an object bounding box on a canvas.

[0145] As an example, layout 0 is (0.1, 0.2, 0.2, 0.8), i.e., the ratios (x1, y1, x2, y2) for width and height as shown in FIG. 14. If the size of the local guide gl is 60x60, then x1=6, y1=12, x2=12, y2=48. The noise of the local guide in the region (6, 12, 12, 48) is extracted to generate the final local noise 01622b.

[0146] In one example, all the above calculations can be performed in latent space with the noise and guide represented as tensors. The layout tensor can be a tensor with non-integer values ​​in the region that defines layout 0 and zero or null values ​​filling the remaining part of the tensor. For example, the noise synthesis unit can detect the populated values ​​in the layout 0 tensor and map these populated values ​​to the local guide tensor. Thus, the synthesis unit can extract the correct noise values ​​in the guide tensor that correspond to the layout of the object node. That is, in this example, the layout tensor can act as a mask. Of course, any other method can be used to extract the noise in the guide that is related to the layout, i.e., the position, of the object node. In this example, the dimensions of the local guide 0 and layout 0 are the same size. For example, the dimensions of the guide and layout can be 64x64.

[0147] In another example, the layout is converted to an index into a tensor. For example, a 60x60 layout of (0.1, 0.2, 0.4, 0.4) is converted to indices (6, 12, 24, 24) (representing row start, column start, row end, column end). Local noise can only be added to the global noise within the range of these indices. Noise and guide can be specified as tensors in the format 4x64x64, no conversion is required.

[0148] In this example, both final local noise 0 and final local noise 2 1722c are extracted using layout information. Final local noise 2 can be extracted using the same method as final local noise 0. That is, local guide gl2 can be calculated by subtracting local noise 2 from global noise and multiplying the difference by coefficient cl2 1715c. The coefficient may be yet another hyperparameter. For example, the hyperparameter may be user-defined. The inventors used best practice hyperparameters 3-8. As shown, layout information is used in the extraction of the final local noise.

[0149] In this example, the final global noise, the final local noise 0, and the final local noise 2 are combined into a combined noise n 1725. The noises are combined, for example, by adding the noises together. For example, the noises may be tensors of the same size and may be added by tensor addition. For example, the sum may be a weighted sum where each noise is weighted by its respective hyperparameter (i.e., cl0, cl1, and cl2). Of course, the noises may be given as different representations and still be summed together. Of course, other combination methods than summation may be used. In a further example, a global background and / or additional features may be loaded into the method. A global background noise and additional feature noises may be determined and combined into a combined noise. See, for example, FIG. 22.

[0150] Fig. 18 shows an example 1800 of generating an image from an input scene graph 1810. The example 1800 provides example values ​​for the image generation process described in relation to Fig. 12. Accordingly, similar reference numbers are used to describe the same or substantially the same elements in the figures. That is, the input scene graph is given reference number 1210 in Fig. 12 and reference number 1820 in Fig. 18. The process of generating an image from an input scene graph is described in Fig. 12 and will not be repeated in detail here. The process of generating an image described in Fig. 12 is incorporated herein.

[0151] In this example, a scene graph is given as input, e.g., the scene graph is a user-generated scene graph. That is, in this example, the input text and layout information are loaded as a scene graph. The scene graph can define object nodes and predicates, as well as the layout of the object nodes on a canvas. The canvas size can be, for example, a 640x640 (height x width) (or 512x512, etc.) canvas. The user can draw a bounding box for each object node to set the size of that object in the final image.

[0152] The scene graph is decomposed into global features 1822 and local features 1824a, 1824b, 1824c using a graph decomposition 1820. The layout of the object nodes is preserved in the graph decomposition. The global features are encoded into global vectors, for example using an encoder 1825a. The encoder may be a text encoder and / or a graph encoder, as described herein. The global vectors may be output in the form of (DxN), where D is the dimension of the vector and N is the maximum token size. For example, D is 768 and N is 50. In this example, the encoded global vectors are in the form of (768x50). Of course, the dimension and maximum number of nodes may differ from this defined value. For example, the inventors have found that a maximum number of nodes or token size between 50 and 100 is suitable, e.g., 77.

[0153] Similar to the global features, each of the local features 1824a, 1824b, 1824c is encoded into a local vector, for example, using an encoder 1825b, 1825c, 1825d. The encoder may be a text encoder and / or a graph encoder, as described herein. Each of the local vectors may be output in the form of (DxN). In this example, the local vectors are output in the form of (768x50). That is, in this example, the local vectors are the same size as each other and the same size as the global vector. To generate global and local vectors (or tensors) of the same length (i.e., the same tensor size), the inputs can be padded. For example, one local feature can have the input "[a][dog][is][white][and][running]" and a second local feature can have the input "[a][man][is][black]". To ensure that the inputs generate outputs of the same size, the second input is padded so that the sentence is "[a][man][is][black][PAD][PAD]". By ensuring that the output vectors are the same size, multiple inputs, i.e. multiple text sentences, can be computed as a batch.

[0154] The global noise 1835 is predicted from one or more global features. The global noise is predicted from the global vectors and the initial image representation 1817 using a trained neural network. In this example, the trained neural network is a diffusion model 1830a. The predicted global noise may be of the form (dxHxW), where d is the dimension of the image representation, H is the height of the representation, and W is the width of the representation. For example, the global noise may have a size (4x64x64). As mentioned above, the diffusion model is a latent diffusion model and thus may operate in a latent space. Thus, the size of the global noise may be the size of the latent space.

[0155] The initial image representation may also have a size (dxHxW). For example, the image representation may have a size (4x64x64). That is, the global noise and the image representation may have the same size.

[0156] Similarly, a local noise 1835b, 1835c, 1835d is predicted for each of the one or more local features. Each local noise is predicted from the respective local vector and the initial image representation using a trained neural network. In this example, the trained neural network is a diffusion model 1830b, 1830c, 1830d. ​​The predicted local noise may be of the form (dxhixwi), where d is the dimension of the image representation, hi is the height of the i-th local region of representation b (i.e., the height of the bounding box of the object node), and wi is the width of the i-th local region of representation b (i.e., the width of the bounding box of the object node).

[0157] In this example, local noise 1835b of local feature 1824a "dog, black, running" has size (4x52x22), local noise 1835c of local feature 1824b "dog, sitting" has size (4x54x22), and local noise 1835d of local feature "man" has size (4x32x16). The difference in size between dog and man correlates to the user-defined size of the bounding box of each object node in the scene graph. For example, the bounding box of dog is larger than the bounding box of man. As explained in the example of FIG. 12, man does not have any local attributes or predicates, so there is no need to specify this object node as a local feature. All information about the man object node is already contained in the global graph.

[0158] In this example, the local noises 1835b, 1835c, 1835d are correlated with the final local noises 1722b, 1722c, 1722d of FIG. 17. That is, before the global noise and the local noise are combined into a combined noise, the noise can be processed according to the method of FIG. 17. That is, the final local noise is determined using the initial local noise and the global noise. While the diffusion model calculates the local noise of the entire image representation, the final local noise can only represent the noise defined by the bounding boxes of the object nodes they represent.

[0159] The global noise and the local noise are combined 1840 to form a combined or composite noise 1837. In this example, the combined noise has dimensions (dxHxW), i.e., the composite noise has the same size as the image representation. The combined noise may be a tensor representation of the noise in the latent space.

[0160] The combined noise is used to denoise the (current) image representation 1845. For example, to denoise the image representation, the combined noise can be subtracted from the image representation. For example, both the combined noise and the image representation may be tensors of size (dxHxW). To denoise the image representation, the noise tensor may be subtracted from the image tensor. The denoised current representation may be referred to as the next representation 1847. The next representation may have the same size as the current representation. That is, the next representation may have size (dxHxW), which in this example is (4x64x64).

[0161] The dashed box 1850 indicates steps that may be repeated iteratively to generate an updated or next image representation. During each iteration, the updated image representation is used at the input as the current image representation. Thus, the diffusion model predicts new noise for each current representation. The steps may be repeated N times, for example 3-5 times. After the fifth iteration, the next image representation may be input to an image decoder 1855 to generate a final generated image 1860. The image decoder may upscale the resolution of the final image representation. For example, the final image representation may have dimensions of 4x64x64 (dxHxW) and the generated image may have dimensions of 3x620x620 (RGBxHxW). Of course, the dimensions of the image representations and / or the final image may be of different sizes.

[0162] Figure 19 shows an example image 1900 generated by the method described herein. The scene graph 1910a shows the bounding boxes for a sheep standing and a man sitting, the man wearing a blue shirt. The layout of the scene graph shows the man to the left of the man. The method 1920a produces a final image representation 1930a that nicely depicts a photorealistic image of the sheep and the man. The method generated the background automatically, but the user can also input a global background. See Figure 20 below.

[0163] The scene graph 1910b shows the bounding box of a dog where the predicates running and white describe the dog. The method 1920b produces a final image representation 1930b that successfully depicts a photorealistic image of "a white dog running". The red bounding box 1935b indicates that the method successfully produced the image using the position and layout specified in the scene graph.

[0164] The scene graph 1910c shows the bounding box of the dog group and the bounding box of the man. The bounding box of the man and the dog group overlap. The method 1920c produces a final image representation 1930c that correctly represents the image of the dog group and the man. The outline of the man is highlighted in black and the dog is highlighted in yellow.

[0165] FIG. 20 is a further example of a method 2000 for generating an image from input text and layout information, but also including the concept of layers. The reader is therefore referred to the previous description, which is supplemented below. Further examples include the methods described in relation to FIGS. 10-18, in which the input text may be further decomposed into background global features. That is, in this method, the loaded input text may specify a background, e.g., a location or a scene or a style. For example, in the example scene graph 1910a described in FIG. 19, a user may further specify the background as "mountains".

[0166] In a further example method, a global background is extracted, and further method steps may include encoding the global background into a global background vector, predicting a global background noise from the background global vector and the initial image representation using a trained neural network, and combining the predicted global background noise, the predicted global noise, and the final local noise into a combined noise using layout information. The reader will understand that in this example method, the predicted global background noise is not used to predict the global noise, but the global noise is used to predict the local noise.

[0167] Fig. 20 shows an example input including input text t1, t2, t3, t4, layout information a1, a2, a3, a4, a5, and layers l1, l2, l3. In this example, the global background 2010 is on a first layer (global background layer) l1, the input text representing global features 2012 is on a second layer l2 (global feature layer), and local features 2014 and further features 2016 are on a third layer l3 (successive layers). The layers can be understood as "concept" layers that exist only for the purpose of explaining how noise is calculated for the input text. They can be defined by the user.

[0168] Each input text can be decomposed, for example, using a decomposition unit, as described in connection with Figures 10 to 18. In this example, the input text is decomposed into a global feature "a man is playing with a dog", a global background "in a park", a local feature "a dog is brown", and a further feature "a shirt is blue". The further features (on additional layers) are described in more detail below. In this example, the further feature belongs to the local feature "a man" and indicates that the man is wearing a blue shirt.

[0169] Similar to the method described in Fig. 10, the input text may be encoded using, for example, a text and / or graph encoder 2025. The encoding vectors are input to a diffusion model 2030, such as a stable diffusion model, to predict noise for each text. The predicted noise 2035 for the local, global and background features (and further features in this example) are combined into a combined noise 2040 using the layout information. An exemplary noise determination and final noise combination is described in relation to Fig. 22 below.

[0170] As in Fig. 10, the initial image representation is updated by denoising it with the synthetic noise. If the image representation is given by a tensor, e.g., if the tensor is a latent space, the combined noise may also be given as a tensor of the same dimension as the representation tensor. The steps may be repeated multiple times, e.g., 50 times, and in a generation step (not shown), a final image may be generated from the final image representation using an image decoder.

[0171] Figure 21 shows an example of text input and layout information with layers. The input may be a user input using a graphical user interface. See, for example, Figure 25.

[0172] In this example, the global background 2110 is input as "in the office", the global features 2112a, 2112b, 2112c are input as "woman", "talking with", and "man", the local features 2014 are input as "woman having a shirt with a brooch", and further features 2016 are input as "brooch", "red and like butterfly".

[0173] Further features may be "overlapping" features of a local feature, i.e. in this example the bounding box B1 of the further feature (i.e. brooch) is smaller than the outer bounding box B2 (i.e. woman). In other words B1 is completely surrounded by B2 and B1 is inside B2. Further features are details of the local feature, e.g. brooch is a detail of woman.

[0174] FIG. 21 shows the global background features, the global features, the local features, and the further features each on separate layers l1, l2, l3, l4. The further method 2000 described in relation to FIG. 20 can include a further decomposition step in which the features are separated (decomposed) into layers. For example, from the input 2100, the method can automatically detect the global background features and generate a global background layer. Similarly, the method can detect the global features and generate a global feature layer, detect the local features and generate a local feature layer, and detect the further features and generate a further feature layer. Alternatively, the layers can be generated by the user. As shown in FIG. 21, the global features and the local features can each occupy separate conceptual successive layers (i.e., each occupies a separate layer). The input can be, for example, the scene graph shown in FIG. 18, with an additional text input for the global background.

[0175] As mentioned above, bounding box B1 may be a further feature of B2. B1 may be considered to be local to B2. In this example, brooch defined by bounding box B1 is on the fourth layer l4, and woman defined by bounding box B2 is on the third layer l3. More generally, if B2 is on layer t, then B1 may be on layer t+1. Layer t+1 may overlap with layer t since B1 is a further feature of B2. That is, further features of a local feature may occupy successive conceptual layers that are overlaid on the conceptual layer of that local feature. If a third bounding box B3 exists, for example for man in the figure, then bounding boxes B1 and B3 may be on the same layer since the bounding box does not overlap with bounding box B1, i.e. man is not a further feature of woman. Alternatively, man bounding box B3 may be on the same layer as brooch bounding box B1 since brooch is not a further feature of man. Those skilled in the art will appreciate that the layers containing the local feature and the further feature need not be adjacent layers, i.e. the further feature may be on layer t and the local feature on layer t+2 or t+3 or t-2. Because the further feature overlaps the local feature, the further feature layer and the local feature layer may be said to be overlapping layers.

[0176] As mentioned above, the method may include decomposing or separating the input into layers. Additionally or alternatively, the layers may be input by a user. That is, the user may define each layer and specify whether each layer includes global features, local features, further features, and / or a global background. The user may generate each layer using a graphical user interface (GUI), such as the GUI described in connection with FIG. 25 below.

[0177] 22 shows an example 2200 of determining a global background noise 2220a, a global noise 2220b, and a final local noise 2220c, 2220d, 2220e for each of one or more local features, and combining the noises into a combined noise 2230. This example of combining noises may be performed, for example, in step s1040 of FIG. 10, the noise combination unit 1140 illustrated in FIG. 11, and / or in step 1240 of FIG. 12.

[0178] An example of a loaded input 2210 including input text and layout information is shown in Figure 22. The unconditional noise nc, the global noise n2, and / or the local noise n3, n4 may be generated using the steps described in relation to Figures 15 and / or 16. That is, the noise may be generated by a diffusion model.

[0179] Further, this example shows an exploded view of each layer 2215 in the loaded input. It includes a global background layer l1 with a global background and a further feature layer l4 with further features of the local features. The global background noise n1 and the further feature noise n5 may be predicted using the same or similar methods as those described in Figures 15 and 16. Figure 22 also shows the global noise n2 and the initial local noise n3, n4 for each of the first and second local features. The input shown in Figure 22 may be the same or substantially similar to the input shown in Figure 21. That is, the input may include "a woman talking with a man in an office, the woman having a shirt with a brooch and the brooch is red and like a butterfly".

[0180] As mentioned above, the input text can be further decomposed into a global background. The global background can be encoded into a global background vector. Then, the global background noise n1 can be predicted from the background global vector and the initial image representation using a trained neural network. Furthermore, the synthesis step, for example synthesis step s1040, can further include synthesizing the global background noise into a combined noise. That is, the combined noise can include a global feature noise, a final local noise of each of the local features, and a global background noise. The combined noise can additionally or alternatively include further feature noise. An example of the combined noise is the final noise 2230 shown in the figure.

[0181] As in FIG. 17, for global features (i.e., from the global vector), a global noise (or global guide) gg2 can be calculated. The global noise gg2 is, for example, the difference between the global (conditional) noise n2 and the unconditional noise nc, and is used in classifier-free calculation of noise. To emphasize the global noise condition, the global guide may be multiplied by some coefficient c21. The coefficient may be a hyperparameter. The hyperparameter may be user-defined. A best practice hyperparameter may be between 3 and 8, e.g., 7.5. Thus, the global guide can be calculated using the formula gg2=(n2-nc)*c21.

[0182] Furthermore, a global background noise (global background guide) gg1 may be calculated. For example, the global background guide is the difference between the global background noise n1 and the unconditional noise nc, and is used in classifier-free calculation of noise. To emphasize the condition of the global background noise, the global background noise may be multiplied by another coefficient c11. The coefficient c11 may be a hyperparameter. The hyperparameter may be user-defined. A best practice hyperparameter may be between 3 and 8, for example 7.5. Thus, the global background guide can be calculated using the formula gg1=(n1-nc)*c11.

[0183] The combined global noise (not shown) can be calculated by adding the unconditional noise to the global guide gg2 and the global background guide gg1. That is, the combined global noise can be calculated as ngf=nc+gg1+gg2. The combined global noise can be determined (or calculated) as part of the synthesis step. In this example, the combined global noise is determined in the final noise 2230 and constitutes the 13th entry in the summation (i.e., final noise=nc+gg1+gg2...).

[0184] FIG. 22 also shows the steps of determining a final local noise or final local guide gl3, gl4 for each of the local features, and determining an additional feature noise gl5 of the additional feature of the local feature. In this example, the additional feature, e.g., brooch in FIG. 21, is on layer 4, and the local feature that overlaps with the additional feature, e.g., woman, is on layer 3. Thus, the additional feature of the local feature occupies a separate layer from the local feature. As described above, the additional feature may be encoded into an additional feature vector, and the initial additional feature noise may be predicted from the additional feature and the initial image representation using a trained neural network, e.g., a diffusion model.

[0185] The final further feature noise gl5 may be calculated using the initial further feature noise and the local feature noise, i.e., the local features that overlap with the further feature. For example, the final further feature noise may be calculated by subtracting the local feature noise n3 from the initial further feature noise n3 and optionally multiplying the difference by the hyper-parameter c41. The final feature noise may be included in a synthesis step, i.e., the synthesis step may include synthesizing the global noise, the local noise and the final further feature noise into a combined noise, which may additionally or alternatively include a global background noise.

[0186] In this example, the final local noises gl3, gl4 are calculated using the method described in relation to Fig. 17. That is, to determine (or calculate) the final local noise gl3, the difference between the initial local noise n3 and the global noise n2 is determined. This difference can then be multiplied by a hyperparameter c31. As mentioned above, this hyperparameter may be user-defined and may be between 3 and 8.

[0187] As described in relation to Fig. 17, the layout information a3, a4, a5 can be used to extract the final noise from the initial local noise. For example, in the synthesis step, the final local noise gl3 can be multiplied by the layout information a3 to extract noise related to the position and / or size of the bounding box defined by the layout information. Thus, for example, the final local noise can be determined by multiplying the local noise gl3 by the layout information a3. Similar calculations can be performed for the local noise gl4 using a4 and for the further feature noise gl5 using a5.

[0188] As a more general example of determining the final noise, when there is a local layer and a further feature layer, each further feature noise (or further feature guide) is calculated from the noise on a layer that overlaps with the further feature noise, i.e., the further feature of a local feature may occupy an additional layer superimposed on the local feature's layer, and the additional layer may use the noise in the local feature's layer in determining the noise on the additional layer.

[0189] An example of a general noise determination for layered input is as follows: At layer t, a global noise n0 can be predicted or obtained from prompt p0 "a dog is chasing a man". At layer t+1, a local noise n1 can be predicted from prompt p1 "a woman is having a shirt with a brooch". At layer t+2, two local noises n2, n3 can be obtained from prompts p2 "a man is wearing the blue glasses" and p3 "a brooch is red and like butterfly". Alternatively, "a woman is having a shirt with a brooch" can also be included in the global noise.

[0190] The local guides, or final local noises, may be determined for p1, p2, and p3 as follows: The local guide g1 for p1 may be determined using the formula (n1-n0)*c1. The local guide g2 for p2 may be determined using (n2-n1)*c2. The local guide g3 for p3 may be determined using (n3-n1)*c3. The coefficients c1, c2, and c3 may be hyperparameters as mentioned above. The noise for each of the local and further features may be extracted using layout information defined, for example, by bounding boxes, and the final noise may be determined by adding each noise as follows: Final noise=nc+gg0+gg1+a1xg1+a2xg2+a3xg3 The final or combined noise may be combined, for example, in a combining step s1040 as described in FIG. 10.

[0191] The inventors have found that by determining the noise on a layer using the noise of that layer and the noise of an additional (small, overlapping) layer, the noise differences between the layers are accentuated, thus improving accuracy in generating the desired image. Furthermore, the inventors have found that using classifier-free guidance for the global layer and the global background layer, i.e., using unconditional noise in the determination, leads to accurate image generation.

[0192] FIG. 23 illustrates the effect of changing hyperparameters on generated images. An input 2300, e.g., a text input along with layout information, may be received or read. In this example, the input has a global feature of "a man talking with a woman" and a local feature of "a man having a blue shirt". A local noise or local guide gl0 for man may be determined by subtracting the global noise ng from the predicted local noise nl0 for man and multiplying by the hyperparameter cl0. That is, the local noise may be determined using the formula gl0=(nl0-ng)*cl0.

[0193] FIG. 23 shows four generated images 2310a, 2310b, 2310c, 2310d, each generated with a different hyper-parameter cl0. In the first image 2310a, the hyper-parameter is set to 0, which has the effect of removing local noise from the joint noise determination. That is, when the local noise is zero, only global noise is used as the final image noise. The inventors have found that when the hyper-parameter is set to 0, it can result in the generation of shirts of the wrong color. For example, a gray shirt can be generated instead of the desired blue shirt. The inventors increased the hyper-parameter as shown and found that each generated image tended toward a blue shirt. In this example, the hyper-parameter cl0=4 generated a deep blue shirt. The inventors have found that best practice hyper-parameters between 3 and 8 produce the desired results. Of course, other hyper-parameter values ​​may be used.

[0194] The hyperparameters may be user-defined. The user may select each hyperparameter when interacting with a GUI, such as the GUI shown in FIG. 25. Alternatively, the hyperparameters may be pre-set, or some hyperparameters may be pre-set and some may be user-editable. In this example, the hyperparameter cl0 of man is shown, but one skilled in the art will understand that the hyperparameters for each noise determination may vary. That is, considering FIG. 22, the hyperparameters c11, c21, c31, c41, c32 may vary and each hyperparameter may have the same or different values.

[0195] 24 is a block diagram of an information processing appliance 2400 or computing device 2400, such as a data storage server, that may be used to embody the present invention, to perform some or all of the operations of the methods embodying the present invention, and to perform some or all of the tasks of the apparatus of the embodiments. The computing device 2400 may be used to implement any of the method steps described above, such as any of steps S1010 to S1060.

[0196] Computing device 2400 includes a processor 2493 and memory 2494. Optionally, the computing device also includes a network interface 2497 for communicating with other such computing devices, such as other computing devices of embodiments of the present invention. Optionally, the computing device also includes one or more input mechanisms, such as a keyboard and mouse 2496, and a display unit, such as one or more monitors 2495. These elements may facilitate user interaction. The components may be connected to each other via a bus 2492.

[0197] The memory 2494 may include a computer-readable medium, which term may refer to a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) configured to carry computer-executable instructions. Computer-executable instructions may include, for example, instructions and data that are accessible by a computer (e.g., one or more processors) and cause it to perform one or more functions or operations. For example, computer-executable instructions may include instructions for implementing a method disclosed herein, or any method disclosed herein, such as steps S1010-S1060. The term "computer-readable storage medium" may include any medium capable of storing, encoding, or retaining a set of instructions for execution by a machine and causing a machine to perform any one or more of the method steps of the present disclosure, and thus the term "computer-readable storage medium" includes, but is not limited to, solid-state memory, optical media, and magnetic media. For example, but not limited to, the term "computer-readable storage medium" may include solid-state memory, optical media, and magnetic media. By way of example, and not limitation, such computer-readable media may include non-transitory computer-readable media including random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM) or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory devices (e.g., solid-state memory devices).

[0198] The processor 2493 is configured to control the computing device and perform processing operations such as executing computer program code stored in the memory 2494, for example to perform any of the method steps described herein. The memory 2494 stores data read and written by the processor 2493 and may store at least one pre-trained diffusion model and / or at least one scene graph and / or at least one encoder and / or at least a graph decomposition unit and / or at least a noise synthesis unit and / or at least one denoiser and / or at least one image generation unit and / or other data and / or programs for performing any of the method steps described above. These entities may be in the form of code blocks that are called as needed and executed in the processor.

[0199] As referred to herein, a processor may include one or more general-purpose processing devices, such as a microprocessor, a central processing unit, etc. A processor may include a processor implementing a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or other instruction set or processor implementing a combination of instruction sets. A processor may also include one or more special purpose processing devices, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. In one or more embodiments, a processor is configured to execute the operations discussed herein and instructions for performing the operations. The processor 2493 may be considered to include any unit described above. Any operation described as being performed by a unit may be implemented as a method by a computer, such as the processor 2493.

[0200] The display unit 2495 may display representations of data stored and / or generated by the computing device, such as generated images and / or GUI windows (such as the canvases discussed above), and / or interactive representations that allow a user to interact with the device 2400, for example by drawing, drag-and-drop or selection interactions, and / or any other output discussed above, and may display cursors and dialog boxes and screens that allow interaction between a user and programs and data stored on the computing device. The input mechanism 2496 may enable a user to input data and instructions into the computing device, such as allowing a user to input any of the user inputs discussed above. For example, the display unit may display a GUI including a canvas for a user to interact with, as described herein. A user may draw and place bounding boxes and enter text descriptions onto the canvas. Alternatively, a user may enter boxes in a form displayed in a GUI window to create a scene graph including layout information. That is, a user may interact with the GUI and the display to generate and display an input scene graph.

[0201] The network interface (network I / F) 2497 can be connected to a network such as the Internet and can be connected to other such computing devices via the network. The network I / F 2497 can control input and output data to and from other devices via the network. Other peripheral devices such as microphones, speakers, printers, power supply units, fans, cases, scanners, tracker balls, etc. may also be included in the computing device.

[0202] Methods embodying the present invention may be performed on a computing device / apparatus 2400 as shown in FIG. 24. Such a computing device need not have all the components shown in FIG. 24, but may consist of a subset of these components. For example, the device 2400 includes a processor 2493 and a memory 2494 coupled to the processor 2493. Alternatively, the device 2400 includes a processor 2493, a memory 2494 coupled to the processor 2493, and a display 2495. Methods embodying the present invention may be performed by a single computing device in communication with one or more data storage servers over a network. The computing device may be a data storage device that stores at least a portion of the data.

[0203] A method embodying the invention may be performed by a number of computing devices operating in cooperation with one another. One or more of the computing devices may be a data storage server that stores at least a portion of the data. For example, the diffusion model may be stored in a server separate from the other units.

[0204] The invention can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The invention can be implemented as a computer program or computer program product, i.e. a computer program embodied in a non-transitory information carrier, for example a machine-readable storage device or a propagated signal, for execution by or controlling the operation of one or more hardware modules.

[0205] A computer program may be in the form of a stand-alone program, computer program portions, or multiple computer programs, may be written in any type of programming language, including compiled or interpreted languages, and may be deployed in any form, such as as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a data processing environment. A computer program may be deployed to be executed on one module or on multiple modules that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0206] The steps of the methods of the present invention may be performed by one or more programmable processors executing computer programs to perform the functions of the present invention by operating on input data and generating output. The apparatus of the present invention may be implemented as programmed hardware or as special purpose logic circuitry including Field Programmable Gate Arrays (FPGAs) or Application-Specific Integrated Circuits (ASICs), etc.

[0207] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, a processor receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions coupled to one or more memories that store instructions and data.

[0208] The above-described embodiments of the invention may be advantageously used independently of any other embodiment or in any feasible combination with one or more other embodiments of the embodiments.

[0209] FIG. 25 is an example of a graphical user interface (GUI) 2500 for inputting text and layout information. In this example, a user may be presented with a blank canvas 2505. The canvas may have dimensions of, for example, 512x512 or 620x620. The user may draw bounding boxes 2510a, 2510b and provide input text 2515a, 2515b. In this example, the bounding boxes provide layout information as input. The layout information may include position and size information and may be in the format described in connection with FIG. 14. That is, the position or layout of the bounding box may be defined using the top left and bottom right positions of the box, such as ((x1,y1)(x2,y2)). The user may input a global background, e.g., background information, scene or style, in a background text box 2517. In one example, the background text may be extracted and appended to the end of the global text extracted from the input text and layout information (i.e., extracted from the scene graph). In the example scene graph where the global text is "a man is walking with a dog" and the background input is "in the park", the background text may be added to the end of the global text, i.e., "a man is walking with a dog + in the park".

[0210] Alternatively, as discussed in connection with Figures 20-23, the global features and the background may be treated as separate features, i.e., the background may form a global background or background feature. Thus, the global text and the background text may be coded separately, and the predicted noise of each text may be summed, e.g., summing the noise from the global text "a man is walking" with the noise from the background text (another global text) "in the park".

[0211] In this example, each box comprises input text 2515a, 2515b. The input text for the first bounding box 2510a in the form of a label defines the bounding box as "dog". The input text for the second bounding box 2510b in the form of a label defines the bounding box as "dog".

[0212] In practice, the GUI may also present the user with user selectable buttons 2525, shown as dashed boxes. For example, a first button, "Add object," when clicked, allows the user to input layout information, e.g., the "Add object" button allows the user to draw a bounding box using drag-and-drop techniques. Once the user has drawn the bounding box, the user may be presented with a prompt to input text to label the bounding box. In this manner, the user may draw a first bounding box 2510a and label it "dog."

[0213] The GUI may also provide interactivity to the user in the form of an "Add predicate" button. This button may be selected with a mouse or keyboard or touch screen and allows the user to add labeled predicates 2520a, 2521a, 2520b to the canvas. For example, the bounding box of the dog is assigned the predicates "brown" and "running". The predicates are shown as labeled links that define the attributes of the object. Similarly, the ball is assigned the predicate "pink" in the form of a labeled link. This example shows only local predicates, i.e. local features that define the attributes of a single object. However, one skilled in the art will appreciate that global predicates may also be added, including, for example, labeled links between objects that give the relationships between the objects. For example, the global predicate "running towards" may be added to condition the generated image 2550 to generate a dog running towards the ball. Global predicates may be added by the user first selecting an object and then another object (or subject), and once two objects are selected, an "add" button appears. The user may click on the add button to add a predicate linking the two objects.

[0214] Other buttons available to the user are "Remove", which allows the user to remove objects and / or predicates from the canvas; "Clear", which clears the canvas of all objects and predicates; "Image", which allows the user to load an image from the computer and display it on the canvas; "Save", which saves the scene graph drawn on the canvas; and "Load", which loads a saved scene graph from the computer.

[0215] The GUI may further have user selectable buttons 2530. For example, the user may be presented with a "Generate" (shown as "Regene"), "Edit", and "Clear" button. The generate button generates an image once the user has entered text and layout information. Once the user clicks "generate" and has obtained the generated image, the "generate" button may change to "regenerate" which allows the user to generate a new image with different settings (boxes, different seeds, etc.). The "Edit" button allows the user to modify or edit an existing image. For example, the user may first load an image with the "Image" button and then draw a box with an object label. The user may then click "Edit" to add a new object to the original image. For example, if the user adds a box containing "squirrel", an image containing a squirrel will be added to the generated image at a specific location. The "Clear" button may clear the output.

[0216] The user may also be given the option to set a seed for the initial image representation. As described herein, the initial image representation may be a random seed. Alternatively, the user may select a known seed. Given the same input and a known seed, the image generation method will generate the same output image each time the generate button is selected. That is, if the scene graph drawn on the canvas is the same, the generated image will be the same and have the same seed. Thus, if a saved scene graph is loaded and the same seed is set, the same image will be generated.

[0217] The GUI may present the user with a "Descriptions" panel 2560 that presents scene-wide information including the layout of the boxes and the conditions of the settings including drawBox. That is, the Descriptions panel may include the text input given by the user and the layout information of each bounding box. The text in the Descriptions panel may be in json format. The text may be sent to the server to generate the image. As shown in the description text, drawBox may be set as "True" or "False". When drawBox is true, the input layout, i.e., the input bounding box, may be drawn on the generated image (i.e., overlapped or superimposed on the generated image as shown in FIG. 25). When drawBox is false, the generated image may be displayed without the layout bounding box overlapping or superimposed. That is, only the original generated image may be displayed.

[0218] The GUI may also present the user with a "Logs" panel 2570 that displays user debugging information.

[0219] The GUI may give the user the option to specify a layer for each bounding box and text label. As shown, a global background may be entered in a separate background text box. The GUI may include buttons that allow the user to select a layer and to draw and label the bounding boxes. Additionally or alternatively, the layers may be determined automatically.

[0220] The GUI may allow the user to specify hyperparameters for each text label and bounding box. For example, as the user enters text, a box may be displayed that allows the user to change the hyperparameters for that text box. Of course, any other display method for toggling hyperparameters may be used, for example, sliders may be used.

[0221] For the avoidance of doubt, the present invention relates to the following numbered items:

[0222] (1) A computer-implemented method for generating an image, the method comprising: receiving input text and layout information; decomposing the input text into one or more global features and one or more local features; encoding the one or more global features into a global vector and the one or more local features into respective local vectors; initializing an initial image representation; predicting a global noise from the global vector and the initial image representation using a trained neural network; predicting, for each of the one or more local features, an initial local noise from the one or more local vectors and the initial image representation using a trained neural network; determining a final local noise for each of the one or more local features using the initial local noise and the (predicted) global noise; using said layout information to combine said predicted global noise and said final local noise into a combined noise (possibly using a refinement vector of said predicted global noise for this combination, e.g. using unconditional noise, where the final global noise is determined from the predicted global noise in the previous step); denoising the initial image representation using the synthetic noise to provide a next image representation for generating the image; 23. A computer-implemented method comprising:

[0223] (2) The computer-implemented method of item 1, wherein the layout information includes position and size information of one or more objects described in the input text.

[0224] (3) The computer-implemented method according to any one of items 1 to 2, wherein the input text and the layout information are received from a user input on a graphical user interface (GUI).

[0225] (4) The computer-implemented method of item 3, wherein the layout information is input as a bounding box on a canvas for each object, and the input text is input as one or more labels added to each bounding box to name the object.

[0226] (5) The computer-implemented method of any one of items 1 to 4, wherein the input text is additionally input as labels together with links to form a predicate of one or more objects, the predicate including any labeled links between objects or labeled links to attributes of objects that provide relationships between the objects.

[0227] (6) The computer-implemented method of item 5, wherein the predicate is input by a user and / or the predicate is automatically generated, and the automatically generated predicate links two adjacent objects together.

[0228] (7) The computer-implemented method of item 6, further comprising using a neural network trained on the scene graph to predict predicates that are likely to link two objects.

[0229] (8) The computer-implemented method of any one of items 1 to 7, wherein the global features include an object (or item or article) or a global predicate connecting two objects.

[0230] (9) The computer-implemented method of item 8, wherein the global predicate defines a relationship between the two objects and / or the global predicate defines an action or a connection or an association between two objects.

[0231] (10) The computer-implemented method according to any one of items 1 to 9, wherein the local features include the object together with local predicates that define attributes of the object.

[0232] (11) The computer-implemented method according to any one of items 1 to 10, wherein the input text and links form a scene graph, and the input text and links are received as a scene graph.

[0233] (12) The computer-implemented method of item 11, wherein the scene graph is generated by a user interacting with a graphical user interface.

[0234] (13) The computer-implemented method of any one of items 11 and 12, wherein the scene graph is an extracted scene graph for an image or video, and optionally a user loads an image with a scene graph overlaid on the image, and the user modifies, adds or removes objects and / or predicates from the scene graph.

[0235] (14) The computer-implemented method according to any one of items 11 to 13, wherein the scene graph is decomposed using a graph decomposition unit by extracting all object nodes and their connections as a global graph including global features, and extracting one or more object nodes and one or more attributes connected to the extracted one or more object nodes, each as a local graph including local features.

[0236] (15) The computer-implemented method of item 14, wherein a local graph is not generated for an object node unless and / or if the object node has only global predicates associated with it.

[0237] (16) The computer-implemented method according to any one of items 1 to 15, wherein the final local noise of each local feature is determined by subtracting the predicted noise of the global vector from the predicted noise of the local vector, preferably then multiplying by a local noise hyperparameter, and extracting the final local noise using layout information of the object associated with the local feature.

[0238] (17) The computer-implemented method according to any one of items 1 to 16, wherein the final global noise is determined by subtracting an unconditional noise from the predicted global noise, preferably multiplying the difference by a global noise hyperparameter, and adding the unconditional noise.

[0239] (18) The computer-implemented method of claim 17, wherein the unconditional noise is predicted by inputting a null vector and the initial image representation into the trained neural network.

[0240] (19) The computer-implemented method according to item 17 or 18, wherein the determination of the final global noise uses a classifier-free guidance technique.

[0241] (20) The computer-implemented method of any of items 1 to 19, wherein the combined noise is synthesized by adding the final global noise and the final local noise, and optionally the global noise is a tensor, and the tensors are preferably of the same size or dimensions and are added using tensor summation.

[0242] (21) The computer-implemented method according to any one of items 1 to 20, wherein the global features and the local features each occupy a separate layer, and preferably the input text and layout information are received on multiple layers.

[0243] (22) The computer-implemented method of item 21, wherein the layers are user-defined, and preferably the layers are selected using a graphical user interface.

[0244] (23) The computer-implemented method of any of items 21 to 22, wherein the layers are “conceptual” layers that exist solely for the purpose of describing how noise is computed for the input text, and some layers are “conceptual” layers and some layers are user-defined.

[0245] (24) A computer-implemented method according to any of items 21 to 23, wherein a further feature of a local feature occupies an additional layer superimposed on the layer of the local feature, and optionally, the further feature occupies a successive conceptual layer superimposed on the conceptual layer of the local feature.

[0246] (25) The computer-implemented method according to any one of items 1 to 24, wherein the further feature of the local feature is a feature related to or overlapping with the local feature.

[0247] (26) The computer-implemented method of item 25, wherein an additional feature is positioned so that it is completely surrounded within the local feature, and if the layout information is input as a bounding box, the additional feature bounding box is within the local feature bounding box.

[0248] (27) The computer-implemented method of any of items 24 to 26, wherein an additional layer populated by the further features uses noise in that local feature layer in determining noise on the additional layer, and optionally the further features are encoded into an additional feature vector, and initial further feature noise is predicted from the further features and the initial image representation using a trained neural network.

[0249] (28) The computer-implemented method of claim 27, wherein a final further feature noise is determined by subtracting the initial local feature noise from the initial further feature noise, optionally multiplying by a hyper-parameter, preferably the further feature noise is extracted using layout information of the further features.

[0250] (29) The computer-implemented method of any one of items 1 to 28, wherein the input text is further decomposed into a global background, the global background is encoded into a global background vector, and the global background noise is predicted from the background global vector and the initial image representation using a trained neural network, and synthesizing a synthetic noise further comprises synthesizing the predicted global background noise.

[0251] 30. The computer-implemented method of claim 29, wherein the global background is a text input describing a background or a scene or a style. Optionally, the global background is on a global background layer.

[0252] (31) The initial image representation is denoised by subtracting the combined noise from the initial image representation, and optionally the denoising comprises multiplying the combined noise by a coefficient to obtain an adjusted noise, and subtracting the adjusted noise from the initial image representation.

[0253] (32) The computer-implemented method of item 31, wherein the combined noise is in the form of a tensor and is the same size as the initial image representation.

[0254] (33) The computer-implemented method according to any one of items 1 to 32, wherein the initial image representation is a k-dimensional vector.

[0255] (34) The computer-implemented method of any one of items 1 to 33, wherein the initial image representation represents random reproduction noise, and optionally, the random reproduction noise is generated from a random seed.

[0256] (35) The computer-implemented method of any one of items 1 to 34, wherein the one or more global features and / or the one or more local features are encoded using a graph encoder.

[0257] (36) The computer-implemented method according to any one of items 1 to 35, wherein the encoding of the one or more global features and / or the encoding of the one or more local features uses a text encoder.

[0258] (37) The computer-implemented method according to any one of items 1 to 36, further comprising iteratively repeating the predicting, synthesizing and denoising steps to replace the initial image representation with a next image representation provided for a first iteration and replacing the next image representation and further image representations with denoised representations generated in further iterations to generate a final image representation.

[0259] (38) The computer-implemented method of item 37, wherein the method further comprises generating a final image using the final image representation and an image decoder, and if the method is not repeated iteratively, the final image representation is the next image representation provided.

[0260] (39) The computer-implemented method according to any one of items 1 to 38, wherein the trained neural network used in the method is a pre-trained denoising diffusion probabilistic model (DDPM).

[0261] (40) The computer-implemented method of claim 39, wherein the DDPM is a latent diffusion model.

[0262] (41) The computer-implemented method of item 40, wherein the latent diffusion model uses a U-net architecture with a cross-attention mechanism, and optionally, the diffusion model is a stable diffusion model, and preferably, the diffusion model is trained on the LAION-5B training set. The computer-implemented method of item 40, wherein the latent diffusion model uses a U-net architecture with a cross-attention mechanism, and optionally, the diffusion model is a stable diffusion model, and preferably, the diffusion model is trained on the LAION-5B training set.

[0263] (42) A computer-implemented method for generating an image, comprising: receiving input text and layout information; decomposing the input text into one or more global features, one or more local features, one or more further features of the one or more local features and a global background, where the global features and the local features each occupy a separate layer, the global background occupying a global background layer, and the further features of the local features occupy an additional layer superimposed on the local feature layer; encoding the one or more global features into a global vector, encoding the one or more local features into respective local vectors, encoding the global background into a global background vector, and encoding one or more additional features into respective separate feature vectors; initializing an initial image representation; predicting a global noise from said global vector and said initial image representation using a trained neural network; predicting an initial local noise for each of the one or more local features from the one or more local vectors and the initial image representation using a trained neural network; predicting a global background noise for the global background vector and the initial image representation using a trained neural network; predicting initial further feature noise for each of the one or more further features from the one or more further feature vectors and the initial image representation using a trained neural network; determining a final local noise for each of the one or more local features using the initial local noise and the global noise; determining a final further feature noise for each of the one or more further features using the initial further feature noise and local feature noises of the local features of the further features; combining the predicted global noise, the predicted global background noise, the final local noise, and the final further feature noise into a combined noise using the layout information; denoising the initial image representation using the combined noise to provide a next image representation for generating the image; 23. A computer-implemented method comprising:

[0264] (43) A computer program which, when executed on a computer, causes the computer to carry out a method, the method comprising: receiving input text and layout information; decomposing the input text into one or more global features and one or more local features; encoding the one or more global features into a global vector and the one or more local features into respective local vectors; initializing an initial image representation; predicting a global noise from the global vector and the initial image representation using a trained neural network; predicting, for each of the one or more local features, an initial local noise from the one or more local vectors and the initial image representation using a trained neural network; determining a final local noise for each of the one or more local features using the initial local noise and the global noise; combining the predicted global noise and the final local noise into a combined noise using the layout information; denoising the initial image representation using the synthetic noise to provide a next image representation for generating the image; A computer program comprising:

[0265] (44) An information processing device for generating an image, comprising: a memory; and a processor coupled to the memory, the processor: Receive input text and layout information; Decomposing the input text into one or more global features and one or more local features; encoding the one or more global features into a global vector and the one or more local features into respective local vectors; Initialize the initial image representation, predicting a global noise from the global vector and the initial image representation using a trained neural network; predicting, for each of the one or more local features, an initial local noise from the one or more local vectors and the initial image representation using a trained neural network; determining a final local noise for each of the one or more local features using the initial local noise and the global noise; combining the predicted global noise and the final local noise into a combined noise; denoising the initial image representation using the synthetic noise to provide a next image representation for generating the image. Information processing equipment.

[0266] (45) A computer program that, when executed by a remote conference device, causes the remote conference device to execute any one of the methods of items 1 to 42.

[0267] (46) The computer program of item 45, wherein the computer program is stored on a computer-readable medium, and optionally the computer-readable medium is non-transitory.

[0268] (47) A computer program which, when executed by a companion device, causes the companion device to execute any one of the methods of items 1 to 42.

[0269] (48) The computer program of item 47, wherein the computer program is stored on a computer-readable medium, and optionally the computer-readable medium is non-transitory.

[0270] (49) A digital electronic circuit configured to carry out the method according to any one of items 1 to 42, or computer hardware, firmware, software, or a combination of them.

[0271] (50) The computer-implemented method according to any one of items 1 to 42, wherein the method is implemented as a computer program or computer program product, and optionally the computer program is embodied in a non-transitory information carrier for execution by or for controlling the operation of one or more hardware modules.

[0272] (51) The computer-implemented method according to item 50, wherein the non-transitory information carrier is a machine-readable storage device or a propagated signal.

[0273] (52) The computer-implemented method according to any one of items 1 to 42 and 50 to 51, wherein the method is implemented as a computer program, the computer program being in the form of a stand-alone program, a computer program portion, or a plurality of computer programs, optionally written in any form of programming language, including a compiled or interpreted language, and optionally deployed only in any form, including as a stand-alone program, or as a module, component, subroutine, or other unit suitable for use in a data processing environment.

[0274] The present invention has been described in terms of specific embodiments. Other embodiments are within the scope of the following claims. For example, the steps of the invention can be performed in a different order and still achieve desirable results.

[0275] Those skilled in the art will appreciate that, unless mutually exclusive, a feature described in relation to any one of the above embodiments may also be applied mutatis mutandis to any other embodiment. Furthermore, unless mutually exclusive, any feature described herein may be applied to any embodiment and / or combined with any other feature described herein.

[0276] In addition to the above-described embodiment, the following supplementary notes are further disclosed. (Appendix 1) 1. A computer-implemented method for generating an image, comprising: receiving input text and layout information; decomposing the input text into one or more global features and one or more local features; encoding the one or more global features into a global vector and the one or more local features into respective local vectors; initializing an initial image representation; predicting a global noise from the global vector and the initial image representation using a trained neural network; predicting, for each of the one or more local features, an initial local noise from the one or more local vectors and the initial image representation using a trained neural network; determining a final local noise for each of the one or more local features using the initial local noise and the predicted global noise; combining the predicted global noise and the final local noise into a combined noise using the layout information; denoising the initial image representation using the combined noise to provide a next image representation for generating the image; 23. A computer-implemented method comprising: (Appendix 2) 2. The computer-implemented method of claim 1, wherein the layout information includes position and size information of one or more objects described in the input text, and / or the input text and the layout information are received from user input on a graphical user interface (GUI), the layout information is entered as bounding boxes on a canvas for each object, and the input text is entered as one or more labels added to each bounding box to name the objects. (Appendix 3) 3. The computer-implemented method of claim 2, wherein the input text is additionally input as labels along with links to form a predicate of one or more objects, the predicate including a labeled link between two objects or a labeled link to an attribute of an object that provides a relationship between the objects. (Appendix 4) 4. The computer-implemented method of claim 1, wherein global features include an object or a global predicate connecting two objects, the global predicate defining a relationship between the two objects, and local features include an object and a local predicate defining an attribute of the object. (Appendix 5) 2. The computer-implemented method of claim 1, wherein the input text and links form a scene graph, and the scene graph is decomposed using a graph decomposition unit by extracting all object nodes and their connections as a global graph containing global features, and extracting one or more object nodes and one or more attributes connected to the extracted one or more object nodes, each as a local graph containing local features. (Appendix 6) the final local noise for each local feature is determined by subtracting the predicted noise of the global vector from the predicted noise of the local vector, then multiplying by a local noise hyperparameter and extracting the final local noise using layout information of the object associated with the local feature; and / or a final global noise is determined by subtracting an unconditional noise from the predicted global noise, multiplying the difference by a global noise hyperparameter, and adding the unconditional noise, the unconditional noise being predicted by inputting blank vectors and the initial image representation to the trained neural network; and / or 2. The computer-implemented method of claim 1, wherein the combined noise is synthesized by adding the final global noise and the final local noise. (Appendix 7) 2. The computer-implemented method of claim 1, wherein the global features and the local features each occupy a separate layer, and a further one of the local features occupies an additional layer that is superimposed on the layer of the local features, and the additional layer uses noise in the layer of the local features when determining noise on the additional layer. (Appendix 8) 2. The computer-implemented method of claim 1, wherein the input text is further decomposed into a global background, the global background is encoded into a global background vector, and a global background noise is predicted from the global background vector and the initial image representation using a trained neural network, and wherein synthesizing a combined noise further comprises synthesizing the predicted global background noise. (Appendix 9) A computer program which, when executed on a computer, causes the computer to carry out a method, the method comprising: receiving input text and layout information; decomposing the input text into one or more global features and one or more local features; encoding the one or more global features into a global vector and the one or more local features into respective local vectors; initializing an initial image representation; predicting a global noise from the global vector and the initial image representation using a trained neural network; predicting, for each of the one or more local features, an initial local noise from the one or more local vectors and the initial image representation using a trained neural network; determining a final local noise for each of the one or more local features using the initial local noise and the predicted global noise; combining the predicted global noise and the final local noise into a combined noise using the layout information; denoising the initial image representation using the combined noise to provide a next image representation for generating an image; A computer program comprising: (Appendix 10) 1. An information processing device for generating an image, comprising: a memory; and a processor coupled to the memory, the processor comprising: Receive input text and layout information; Decomposing the input text into one or more global features and one or more local features; encoding the one or more global features into a global vector and the one or more local features into respective local vectors; Initialize the initial image representation, predicting a global noise from the global vector and the initial image representation using a trained neural network; predicting, for each of the one or more local features, an initial local noise from the one or more local vectors and the initial image representation using a trained neural network; determining a final local noise for each of the one or more local features using the initial local noise and the predicted global noise; combining the predicted global noise and the final local noise into a combined noise; denoising the initial image representation using the combined noise to provide a next image representation for generating the image. Information processing equipment. [Explanation of symbols]

[0277] 1120 Graph Decomposition Unit 1125 Encoder Unit 1130 Global Noise Prediction Unit 1135 Local Noise Prediction Unit 1140 Noise Synthesis Unit 1145 Representation Update Unit 1150 Image Decoder Unit

Claims

1. 1. A computer-implemented method for generating an image, comprising: receiving input text and layout information; decomposing the input text into one or more global features and one or more local features; encoding the one or more global features into a global vector and the one or more local features into respective local vectors; initializing an initial image representation; predicting a global noise from the global vector and the initial image representation using a trained neural network; predicting, for each of the one or more local features, an initial local noise from the one or more local vectors and the initial image representation using a trained neural network; determining a final local noise for each of the one or more local features using the initial local noise and the predicted global noise; combining the predicted global noise and the final local noise into a combined noise using the layout information; denoising the initial image representation using the combined noise to provide a next image representation for generating the image; 23. A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, wherein the layout information includes position and size information of one or more objects described in the input text, and / or the input text and the layout information are received from user input on a graphical user interface (GUI), the layout information being entered as bounding boxes on a canvas for each object, and the input text being entered as one or more labels added to each bounding box to name the object.

3. 3. The computer-implemented method of claim 2, wherein the input text is additionally input as labels along with links to form a predicate of one or more objects, the predicate including a labeled link between two objects providing a relationship between the objects or a labeled link to an attribute of an object.

4. 4. The computer-implemented method of claim 1, wherein global features include an object or a global predicate connecting two objects, the global predicate defining a relationship between the two objects, and local features include an object and a local predicate defining an attribute of the object.

5. 2. The computer-implemented method of claim 1, wherein the input text and links form a scene graph, and the scene graph is decomposed using a graph decomposition unit by extracting all object nodes and their connections as a global graph containing global features, and extracting one or more object nodes and one or more attributes connected to the extracted one or more object nodes, each as a local graph containing local features.

6. the final local noise for each local feature is determined by subtracting the predicted noise of the global vector from the predicted noise of the local vector, then multiplying by a local noise hyperparameter and extracting the final local noise using layout information of the object associated with the local feature; and / or a final global noise is determined by subtracting an unconditional noise from the predicted global noise, multiplying the difference by a global noise hyperparameter, and adding the unconditional noise, the unconditional noise being predicted by inputting blank vectors and the initial image representation to the trained neural network; and / or The computer-implemented method of claim 1 , wherein the combined noise is synthesized by adding the final global noise and the final local noise.

7. 2. The computer-implemented method of claim 1, wherein the global features and the local features each occupy a separate layer, and further features of the local features occupy an additional layer that is superimposed on the layer of the local features, and the additional layer uses noise in the layer of the local features in determining noise on the additional layer.

8. 2. The computer-implemented method of claim 1, wherein the input text is further decomposed into a global background, the global background is encoded into a global background vector, and a global background noise is predicted from the global background vector and the initial image representation using a trained neural network, and wherein synthesizing a combined noise further comprises synthesizing the predicted global background noise.

9. A computer program which, when executed on a computer, causes the computer to carry out a method, the method comprising: receiving input text and layout information; decomposing the input text into one or more global features and one or more local features; encoding the one or more global features into a global vector and the one or more local features into respective local vectors; initializing an initial image representation; predicting a global noise from the global vector and the initial image representation using a trained neural network; predicting, for each of the one or more local features, an initial local noise from the one or more local vectors and the initial image representation using a trained neural network; determining a final local noise for each of the one or more local features using the initial local noise and the predicted global noise; combining the predicted global noise and the final local noise into a combined noise using the layout information; denoising the initial image representation using the combined noise to provide a next image representation for generating an image; A computer program comprising:

10. 1. An information processing device for generating an image, comprising: a memory; and a processor coupled to the memory, the processor comprising: Receive input text and layout information; Decomposing the input text into one or more global features and one or more local features; encoding the one or more global features into a global vector and the one or more local features into respective local vectors; Initialize the initial image representation, predicting a global noise from the global vector and the initial image representation using a trained neural network; predicting, for each of the one or more local features, an initial local noise from the one or more local vectors and the initial image representation using a trained neural network; determining a final local noise for each of the one or more local features using the initial local noise and the predicted global noise; combining the predicted global noise and the final local noise into a combined noise; denoising the initial image representation using the combined noise to provide a next image representation for generating the image. Information processing equipment.

Images