Visualizing stories using machine learning models

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a visualization of a story. In one aspect, a method comprises receiving an input from a user of a user device that comprises text representing a story to be visualized through a plurality of images; in response to receiving the input: obtaining a plurality of sets of words from the text; for each set of words in the plurality of sets of words: generating an input prompt for the set of words; obtaining an image that represents the input prompt by providing the input prompt to a first machine learning model; and providing, for presentation on the user device, the plurality of images generated by the first machine learning model that visualize the story.

Description

[0001] VISUALIZING STORIES USING MACHINE LEARNING MODELS

[0002] BACKGROUND

[0003] This specification relates to generating outputs conditioned on inputs using machine learning models.

[0004] Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

[0005] Some machine learning models are deep models that employ multiple layers of operations to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output.

[0006] SUMMARY

[0007] This specification describes a system implemented as computer programs on one or more computers in one or more locations that generates and displays an output conditioned on inputs. The output can include a video or multiple images that visualize a story. The system can use one or more machine learning models to generate the output conditioned on input text that represents the story to be visualized.

[0008] According to a first aspect there is provided a method comprising: receiving an input from a user of a user device that comprises text representing a story to be visualized through a plurality of images; in response to receiving the input: obtaining a plurality of sets of words from the text; for each set of words in the plurality of sets of words: generating an input prompt for the set of words, wherein the input prompt for the set of words comprises one or more words from the set of words; obtaining an image that represents the input prompt by providing the input prompt to a first machine learning model, wherein the first machine learning model is configured to generate an image responsive to the input prompt; and providing, for presentation on the user device, the plurality of images generated by the first machine learning model that visualize the story.

[0009] In some implementations, the method further comprises generating the text from a voice input from the user. In some implementations, the method further comprises receiving an input from the user that identifies a certain number of images to generate, and wherein the plurality of sets of words includes the certain number of sets of words.

[0010] In some implementations, providing, for presentation on the user device, the plurality of images comprises: providing the plurality of images and a respective transition between each of the images in the plurality of images for presentation on the user device.

[0011] In some implementations, providing, for presentation on the user device, the plurality of images comprises: providing the plurality of images for presentation in a user interface of the user device, wherein the user interface comprises a plurality of user interface elements each controllable by the user to modify a respective property of a particular image being presented in the user interface by selecting among a set of options for the property, and wherein each option is associated with a different modification of the input prompt corresponding to the particular image; in response to receiving an input from the user to a user interface element of the plurality of user interface elements: generating a modified input prompt by applying the modification associated with the option to the input prompt; obtaining a modified particular image that represents the modified input prompt by providing the modified input prompt to the first machine learning model; and providing, for presentation on the user device, the modified particular image.

[0012] In some implementations, the property of the particular image comprises any one of: a visual theme, or a graphic style.

[0013] In some implementations, the method further comprises: receiving an input from the user that identifies one or more options for modifying one or more corresponding properties of a second image being presented in the user interface; in response to receiving the input: generating a modified input prompt by applying the modification associated with each identified option to the input prompt for the second image; obtaining a modified second image by providing the modified input prompt to the first machine learning model; and providing, for presentation on the user device, the modified second image.

[0014] In some implementations, providing, for presentation on the user device, the plurality of images comprises: providing the plurality of images for presentation in a user interface of the user device, wherein the user interface comprises a user interface element controllable by the user to select regions in a particular image being presented in the user interface to replace corresponding regions in a second image being presented in the user interface; in response to receiving an input from the user to the user interface element: generating an image mask comprising pixels from the selected regions in the particular image, and noisy pixels in regions in the particular image that have not been selected by the user; generating a description of the not selected regions; generating a modified input prompt by applying the description of the not selected regions to the input prompt of the second image; obtaining a modified second image comprising the selected regions by providing the modified input prompt and the image mask to a second machine learning model, wherein the second machine learning model is configured to generate an image responsive to the modified input prompt with regions from an image mask; and providing, for presentation on the user device, the modified second image.

[0015] In some implementations, wherein providing, for presentation on the user device, the plurality of images comprises: providing the plurality of images for presentation in a user interface of the user device, wherein the user interface comprises a user interface element controllable by the user to select regions in a first image and a second image to generate a combined image; in response to receiving an input from the user to the user interface element: generating a first image mask comprising pixels in the selected regions in the first image; generating a second image mask comprising pixels in the selected regions in the second image; processing the first and second image mask to generate the combined image; and providing, for presentation on the user device, the combined image.

[0016] In some implementations, the method further comprises: receiving an input from the user that identifies an additional set of words from the text; in response to receiving the input: generating an additional input prompt comprising the additional set of words; obtaining an additional image that represents the additional input prompt by providing the additional input prompt to the first machine learning models; and providing, for presentation on the user device, the additional image with the plurality of images, in an order corresponding to a chronological order of the plurality of sets of words and the additional set of words in the text.

[0017] In some implementations, providing, for presentation on the user device, the plurality of images comprises: providing the plurality of images for presentation in a user interface of the user device in an order corresponding to a chronological order of the plurality of sets of words in the text, wherein the user interface comprises a user interface element controllable by the user to adjust the order of the images in the plurality of images; receiving an input from the user that identifies the order of the images to adjust; in response to receiving the input: providing, for presentation on the user device, the plurality of images in the adjusted order. In some implementations, the method further comprises: providing, for presentation on the user device, the plurality of sets of words in a user interface of the user device, wherein the user interface comprises a user interface element controllable by the user to modify one or more words in a particular set of words; in response to receiving an input from the user to the user interface element: generating a modified input prompt comprising the modified words of the particular set of words; obtaining a modified image representing the modified input prompt by providing the modified input prompt to the first machine learning model; and providing, for presentation on the user device, the modified image.

[0018] In some implementations, the method further comprises: receiving a first input from the user that comprises a reference image; in response to receiving the first input: obtaining text that describes the reference image by providing the reference image to a third machine learning model, wherein the third machine learning model is configured to generate text describing an image; providing, for presentation on the user device, the text; receiving a second input from the user that identifies a subset of the text; identifying at least one set of words of the plurality of sets of words as relevant to the subset of the text; for each relevant set of words: generating an input prompt comprising the subset of the text and one or more words from the relevant set of words; obtaining an image that represents the input prompt by providing the input prompt to the first machine learning model; and providing, for presentation on the user device, each image.

[0019] In some implementations, the method further comprises: receiving an input from the user that comprises different text that describes the reference image than was obtained by providing the reference image to the third machine learning model; in response to receiving the input: generating a modified input prompt comprising one or more words of the text of the input; obtaining a modified image representing the modified input prompt by providing the modified input prompt to the first machine learning model; and providing, for presentation on the user device, the modified image.

[0020] In some implementations, the method further comprises: receiving a first input from the user that comprises a reference image; receiving a second input from the user that identifies an aspect of the reference image; in response to receiving the second input: obtaining text that describes the reference image by providing the reference image to a third machine learning model, wherein the third machine learning model is configured to generate text describing an image; identifying a subset of the text that is relevant to the aspect of the reference image; identifying at least one set of words of the plurality of sets of words as relevant to the subset of the text; for each relevant set of words: generating an input prompt comprising the subset of the text and one or more words from the relevant set of words; obtaining an image that represents the input prompt by providing the input prompt to the first machine learning model; and providing, for presentation on the user device, each image.

[0021] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0022] Conventional techniques for visualizing stories have limitations which make them difficult or impractical for some users to use. For example, some techniques require the user to manually decide which scenes of a story to visualize. In addition, some techniques require the user to manually describe what subjects should be depicted in an individual frame or image that visualizes a scene of the story, and how those subjects should be depicted. These limitations may result in a less efficient user experience for visualizing a story.

[0023] The system described in this specification generates images that visualize a story without requiring a user to decide which scenes of the story to visualize, and without having to describe what parts of the story should be visualized, or how those parts of the story should be visualized. For example, the system can receive an input from a user that includes text representing a story to be visualized. The system can obtain multiple sets of words from the text, and for each set of words, generate an input prompt including one or more words from the set of words and obtain an image representing the input prompt using a machine learning model. The system can then provide the images for presentation on a user device of the user without any further inputs from the user. The system can thus determine a number of scenes to visualize from text representing the story, and what to include in each scene based on only the text and without explicit guidance from the user.

[0024] Some conventional techniques do not allow users to modify what is displayed in a visualization after it is generated, or to modify properties of the visualization. The system described in the specification provides for a flexible and interactive user experience. For example, in some implementations, the system can include a user interface with user interface elements that the user can control to modify a property of a generated image. For example, the user can select from a set of options for the property. Each option can be associated with a different modification of the input prompt for the generated image. The system can thus provide a modified input prompt to a machine learning model to generate an image that is modified according to the user’s intent. In some implementations, the system can also retain the options used to generate a modified image, and apply the options to generate another image. In addition, the system allows a user to select regions in different images and combine regions into a new image. Furthermore, while the user is not required to decide which scenes of a story to visualize, or how to visualize them, the system provides transparency and flexibility for the user to view or edit what is visualized from the story. In some implementations, the system allows a user to define the number of images they would like in the visualization of the story. For example, the system can receive an input from the user that identifies a number of images to visualize, and the system can generate a story that includes that number of images. In some implementations, the system also allows a user to customize the scenes of the story to visualize. For example, the system allows a user to select additional scenes from the text to include in the visualization of the story. In some implementations, the system also allows a user to remove images from the visualization of the story. In some implementations, the system also allows a user to view and edit the set of words corresponding to a scene of the story. For example, the system allows a user to select different words for a set of words. Furthermore, in some implementations, the system allows a user to re-arrange the images in the visualization of the story. For example, the system allows a user to adjust the order in which the images are provided for display.

[0025] Some conventional techniques do not allow a user to provide additional inputs to influence the properties or contents of the visualization of the story. The system described in this specification allow s users to provide data such as image or video data that inspired them during their creative process, and uses aspects of the provided data during generation of the visualization of the story. For example, in some implementations, the system can generate a visualization of the story with similar lighting or graphic style as the provided data. Furthermore, the system allows users to view and edit what aspects of the provided data they would like to include in the visualization of the story. For example, the user can specify that the lighting in a particular image inspired them, or that the graphic style of the image inspired them.

[0026] Some conventional techniques do not allow users to re-apply a modification or multiple modifications to properties of an image, which can lead to inefficient use of computing resources if a user later decides to make the same modifications on another image. In some implementations, the system described in this specification can save computing resources by allowing a user to easily make the same modifications to multiple images. For example, after a user has made multiple modifications to properties of an image, the system can save data representing the modifications to the user device or to a remote server for later application by the user or another user to a second image. The system saves computing time and resources that would otherwise have been spent waiting for the user to input the same modifications for the second image or processing the modifications for the second image.

[0027] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0028] BRIEF DESCRIPTION OF THE DRAWINGS

[0029] FIG. 1 is a diagram of an example system for generating a visualization of a stor .

[0030] FIG. 2 is a flow diagram of an example process for generating a visualization of a story.

[0031] FIG. 3 shows an example presentation on a user device of an input from a user.

[0032] FIG. 4 shows an example presentation on a user device of multiple sets of words in a story.

[0033] FIG. 5 is a flow diagram of an example process for generating a modified image from a modified set of words.

[0034] FIG. 6 shows an example presentation on a user device of a modified set of words.

[0035] FIG. 7 shows an example presentation on a user device of an image generated by a machine learning model.

[0036] FIG. 8 is a flow diagram of an example process for modifying a particular image.

[0037] FIGS. 9A-9E shows an example presentation on a user device of modifying a visual theme of an image.

[0038] FIGS. 10A-10E shows an example presentation on a user device of modifying a graphic style of an image.

[0039] FIG. 11 is a flow diagram of an example process for generating images based on reference images.

[0040] FIG. 12 shows an example presentation on a user device of reference images.

[0041] FIG. 13 shows an example presentation on a user device for generating images based on reference images.

[0042] FIG. 14 shows an example presentation on a user device of an image generated based on a reference image.

[0043] FIG. 15 is a flow diagram of an example process for generating an image with elements from a previously generated image. FIG. 16 is a flow diagram of an example process for combining elements of generated images.

[0044] FIG. 17A shows an example presentation on a user device of images that are part of a visualization of a story.

[0045] FIG. 17B shows an example presentation on a user device of an image that is part of a visualization of a story.

[0046] Like reference numbers and designations in the various drawings indicate like elements.

[0047] DETAILED DESCRIPTION

[0048] FIG. 1 is a diagram of an example system 100 for generating a visualization of a story. The system 100 includes multiple components, such as an extraction engine 110, a prompt engine 120, or a machine learning model 130. The system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0049] The system 100 can provide a user 102 with a visualization of the user’s story. A story can refer to a description of a series of scenes. The description can include details such as subjects or characters, settings, visual descriptors, and plots. A visualization of the story can include visual elements such as videos or images that depict scenes described in the story.

[0050] To provide the user 102 with a visualization of a story, the system 100 can receive an input that includes text representing the story from the user 102 of a user device 104. The system 100 can generate images 132 that visualize the story using one or more machine learning models, as described in further detail below. The system 100 can then present the images 132 to the user 102 for display on the user device 104 through the user interface 106.

[0051] In some implementations, the components of the system 100 can run on the user device 104. The user device 104 can be any type of computer or computing device that has a display and is configured to receive an input from the user 102. For example, the user device 104 can be a computer, laptop, tablet, or mobile phone that has a display and can receive user inputs through interfaces such as a keyboard, mouse, touchpad, or touchscreen.

[0052] In some implementations, the components of the system 100 can run on one or more computers remote from the user device 104. The components of the system 100 can communicate with the user device 104 over a data communication network, e.g., the Internet. The user device 104 is configured to receive inputs such as text, visual data, and / or audio data. For example, the user device 104 can be configured to receive audio inputs and generate audio recordings of the audio inputs.

[0053] The user device 104 can be configured to display images 132 on a display of the user device 104. For example, the system 100 can provide the images 132 to a user interface 106 of the user device 104. The system 100 can also provide a transition between each of the images 132 to the user interface 106 for presentation on the user device 104.

[0054] Example presentations that can be displayed on the user device 104 are shown below in FIGS. 3-4, 6-7, 9-12, and 14-16.

[0055] The user interface 106 can include user interface elements such as buttons that allow the user 102 to interact with the content displayed in the user interface 106. For example, the user interface elements can allow the user 102 to select among a set of options to modify a property of an image displayed in the user interface 106, or to select regions in an image displayed in the user interface 106. The system 100 can process the user inputs from the user 102 to update the content displayed in the user interface 106.

[0056] In some examples, the user 102 can type the text of their story into the user interface 106. In some examples, the user 102 can speak the text of their story into a microphone of the user device 104. The system 100 can generate the text of the story from the audio data. For example, the system 100 can use a speech-to-text program such as a machine learning model to convert the audio data into text. The system 100 can provide the text of the story to the extraction engine 110 as text 108.

[0057] The system 100 can use the extraction engine 110, prompt engine 120, and machine learning model 130 to generate images 132 that visualize the story described in the text 108.

[0058] The extraction engine 110 can extract sets of words from text 108. Each set of words can correspond to a scene in the story. For example, the extraction engine 110 can be a computer program that is configured to process an input string with multiple words and output multiple sets of words 112 from the multiple words of the input string. The extraction engine 110 can perform natural language analyses such as entity analysis, entity sentiment analysis, and syntactic analysis, for example. For example, the extraction engine 110 can include a machine learning model that is trained to extract salient or important terms from the text 108. The extraction engine 110 can include a named entity recognition (NER) model to extract salient tenns, for example.

[0059] The extraction engine 110 can include salient or important terms in the sets of words 112. As an example, the extraction engine 110 can determine that a subject, a “girl,” is an important term. The extraction engine 110 can also determine that verbs or actions are important terms. The extraction engine 110 can also determine that descriptors, such as “who wore a red cloak” are important or salient terms. Extracting sets of words is described in more detail below with reference to FIG. 4.

[0060] The system 100 can provide the sets of words 112 to the prompt engine 120. The prompt engine 120 can generate, for each set of words, an input prompt including one or more words from the set of words in the sets of words 112. The prompt engine 120 can be a computer program that is configured to process an input string with multiple words and output an input prompt for a machine learning model that includes the words of the input string. For example, for each set of words, the prompt engine 120 can generate a corresponding input prompt 122 that includes one or more words from the set of words. For example, the input prompt 122 can include all of the words from the set of words. The input prompt 122 can also include synonyms, related words, or other tenses for one or more words from the set of words. Each input prompt 122 can also include other text. For example, the prompt engine 120 can include text that defines visual properties, such as graphic style or visual theme.

[0061] The system 100 can provide each input prompt 122 to the machine learning model 130. The machine learning model 130 can process the input prompt 122 to generate an image 132 that represents the set of words corresponding to the input prompt 122. For example, the machine learning model 130 can be a generative model configured to generate images that reflect the text that is input to the generative model.

[0062] The system can provide each image 132 to the user device 104 for presentation to the user 102. The system can provide each image 132 in the order corresponding to the order of the sets of words 112 in the text 108. The system can also provide a transition between each image 132 that animates the visualization of the story. For example, each transition can include one or more images, or a video, that minimizes the visual disruption between two images. The system 100 can thus provide a visualization of the story that was input by the user 102 to the user 102.

[0063] The extraction engine 110, prompt engine 120, and machine learning model 130 are shown in FIG. 1 as executing separately from the user device 104. For example, the extraction engine 110, prompt engine 120, and machine learning model 130 can execute on a remote server or other computer(s) remote from the user device 104. In some implementations, the extraction engine 110, prompt engine 120, and machine learning model 130 can execute on the user device 104. In some implementations, the machine learning model 130 can run on one or more computers independently from the other components. For example, the other components of the system 100 can run on the user device 104, and the machine learning model 130 can run on one or more computers remote from the user device 104. In other examples, the components of the system 100 can run on one or more computers remote from the user device 104, and the machine learning model 130 can run on other computers remote from the components of the system 100. The other components of the system 100 can communicate with the machine learning model 130 using an application programming interface (API), for example. In these implementations, the system 100 can communicate with the machine learning model 130 by sending input prompts 122 to the machine learning model 130 through the API, and receiving images generated by the machine learning model.

[0064] FIG. 2 is a flow diagram of an example process 200 for generating a visualization of a story. The process 200 can be performed by any appropriate system, e.g., the system 100 described above with reference to FIG. 1.

[0065] The system receives an input from a user of a user device that includes text (step 210). The text represents a story to be visualized through multiple images, or through a video.

[0066] In some implementations, the input can include data representing text. In some implementations, the system can generate the text from a voice input from the user. In these implementations, the input can include audio data that includes a recording of spoken text. For example, the system can convert the audio data into data representing the text in the recording. An example of an input from a user is shown below in FIG. 3.

[0067] In response to receiving the input, the system obtains a plurality of sets of words from the text (step 220). For example, the system can provide the text to an extraction engine that is configured to extract multiple sets of words from the text that the extraction engine determines to be important or salient.

[0068] For example, the extraction engine can identify groups of words, such as sentences, that include important or salient terms. The extraction engine can assign an importance score to each group of words that includes important or salient terms. The extraction engine can extract sets of words from groups of words that have importance scores that are over a threshold score. For example, for each group of words that has an importance score over the threshold score, the extraction engine can include the important or salient terms in the corresponding set of words. In some implementations, the extraction engine can extract sets of words from the groups of words that have the top-X highest importance scores, where X is an integer greater than or equal to one. For example, for each group of words that has a top importance score, the extraction engine can include the important or salient terms in the corresponding set of words. The system can thus obtain the sets of words using the extraction engine. An example of extracting sets of words of the story is described below with reference to FIG. 4.

[0069] In some implementations, the system can receive an input from the user that identifies a certain number of images to generate. The system can thus provide the certain number of images to generate to the extraction engine as X, which can extract the certain number of sets of words from the text. For example, the extraction engine can use the groups of words that have the top-X highest importance scores, and extract the important or salient terms from each into the certain number of sets.

[0070] For each set of words, the system generates an input prompt (step 230). The system can format the input prompt to describe to a machine learning model what to generate. The input prompt can include one or more words from the set of words. The input prompt can also include variations of words in the set of words, such as synonyms or different tenses of the words in the set of words. The input prompt can also include other words that describe visual properties. For example, if the set of words includes “girl,” “who wore a red cloak,” and “woods are dangerous,” the input prompt can include “a cartoon image of a girl who is wearing a red cloak. The woods are dangerous.” In some implementations, the system can use a model to generate the input prompt for the set of words. For example, the model can be a machine learning model that is configured to generate natural language text that describes the set oP w ords, and other properties such as visual properties.

[0071] For each set of words, the system obtains an image that represents the input prompt using a first machine learning model (step 240). For example, the system can provide the input prompt to a first machine learning model that is configured to output an image responsive to a given input prompt.

[0072] The first machine learning model can be a machine learning model that is trained to output an image given text that describes the image. For example, the first machine learning model can be a text-conditioned diffusion model. Some examples of text-conditioned diffusion models include Imagen, Stable Diffusion, and DALL-E. The first machine learning model can also be another type of generative model. For example, the first machine learning model can be a Transformer-based model such as Muse. The first machine learning model can also be an autoregressive model such as Parti. The system provides the plurality of images for presentation on the user device (step 250). The plurality of images, where each image corresponds to a set of words, thus visualize the story. An example generated image is shown below in FIG. 7.

[0073] The system can provide the images for presentation in an order corresponding to the chronological order of the sets of words in the story. In some implementations, the system provides the plurality of images and a transition between each of the images for presentation on the user device. For example, the system can cause the user device to display each image sequentially. In some implementations, the system can provide a transition between each pair of images to display. A transition between two images can include one or more image frames that include pixels that make the visual changes between two images less abrupt.

[0074] In some implementations, the system can cause the user device to display multiple images in separate portions of the display. For example, the images can be displayed in a grid layout. In some examples, the system can cause the user device to display a subset of the images. For example, the system can provide the images for display in a user interface of the user device. The user interface can allow the user to scroll through the subset of images that are displayed to view a different subset of images.

[0075] In some implementations, the system can cause the user device to display subsets of the images in separate portions of the display. For example, as depicted in FIG. 17A, images corresponding to different sections of the story such as “Introduction,” “Body,” and “Conclusion” can be grouped together into subsets. Each image can be displayed in the same portion of the display as other images of the same subset. In some implementations, the different sections of the story can be defined by the user, such as through a narrative label input described below.

[0076] In some implementations, providing the images for presentation can include providing audio to play on the user device. For example, the system can generate or obtain music or sound effects that match the text of the story and / or the content of the images that are generated.

[0077] In some implementations, the system can allow a user to annotate the sets of words. For example, the system can allow a user to input a tone for a set of words, such as sad, surprised, or happy. The system can also allow a user to input a narrative label for a set of words, such as a cliffhanger, a plot twist, a character, a scene, a background, or a conflict. The system can use the inputs as context when generating the visualization. For example, the system can include the inputs in the input prompts generated in step 230. The system can also allow a user to input audio or effects to a set of words. The system can include the input audio or effects when providing the visualization for display to the user.

[0078] The system can allow a user to customize the length of the visualization that is generated. For example, after viewing the visualization, the user may decide they would like a shorter visualization. The system can receive an input from the user that identifies a shorter length of the visualization to be generated. The system can repeat the process 200. To focus the visualization on critical moments in the story, the system can increase the threshold score that determines whether the system extracts a set of words from a group of words in step 220, for example. In some implementations, the system can combine multiple sets of words into one set of words. The system can thus obtain fewer sets of words and generate a visualization with fewer images.

[0079] As another example, after viewing the visualization, the user may decide they would like a longer visualization. The system can receive an input from the user that identifies a longer length of the visualization to be generated. The system can repeat the process 200. To generate a longer visualization, the system can decrease the threshold score that determines whether the system extracts a set of words from a group of words in step 220, for example. The system can thus obtain more sets of words and generate a visualization with more images.

[0080] The system can allow a user to add an image to the images that are generated. For example, after viewing the images, the user may decide that they would like an additional scene from the story to be visualized. Tn some implementations, the system can receive an input from the user that identifies an additional set of words from the text. In response to receiving the input, the system can generate an additional input prompt that includes the additional set of words. The system can obtain an additional image that represents the additional input prompt by providing the additional input prompt to the first machine learning model. The system can provide the additional image with the plurality of images for presentation on the user device. The system can determine what order to provide the additional image and the plurality of images based on the order of the corresponding sets of words within the text. The system can provide the additional image with the plurality of images in an order that corresponds to a chronological order of the plurality of sets of words and the additional set of words in the text.

[0081] The system can allow a user to change the order of the presentation of the images. For example, after viewing the images, the user may decide that they would like the images to be displayed in a different order than the system provided for display. In some implementations, the system can provide the plurality of images for presentation in a user interface of the user device in an order corresponding to a chronological order of the plurality of sets of words in the text. For example, the user interface can display all of the images in the order in separate regions of the display. An example of images displayed in order is shown below in FIG. 17B. The user interface can include a user interface element that is controllable by the user to adjust the order of the images. The system can receive an input from the user that identifies the order of the images to adjust. For example, the user interface can include a drag-and-drop user interface element that allows the user to select a particular image that is located in a particular region of the display in a position relative to other images, and move the particular image to another region of the display to a second position relative to other images. The input can thus identify that the order of the images after the particular image has been moved to its second position is the desired adjusted order. In response to receiving the input, the system can provide the plurality of images in the adjusted order for presentation on the user device.

[0082] The system can allow a user to modify which words of the text are included in each set of words. The system can thus allow a user to customize what is included in each scene. For example, the user may decide that they would like to include additional words from the text in a particular set of words. The user may also decide they would like to remove words from the particular set of words. The user may also decide they would like to include different words in the particular set of words. An example process is described below with reference to FIG. 5. An example user interface element that allows a user to modify words in a set of words is shown in FIG. 6. The system can allow the user to modify the words of the text that are included in each set of words before the system generates images for the sets of words. For example, the system can allow the user to modify the words that are included in each set of words before the system generates an input prompt for the set of the words. In some examples, the system can allow the user to modify the words of the text that are included in each set of words after the system generates images for the sets of words. For example, the system can allow the user to modify the words that are included in each set of words after the system provides the images generated by a machine learning model for presentation on the user device.

[0083] The system can allow a user to modify properties of the images that are generated. The property can be, for example, a visual theme or a graphic style of the image. An example process for modifying properties of the images that are generated is discussed below with reference to FIG. 8. The system can also allow a user to quickly modify properties of multiple images that are generated after selecting options for properties for one image. Examples of modified images are shown below in FIGS. 9-10.

[0084] The system can allow a user to provide sources of inspiration for the images that are generated. For example, the user may like the graphic style of an existing image, the lighting in an existing video, or the audio from an existing video. An example process is described below with reference to FIG. 11.

[0085] The system can allow a user to combine elements of images that are generated. Example processes are discussed below with reference to FIGS 15-16.

[0086] FIG. 3 shows an example presentation 300 on a user device 104 of an input from a user. The system can provide the input for presentation on the user device. The input includes text 108 that represents a story that the system described in this specification can visualize through images. For example, the text 108 includes “The title is Little Red Riding Hood” and “Description: This is a famous and old story of a girl who wore a red cloak...” The system can receive the text 108 and generate images that visualize the story of text 108 as described above. In some implementations, the system can generate the text 108 from a voice input from the user. For example, the user can use a buton 310 to start a voice recording function on the user device. The system can convert the spoken words of the voice recording to text while the user records their voice. For example, the presentation 300 shows that the text 108 of the story is not yet complete and that the user may still be inputting the text 108. The system can use a speech-to-text program, for example, to generate the text 108 from the voice input. The user can use the buton 31 to stop the voice recording function. In some implementations, the system can save a complete voice recording. The system can convert the spoken words of the voice recording into text 108 using a speech-to-text program, for example.

[0087] FIG. 4 shows an example presentation 400 on a user device 104 of multiple sets of words in a story. The system can indicate the words that are extracted from the text in the presentation 400. For example, the extracted sets of words 410 are shown in the presentation 400 as highlighted in boxes. The system can extract sets of words 410 from the text that the system determines to be important or salient. The system can use an extraction engine to obtain the multiple sets of words. The extraction engine can include a machine learning model, for example, that is trained to determine important or salient terms in an input string of text. The machine learning model can be trained to determine that subjects, actions, and descriptors of subjects are important or salient. The extraction engine can group the important or salient terms into sets. Each set can represent a scene in the story .

[0088] The extraction engine can include salient or important terms in the sets of words. As an example, the extraction engine can determine that a subject, a “girl,” is an important term. The extraction engine can also determine that verbs or actions are important terms. The extraction engine can also determine that descriptors of the girl, such as “who wore a red cloak” are important or salient terms. In some implementations, each salient term can be associated with a class For example, a term such as “girl” can be in a character class. Terms such as “wore a red cloak” can be in a descriptor class. Other examples of classes can include actions, locations, and objects.

[0089] The extraction engine can divide the text into high priority groups of words that each correspond to a scene. A high priority group of words can be a group of words within a certain proximity to each other that include one or more salient terms. For example, a high priority group of words can be a group of words, such as a sentence or paragraph, that includes one or more important or salient terms. For example, the extraction engine can identify the sentence that includes “girl” as a high priority group of words. In the example of FIG. 4, the high priority group of words that includes “girl” includes the sentence that ends with “story of a girl who wore a red cloak.”

[0090] In some implementations, the extraction engine can rank the high priority groups of words and select the top-ranked high priority groups of words to include as scenes. For example, the extraction engine can determine an importance score for each high priority group of words. The importance score for each high priority group of words can be based on a number of salient terms in the group of words, a proportion of salient terms in the group of words, and / or the classes of the salient terms in the group of words. For example, the importance score can be based on the number and classes of the salient terms. Each class can have an associated weight. For example, the character class can have a higher weight than the location class. A high priority group of words that includes two characters can have a higher importance score than a high priority group of words that includes two locations, for example.

[0091] In the example of FIG. 4, the sentence of box 410c can have a higher importance score than the sentence of box 410b, because the sentence of box 410b includes one character and one location, and the sentence of box 410c includes two characters.

[0092] The extraction engine can select the top-X high priority groups of words that have the highest importance scores, where X is an integer greater than or equal to one. In some implementations, X can be a default number. In some implementations, the system can receive an input that defines X. In some implementations, the extraction engine can select high priority groups of words that have importance scores over a threshold importance score.

[0093] The extraction engine can include salient or important terms from each selected high priority group of words in a corresponding set of words. In some implementations, the extraction engine can include other terms from the selected high priority group of words in a corresponding set of words.

[0094] In the example of FIG. 4, the extraction engine can group “girl who wore a red cloak” (box 410a) into a first set, “She lived in a village” (box 410b) into a second set, “Little Red Riding Hood asked her mother” (box 410c) into a third set, “they packed a nice basket” (box 41 Od) into a fourth set, and “woods are dangerous” (box 41 Oe) into a fifth set. .

[0095] In some implementations, the extraction engine can include terms from another set of words in a particular set of words. For example, if a particular set of words includes the salient term “she,” the extraction engine may include descriptors from another set of words such as “wore a red cloak” in the particular set of words. The extraction engine can determine that “she” in the context of the text refers to the same character who was described as wearing a red cloak, for example.

[0096] As another example, the extraction engine can combine terms from multiple sets of words into one set of words for a more detailed scene. For example, the extraction engine may extract the sets of words “girl who wore a red cloak,” “she lived in a village near the forest,” and “woods are dangerous.” The extraction engine can combine the terms into one set of words.

[0097] In some implementations, the extraction engine can extract sets of words from the text while the input is received. That is, the system can provide the text of the input to the extraction engine and obtain sets of words using the extraction engine while the input is received. For example, the extraction engine can determine that terms are likely to be important or salient without having received the complete text of the story, as shown in FIG.

[0098] 3 where the story is not complete and some terms are already highlighted.

[0099] In some implementations, the system can receive an input from the user that identifies a certain number of images to generate. In some implementations, the input that identifies a certain number of images to generate can be generated by the system. For example, the certain number of images to generate can be a default number of images to generate. The extraction engine can group important or salient terms in the text into the certain number of sets. For example, the system can receive an input from the user that indicates the user wants three images to be generated. The system can provide the input from the user to the extraction engine. The extraction engine can group the extracted words 410 into three sets of words. For example, the extraction engine can rank the groups of extracted words, that is, the sentences that include one or more salient terms 410, by importance score. The extraction engine can extract a set of words for each of the top three groups that have the highest importance scores. That is, the extraction engine can include the important or salient terms from each of the three groups in three corresponding sets of words. The system can obtain the sets of words from the extraction engine.

[0100] FIG. 5 is a flow diagram of an example process 500 for generating a modified image from a modified set of words. The process 500 can be performed after the process 200 of FIG. 2, for example.

[0101] The system can provide the plurality of sets of words for presentation in a user interface of the user device (step 510). The user interface can include a user interface element that is controllable by the user to modify one or more words in a particular set of words. For example, the user interface can include a user interface element that allows a user to select words that are not in a set of words, or remove words from the set of words. An example user interface element is shown in FIG. 6. The user interface can include a user interface element 610 that is controllable by the user to modify one or more words in a particular set of words. For example, the user interface element 610 can allow the user to highlight words that are not currently highlighted. For example, the plurality of generated images may include a first image of Little Red Riding Hood, a second image of Little Red Riding Hood in a village near the forest, a third image of Little Red Riding Hood and her mother, a fourth image of a nice basket, and a fifth image of dangerous woods. The user may decide they would like one of the images to include only the red cloak. The user can control user interface element 610 to highlight additional words.

[0102] In response to receiving an input from the user to the user interface element, the system can generate a modified input prompt (step 520). The modified input prompt can include the modified words of the particular set of words. For example, the modified input prompt can include additional words, fewer words, or different words than the input prompt.

[0103] For example, referring to FIG. 6, the system can receive an input to the user interface element 610 that indicates a highlight was added to the words “near the forest” (box 620). The system can determine that the additional words are in the same set of words that includes “She lived in a village” (box 410b). The system can thus determine that some words of the set of words have been modified. The system can generate a modified input prompt that includes one or more words of the modified words. For example, the modified input prompt can include the additional words “near the forest” (box 620) as well as the currently highlighted words of the set of words. In some examples, the modified input prompt can include synonyms or different tenses of the modified words.

[0104] The system can obtain a modified image (step 540). The modified image can represent the modified input prompt. For example, the system can provide the modified input prompt to the first machine learning model. In the example above, the modified first image can depict a village that is near a forest.

[0105] The system can provide the modified image for presentation on the user device (step 550). In some implementations, the system can provide the modified image for presentation with the plurality of images that were not modified. For example, the system can provide the modified image for presentation along with the other images in the same order as the original plurality of images. In the example above, the system can provide an image of Little Red Riding Hood, a transition, and the modified image of a village near a forest.

[0106] As another example, referring to FIG. 6, the user interface element 610 can allow the user to un-highlight words that are currently highlighted. For example, the user may decide they do not want the third image to include Little Red Riding Hood’s mother. The user can control user interface element 610 to remove the highlight from words that are currently highlighted. The system can receive an input to the user interface element 610 that indicates the highlight was removed from the words “her mother” (box 410c). The system can determine that some words of the set of words that includes “her mother” (box 410c) have been modified. The system can generate a modified input prompt that includes the modified words. For example, the modified input prompt can include the currently highlighted words of the set of words, except for “her mother” (box 410c). The system can obtain a modified image that represents the modified input prompt by providing the modified input prompt to the first machine learning model. For example, the modified image can depict Little Red Riding Hood. The system can provide the modified image for presentation on the user device. For example, the system can provide the modified image for presentation along with the other images in the same order as the original image.

[0107] In some examples, the user interface element 610 can allow the user to change the w ords that are currently highlighted, or replace a currently highlighted word with another w ord. The user interface element 610 can allow the user to un-highlight words that are currently highlighted and highlight words that are not currently highlighted in a particular set. The system can receive an input to the user interface element that indicates a highlight was added to some words, and the highlight was removed from some words. The system can determine that some words in the particular set of words have been modified. The system can generate a modified input prompt that includes the modified words. The system can obtain a modified image that represents the modified input prompt by providing the modified input prompt to the first machine learning model. The system can provide the modified image for presentation on the user device. For example, the system can provide the modified image for presentation along with the other images in the same order as the original image FIG. 7 shows an example presentation 700 on a user device 104 of an image 710 generated by a machine learning model. The image 710 depicts a girl wearing a red cloak walking through a shaded forest. For example, the image 710 can be generated in step 240 of the process 200. The system can obtain an image that represents an input prompt corresponding to a set of words by providing the input prompt to a machine learning model. Referring to the text in FIG. 4 and 6, the input prompt for the image 710 can include “girl who wore a red cloak,” “lived in a village near the forest,” and “woods are dangerous.” In some implementations, the input prompt for the image 710 can include synonyms or variations of the words in the set of words. In some implementations, the input prompt can include other text that describes properties of the image to be generated by the machine learning model. For example, the input prompt for the image 710 can include “Generate an image of a girl who wore a red cloak who lives in a village near the forest. The forest is dangerous.”

[0108] FIG. 8 is a flow diagram of an example process 800 for modifying a particular image. The process 800 can be performed as part of step 250 of process 200, for example.

[0109] The system provides the plurality of images for presentation in a user interface of the user device. The user interface can include a plurality of user interface elements. Each user interface element can be controlled by the user to modify a corresponding property of a particular image being presented. The user can control a user interface element to select among a set of options for a property (step 810). Each option can be associated with a different modification of the input prompt for the particular image being presented.

[0110] In response to receiving an input to a user interface element, the system can generate a modified input prompt (step 820). The system can apply the modification associated with the option to the input prompt. For example, the system can include words describing the option in the input prompt. The system can obtain a modified particular image (step 830). The modified particular image can represent the modified input prompt. The system can provide the modified input prompt to the first machine learning model to obtain the modified particular image.

[0111] The system can provide the modified particular image for presentation on the user device (step 840). Examples are discussed below with reference to FIGS. 9-10.

[0112] The system can also allow a user to modify properties of multiple images that are generated after selecting options for properties for one image. The system can receive an input from the user that identifies one or more options for modifying one or more corresponding properties of a second image being presented. For example, the options can have been selected by the user for a first image. The system can save data representing the options selected for the first image, for example. In response to receiving the input, the system can generate a modified input prompt. The system can apply the modification associated with each option identified in the input to the input prompt. For example, the system can include the modification of the input prompt associated with each option. The system can then obtain a modified second image that represents the modified input prompt by providing the modified input prompt to the first machine learning model. The system can provide the modified second image for presentation on the user device.

[0113] In some implementations, the system can allow a user to modify portions of a particular image. For example, the system can receive an input from the user that includes text that describes an aspect of the particular image to be changed. For example, the particular image may depict a gray house. The text may include “brown house.” In response to receiving an input from the user to the user interface element, the system can generate a modified input prompt that includes the input from the user and the particular image. The system can obtain a modified particular image by providing the modified input prompt to a machine learning model that is configured to perform image editing. Example models include Muse or other image generation models that perform mask-free image editing.

[0114] FIGS. 9A-9E show an example presentation 900 on a user device 104 of modifying a visual theme of an image 910. As shown in FIG. 9A, the system can provide an image 910 for presentation in a user interface of the user device 904. The image 910 was generated in a similar manner to the image 710 of FIG. 7, for example. The user interface can include a user interface element 920 that allows the user to choose a property of the image 910 to modify. For example, the user interface element 920 can be a button. In the example of FIG. 9A, the user interface element 920 allows the user to choose to modify a “theme” or visual theme of the image 910. Other properties can include a graphic style, for example.

[0115] The user interface can also include user interface elements 930. The user interface elements 930 can correspond to a set of options for the visual theme property. For example, user interface element 930a can correspond to a childish or “kids” theme. User interface element 930b can correspond to a romantic theme. User interface element 930c can correspond to an action theme. User interface element 930d can correspond to a scary theme. Other options for the visual theme for an image can include a comedy theme, a scientific fiction theme, a thriller theme, a drama theme, etc. Each option can be associated with a different modification of an input prompt. For example, the “kids” theme option can be associated with a modification of “appropriate for a children’s book.” The romantic theme option can be associated with a modification of “in a romantic mood,” for example.

[0116] FIG. 9B shows that the user has selected the “kids” theme using user interface element 930a. The system can receive an input to the user interface element 930a. As described above with reference to FIG. 8, the system can generate a modified input prompt by applying the modification associated with the option to the input prompt. The input prompt for the image 910 could have included, for example, “Generate an image of a girl who wore a red cloak, and lived in a village near the forest. The forest is dangerous.” The modified input prompt can include: “Generate an image appropriate for a children’s book of a girl who wore a red cloak, and lived in a village near the forest. The forest is dangerous,” for example.

[0117] FIG. 9C shows the modified image 950. The system can provide the modified image 950 for presentation on the user device 104. The modified image 950 depicts the set of words in a more child-friendly manner than the image 910.

[0118] FIG. 9D shows that the user has selected the “scary” theme using user interface element 930d. The system can receive an input to the user interface element 930d. As described above with reference to FIG. 8, the system can generate a modified input prompt by applying the modification associated with the option to the input prompt. The input prompt for the image 950 could have included, for example, “Generate an image appropriate for a children’s book of a girl who wore a red cloak, and lived in a village near the forest. The forest is dangerous.” The modified input prompt can include: “Generate an image that might evoke a sense of fear in a viewer of a girl who wore a red cloak, and lived in a village near the forest. The forest is dangerous,” for example. FIG. 9E shows the modified image 960. The system can provide the modified image 960 for presentation on the user device 104. The modified image 960 depicts the set of words in a scarier manner than the image 950.

[0119] In some implementations, the user interface can include additional user interface elements for each property. For example, the “scary” theme can have multiple options with different levels of intensity. For example, the user can select among options such as “very scary,” “slightly scary.” The system can thus allow a user to further refine the visualization of the story.

[0120] FIGS. 10A-10E show an example presentation 1000 on a user device 104 of modifying a graphic style of an image 1010. As show n in FIG. 9A, the system can provide the image 1010 for presentation in a user interface of the user device 904. The image 1010 can have been generated in a similar manner to the image 710 of FIG. 7, for example. The image 1010 can also have been generated in a similar manner to the image 960 of FIG. 9, for example. The user interface can include a user interface element 1020 that allows the user to choose a property of the image 1010 to modify. For example, the user interface element 1020 can be a button. In the example of FIG. 10A, the user interface element 1020 allows the user to choose to modify a “style” or graphic style of the image 1010.

[0121] The user interface can also include user interface elements 1030. The user interface elements 1030 can correspond to a set of options for the graphic style property. For example, user interface element 1030a can correspond to a cartoon graphic style. User interface element 1030b can correspond to a realistic graphic style. User interface element 1030c can correspond to a dark graphic style. User interface element 1030d can correspond to a sketched graphic style. Other options for the graphic style for an image can include a colorful graphic style, a vibrant graphic style, etc. Each option can be associated with a different modification of an input prompt. For example, the cartoon style option can be associated with a modification of “in a cartoon style.” The dark style option can be associated with a modification of “in a dark style,” for example.

[0122] FIG. 10B shows that the user has selected the cartoon graphic style using user interface element 1030a. The system can receive an input to the user interface element 1030a. As described above with reference to FIG. 8, the system can generate a modified input prompt by applying the modification associated with the option to the input prompt. The input prompt for the image 1010 could have included, for example, “Generate an image that might evoke a sense of fear in a viewer of a girl who wore a red cloak, and lived in a village near the forest. The forest is dangerous.” The modified input prompt can include: “Generate an image that might evoke a sense of fear in a viewer of a girl who wore a red cloak, and lived in a village near the forest. The forest is dangerous. The image is in a cartoon style,” for example.

[0123] FIG. IOC shows the modified image 1050. The system can provide the modified image 1050 for presentation on the user device 104. The modified image 1050 depicts the set of words in a cartoon style.

[0124] FIG. 10D shows that the user has selected the dark graphic style using user interface element 1030c. The system can receive an input to the user interface element 1030c. As described above with reference to FIG. 8, the system can generate a modified input prompt by applying the modification associated with the option to the input prompt. The input prompt for the image 1050 could have included, for example, “Generate an image that might evoke a sense of fear in a viewer of a girl who wore a red cloak, and lived in a village near the forest. The forest is dangerous. The image is in a cartoon style.” The modified input prompt can include: “Generate an image that might evoke a sense of fear in a viewer of a girl who wore a red cloak, and lived in a village near the forest. The forest is dangerous. The image is in a dark style,” for example.

[0125] FIG. 10E shows the modified image 1060. The system can provide the modified image 1060 for presentation on the user device 104. The modified image 1060 depicts the set of words in a dark graphic style.

[0126] FIG. 11 is a flow diagram of an example process 1100 for generating images based on reference images. The process 1100 can be performed with the process 200, for example.

[0127] The system receives a first input from the user that includes a reference image (step 1110). In some implementations, the system can also receive inputs from the user that include other types of data. For example, the input can include reference videos, reference text, and / or reference audio. An example of reference images is shown in FIG. 12. For example, FIG. 12 shows an example presentation 1200 on a user device 104 of reference images 1210. The reference images 1210 can be uploaded or selected by the user. In some examples, the user can upload reference video, audio, and / or text. The system can receive a first input from the user that includes a reference image such as the reference images 1210.

[0128] In response to receiving the first input, the system can obtain text that describes the reference image (step 1120). For example, the system can obtain text that describes the reference image by providing the reference image to a third machine learning model. The third machine learning model can be configured to generate text that describes an image. For example, the third machine learning model can be a machine learning model that is trained to output text that describes an inputted image. The third machine learning model can be a caption generating model. For example, the third machine learning model can be configured to detect and describe the content of an inputted image. For example, the output text can include a description of what is depicted in the inputted image, and / or a description of visual properties of the image such as graphic style and visual theme. In implementations where the input includes other types of data such as video or audio, the system can use a machine learning model that is configured to generate text that describes the reference video or reference audio, for example.

[0129] The system can provide the text for presentation on the user device (step 1130). For example, the text that describes the reference image 1210a of FIG. 12 can include “a gray house with a gable roof. The house has multiple windows and garage doors. The house has a black lamp in between the garage doors. The image has a dull color palette. It is daytime.” The system can provide the text for presentation, overlaid over the reference image 1210a or near the reference image 1210a, for example.

[0130] The system can receive a second input from the user that identifies a subset of the text (step 1140). For example, the text that the system generates to describe the reference image may include descriptions of aspects of the image that a user may not want to include in their visualization of the story. For example, the user may have been inspired by the color and lamp of the house of reference image 1210a. The system can allow the user to filter the text. For example, the system can allow the user to select or highlight certain words of the text. In the example of the reference image 1210a, the user can highlight “gray house with a gable roof’ and “the house has a black lamp.” The system can thus receive a second input from the user that identifies “gray house with a gable roof’ and “the house has a black lamp” as the subset of the text.

[0131] The system can identify at least one set of words from the plurality of sets of words as relevant to the subset of the text (step 1150). For example, the aspects of the reference image that the user identifies may not be applicable to all of the scenes of the story. As an example, a first set of words can include “girl who wore a red cloak” and “lived in a village near the forest.” A second set of words can include “woods are dangerous.” The system can determine that the subset of the text, “gray house with a gable roof’ and “the house has a black lamp” is relevant to the first set of words because the subset of the text describes a house, which is a dwelling structure that a “girl who wore a red cloak” can live in and be located in a village near the forest. The system can thus identify the first set of words as relevant to the subset of the text. In some implementations, the system can use a machine learning model that is configured to output a score that represents a relevancy between two input texts. The system can provide each set of words and the subset of the text as input to the machine learning model. The system can identify a set of words as relevant to the subset if the score for the set of words is over a threshold score, for example.

[0132] In some implementations, the system can receive an input from the user that identifies which set of words is relevant to the subset of the text. For example, the input can identify that the user wants to use the style of the forest from the reference image 1210c in the first generated image that corresponds to the first set of words, but not the second generated image that corresponds to the second set of words.

[0133] For each set of words that the system identified as relevant to the subset of the text, the system can generate an input prompt (step 1160). The input prompt can include the subset of the text and one or more words from the relevant set of words. For example, the input prompt can include “girl who wore a red cloak,” “lived in a village near the forest,” “gray house with a gable roof,” and “the house has a black lamp.” The input prompt can also include variations of the subset of the text and the relevant set of words, or additional words, as described above. For example, the input prompt can include “an image of a girl who is w earing a red cloak. She lives in a gray house with a gable roof and the house has a black lamp. The house is located in a village near the forest.” In some examples, the input prompt can include synonyms or different tenses of one or more words in the subset of the text, or different tenses of one or more words in the relevant set of words.

[0134] In some implementations, for each set of words that the system identified as relevant to the subset of the text, the system can generate an input prompt that includes the subset of the text and one or more words from the relevant set of words, and other words of the text that are relevant to the subset of the text. For example, the subset of the text for reference image 1210a can include “house.” The system can include in the input prompt other words of the text that are relevant to the “house” such as descriptors like “gray” and “with a gable roof.”

[0135] The system can obtain an image that represents the input prompt (step 1170). The system can obtain an image that represents the input prompt by providing the input prompt to the first machine learning model. The system can thus generate an image representing the subset of the text and the relevant set of words in the input prompt. For example, the system can generate an image based on the reference image 1210a that depicts a girl wearing a red cloak and a gray house with a gable roof with a black lamp, as shown in FIG. 14. The system can provide each image for presentation on the user device (step 1180). An example presentation 1400 of the image 1410 corresponding to the first set of words and based on reference image 1210a is shown in FIG. 14. For example, the system can provide each image that corresponds to a set of words that was identified as relevant to the subset of the text for presentation. In some implementations, the system can provide each image that corresponds to a set of words that was identified as relevant in the context of all of the generated images. For example, the system can provide each image for presentation along with the other images in the same order as the original images.

[0136] The system can allow a user to edit the type of inspiration that the system derives from a reference image. For example, the system can generate text that describes the reference image 1210b that includes “an object on a wood table in bright lighting.” After viewing the text, the user may decide that they are more inspired by the color palette of the reference image 1201b rather than the subject or lighting. The system can receive an input from the user that includes different text that describes the reference image than was generated by the third machine learning model. For example, the input can include text such as “warm neutral colors.” As another example, the user may want to refine the text. The input can include text such as “the object is a coffee cup.”

[0137] In response to receiving the input, the system can generate a modified input prompt that includes one or more words from the text of the input. For example, the modified input prompt can include one or more words from the relevant set of words and “warm neutral colors.” The system can obtain a modified image that represents the modified input prompt by providing the modified input prompt to the first machine learning model. The system can thus generate a modified image representing the modified input prompt. The system can then provide the modified image for presentation on the user device. The modified image can thus depict the same set of words but with a warm neutral color palette.

[0138] FIG. 13 shows an example presentation 1300 on a user device 104 for generating images based on reference images. The user can use the example presentation 1300 to select which aspects of reference images they would like to use. As described above with reference to FIGS. 11-12, the system can receive a first input from the user that includes a reference image. For example, the first input from the user can include the reference image 1210a.

[0139] The system can receive a second input from the user that identifies an aspect of the reference image. The second input can identify the aspect of the reference image that the user was inspired by. For example, the aspect can be an object or subject depicted in the reference image, such as “house” or “forest.” The aspect can also be a property of the reference image, such as the graphic style or visual theme of the reference image. The aspect can also be another visual feature of the reference image, such as lighting.

[0140] The system can receive the second input through user interface elements 1320, for example. User interface element 1320a identifies an aspect of picture 1 as “house.” User interface element 1320b identifies an aspect of picture 2 as “light.” The user interface elements 1320 can be text boxes, for example. The system can distinguish between different reference images through user interface elements 1330. For example, user interface element 1330e corresponds to reference image 1210e. User interface element 1330c corresponds to reference image 1210c. User interface element 1330a corresponds to reference image 1210a. The system can receive an input through user interface elements 1330 that identifies which reference image 1210 the system should process.

[0141] In response to receiving the second input, as described above, the system can obtain text that describes the reference image 1210. For example, the system can provide the reference image 1210 to the third machine learning model.

[0142] The system can identify a subset of the text that is relevant to the aspect of the reference image. For example, the system can use a machine learning model that, given the text and an aspect of the reference image, is configured to output a subset of the text that is relevant to the aspect of the reference image. For the example of reference image 1310a, the text can include “a gray house with a gable roof. The house has multiple windows and garage doors. The house has a black lamp in between the garage doors. The image has a dull color palette. It is daytime.” The aspect of picture 1 identified by the user can be “house.” The machine learning model can identify that the subset of the text that is relevant to a house is “a gray house with a gable roof. The house has multiple windows and garage doors. The house has a black lamp in between the garage doors.”

[0143] The system can then perform steps similar to steps 1150- 1180 of FIG. 11 for the subset of the text that is relevant to the aspect of the reference image. For example, the system can identify at least one set of words of the sets of words as relevant to the subset of the text. For each relevant set of words, the system can generate an input prompt including the subset of the text and one or more words from the relevant set of words, and obtain an image that represents the input prompt by providing the input prompt to the first machine learning model. The system can provide each image for presentation on the user device.

[0144] FIG. 1 is a flow diagram of an example process 1500 for generating an image with elements from a previously generated image. The process 1500 can be performed as part of step 250 of process 200, for example. The system provides the plurality of images for presentation in a user interface of the user device. The user interface can include a plurality of user interface elements. Each user interface element can be controlled by the user to select regions in a particular image being presented to replace corresponding regions in a second image being presented (step 1510). For example, the region can include a subject in the foreground of the particular image. The region can also include the background of the particular image. The user interface element can be, for example, a snippet tool or a selection tool that can be used to select regions in the particular image. For example, the particular image being presented can depict a girl wearing a red cloak in a dark forest. The second image being presented can depict a wolf in a dark forest that has different features from the dark forest of the particular image. The user can use a snippet tool to select the background of the particular image.

[0145] In response to receiving an input from the user to the user interface element, the system can generate an image mask (step 1 20). The image mask can include pixels from the selected regions in the particular image, and noisy pixels in regions in the particular image that have not been selected by the user. For example, if the input indicates that a subject in the foreground of the particular image was selected, the image mask can include the pixels of the subject in the foreground. The image mask can include noisy pixels in the regions in the background. In the above example, the image mask can include the pixels of the background of the particular image, and noisy pixels in the foreground that replace the original subject (the girl wearing a red cloak).

[0146] The system can generate a modified input prompt (step 1540) The system can apply the description of the not selected regions to the input prompt of the second image. The description of the not selected regions can include a description of the location of the not selected regions in the particular image. For example, the description can include “foreground of the image” or “lower left comer of the image.” In some implementations, the system can format the modified input prompt to describe to a second machine learning model what to generate. For example, the original input prompt of the second image can include “wolf in a dangerous forest.” The modified input prompt can include “wolf in a dangerous forest. Use the input image mask and put the wolf in the foreground.”

[0147] The system can obtain a modified second image (step 1550). The modified second image can include the selected regions. The system can obtain a modified second image by providing the modified input prompt and the image mask to a second machine learning model. The second machine learning model can be configured to generate an image responsive to a given input prompt with regions from an image mask. That is, the image generated by the second machine learning model includes regions from an image mask and represents the input prompt. For example, the system can obtain an image from the second machine learning model that depicts a wolf in the foreground, with the background of the particular image. The second machine learning model can be a text-conditioned diffusion model. Some examples of text-conditioned diffusion models include Imagen, Stable Diffusion, and DALL-E. The second machine learning model can also be another type of generative model. For example, the second machine learning model can be a Transformerbased model such as Muse. The second machine learning model can also be an autoregressive model such as Parti.

[0148] The system can provide the modified second image for presentation on the user device (step 1560). In some implementations, the system can provide the modified second image for presentation with the other images. For example, the system if the particular image was the first image generated, and the second image was the second image generated, the system can provide the particular image followed by a transition and the modified second image. FIG. 16 is a flow diagram of an example process 1600 for combining elements of generated images. The process 1600 can be performed as part of step 250 of process 200, for example.

[0149] The system can provide the plurality of images for presentation in a user interface of the user device. The user interface can include a user interface element controllable by the user to select regions in a first image and a second image to generate a combined image (step 1610). The user interface element can be, for example, a snippet tool or a selection tool that can be used to select regions in the first and second images. For example, the first image can depict a girl wearing a red cloak in a dark forest. The second image can depict a wolf in a dark forest that has different features from the dark forest of the first image. The user can use a snippet tool to select the girl from the first image and the forest from the second image.

[0150] In response to receiving an input from the user to the user interface element, the system can generate a first image mask (step 1620). The first image mask can include pixels in the selected regions in the first image. For example, the first image mask can include pixels of the girl from the first image and noisy pixels elsewhere.

[0151] The system can generate a second image mask (step 1 30). The second image mask can include pixels in the selected regions in the second image. For example, the second image mask can include pixels of the forest from the second image and noisy pixels elsewhere. The system can process the first and second image mask to generate a combined image (step 1640). For example, the system can overlay the first and second image masks so that there are no noisy pixels in the combined image.

[0152] The system can provide the combined image for presentation on the user device (step 1650). For example, the system can provide the combined image with the other generated images. In some implementations, the system can include a user interface element that allows the user to adjust the order of the images, or to remove images.

[0153] FIG. 17A shows an example presentation 1700 on a user device of images that are part of a visualization of a story. For example, each of the images 1-8 can have been generated by the machine learning model for sets of words. The system can allow the user to select each image such as image 1710 to edit the image, for example. FIG. 17B shows an example presentation 1750 on a user device of the image 1710 that is part of a visualization of a story. The presentation 1750 also shows example user interface elements 1720 and 1730 that the system can receive further user inputs through. For example, the user can use the user interface element 1720 to publish or share the visualization of the story with other users. For example, the system can store data representing the visualization of the story and generate a link to access the data. The user can use the user interface element 1730 to edit the visualization of the story. For example, as described above, the system can allow the user to customize the visualization of the story through adjusting the order of the images, combining elements from images, adding images, removing images, and modifying the properties of images

[0154] Although this specification describes generating multiple images, similar techniques can be used to generate videos with multiple frames per scene, templates, filters, documents, presentations, or other types of media content. For example, to generate a video with multiple frames per scene, the system can use a machine learning model that is configured to generate videos that reflect the input text.

[0155] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially -generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0156] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0157] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0158] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0159] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perfomi functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0160] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0161] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0162] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0163] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0164] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a Jax framework.

[0165] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0166] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0167] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what can be claimed, but rather as descriptions of features that can be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features can be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0168] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0169] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing can be advantageous.

[0170] What is claimed is:

Claims

CLAIMS1. A method comprising: receiving an input from a user of a user device that comprises text representing a story to be visualized through a plurality of images; in response to receiving the input: obtaining a plurality of sets of words from the text; for each set of words in the plurality of sets of words: generating an input prompt for the set of words, wherein the input prompt for the set of words comprises one or more words from the set of words; obtaining an image that represents the input prompt by providing the input prompt to a first machine learning model, wherein the first machine learning model is configured to generate an image responsive to the input prompt; and providing, for presentation on the user device, the plurality of images generated by the first machine learning model that visualize the story.

2. The method of claim 1 , further comprising generating the text from a voice input from the user.

3. The method of claim 1, further comprising receiving an input from the user that identifies a certain number of images to generate, and wherein the plurality of sets of words includes the certain number of sets of words.

4. The method of claim 1, wherein providing, for presentation on the user device, the plurality of images comprises: providing the plurality of images and a respective transition between each of the images in the plurality of images for presentation on the user device.

5. The method of claim 1, wherein providing, for presentation on the user device, the plurality of images comprises: providing the plurality of images for presentation in a user interface of the user device, wherein the user interface comprises a plurality of user interface elements each controllable by the user to modify a respective property of a particular image being presented in the user interface by selecting among a set of options for the property, and wherein each option is associated with a different modification of the input prompt corresponding to theparticular image; in response to receiving an input from the user to a user interface element of the plurality of user interface elements: generating a modified input prompt by applying the modification associated with the option to the input prompt; obtaining a modified particular image that represents the modified input prompt by providing the modified input prompt to the first machine learning model; and providing, for presentation on the user device, the modified particular image.

6. The method of claim 5, wherein the property of the particular image comprises any one of: a visual theme, or a graphic style.

7. The method of claim 5, further comprising: receiving an input from the user that identifies one or more options for modifying one or more corresponding properties of a second image being presented in the user interface; in response to receiving the input: generating a modified input prompt by applying the modification associated wi th each identified option to the input prompt for the second image; obtaining a modified second image by providing the modified input prompt to the first machine learning model; and providing, for presentation on the user device, the modified second image.

8. The method of claim 1, wherein providing, for presentation on the user device, the plurality of images comprises: providing the plurality of images for presentation in a user interface of the user device, wherein the user interface comprises a user interface element controllable by the user to select regions in a particular image being presented in the user interface to replace corresponding regions in a second image being presented in the user interface; in response to receiving an input from the user to the user interface element: generating an image mask comprising pixels from the selected regions in the particular image, and noisy pixels in regions in the particular image that have not been selected by the user; generating a description of the not selected regions; generating a modified input prompt by applying the description of the notselected regions to the input prompt of the second image; obtaining a modified second image comprising the selected regions by providing the modified input prompt and the image mask to a second machine learning model, wherein the second machine learning model is configured to generate an image responsive to the modified input prompt with regions from an image mask; and providing, for presentation on the user device, the modified second image.9 The method of claim 1, wherein providing, for presentation on the user device, the plurality of images comprises: providing the plurality of images for presentation in a user interface of the user device, wherein the user interface comprises a user interface element controllable by the user to select regions in a first image and a second image to generate a combined image; in response to receiving an input from the user to the user interface element: generating a first image mask comprising pixels in the selected regions in the first image; generating a second image mask comprising pixels in the selected regions in the second image; processing the first and second image mask to generate the combined image; and providing, for presentation on the user device, the combined image.

10. The method of claim 1 , further comprising: receiving an input from the user that identifies an additional set of words from the text; in response to receiving the input: generating an additional input prompt comprising the additional set of words; obtaining an additional image that represents the additional input prompt byproviding the additional input prompt to the first machine learning models; and providing, for presentation on the user device, the additional image with the plurality of images, in an order corresponding to a chronological order of the plurality of sets of words and the additional set of words in the text.

11. The method of claim 1 , wherein providing, for presentation on the user device, the plurality of images comprises:providing the plurality of images for presentation in a user interface of the user device in an order corresponding to a chronological order of the plurality of sets of words in the text, wherein the user interface comprises a user interface element controllable by the user to adjust the order of the images in the plurality of images; receiving an input from the user that identifies the order of the images to adjust; in response to receiving the input: providing, for presentation on the user device, the plurality of images in the adjusted order.

12. The method of claim 1, further comprising: providing, for presentation on the user device, the plurality of sets of words in a user interface of the user device, wherein the user interface comprises a user interface element controllable by the user to modify one or more words in a particular set of words; in response to receiving an input from the user to the user interface element: generating a modified input prompt comprising the modified words of the particular set of words; obtaining a modified image representing the modified input prompt by providing the modified input prompt to the first machine learning model; and providing, for presentation on the user device, the modified image.

13. The method of claim 1 , further comprising: receiving a first input from the user that comprises a reference image; in response to receiving the first input: obtaining text that describes the reference image by providing the reference image to a third machine learning model, wherein the third machine learning model is configured to generate text describing an image; providing, for presentation on the user device, the text; receiving a second input from the user that identifies a subset of the text; identifying at least one set of words of the plurality of sets of words as relevant to the subset of the text; for each relevant set of words: generating an input prompt comprising the subset of the text and one or more words from the relevant set of words; obtaining an image that represents the input prompt by providing theinput prompt to the first machine learning model; and providing, for presentation on the user device, each image.

14. The method of claim 13, further comprising: receiving an input from the user that comprises different text that describes the reference image than was obtained by providing the reference image to the third machine learning model; in response to receiving the input: generating a modified input prompt comprising one or more words of the text of the input; obtaining a modified image representing the modified input prompt by providing the modified input prompt to the first machine learning model; and providing, for presentation on the user device, the modified image.

15. The method of claim 1, further comprising: receiving a first input from the user that comprises a reference image; receiving a second input from the user that identifies an aspect of the reference image; in response to receiving the second input: obtaining text that describes the reference image by providing the reference image to a third machine learning model, wherein the third machine learning model is configured to generate text describing an image; identifying a subset of the text that is relevant to the aspect of the reference image; identifying at least one set of words of the plurality of sets of words as relevant to the subset of the text; for each relevant set of words: generating an input prompt comprising the subset of the text and one or more words from the relevant set of words; obtaining an image that represents the input prompt by providing the input prompt to the first machine learning model; and providing, for presentation on the user device, each image.

16. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising: receiving an input from a user of a user device that comprises text representing a story to be visualized through a plurality of images; in response to receiving the input: obtaining a plurality of sets of words from the text; for each set of words in the plurality of sets of words: generating an input prompt for the set of words, wherein the input prompt for the set of words comprises one or more words from the set of words; obtaining an image that represents the input prompt by providing the input prompt to a first machine learning model, wherein the first machine learning model is configured to generate an image responsive to the input prompt; and providing, for presentation on the user device, the plurality of images generated by the first machine learning model that visualize the story.

17. The system of claim 16, further comprising generating the text from a voice input from the user.

18. The system of claim 16, further comprising receiving an input from the user that identifies a certain number of images to generate, and wherein the plurality of sets of words includes the certain number of sets of words.

19. The system of claim 16, wherein providing, for presentation on the user device, the plurality of images comprises: providing the plurality of images and a respective transition between each of the images in the plurality of images for presentation on the user device.

20. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: receiving an input from a user of a user device that comprises text representing a story to be visualized through a plurality of images; in response to receiving the input:obtaining a plurality of sets of words from the text; for each set of words in the plurality of sets of words: generating an input prompt for the set of words, wherein the input prompt for the set of words comprises one or more words from the set of words; obtaining an image that represents the input prompt by providing the input prompt to a first machine learning model, wherein the first machine learning model is configured to generate an image responsive to the input prompt; and providing, for presentation on the user device, the plurality of images generated by the first machine learning model that visualize the story.

Images