Techniques for generating scene based on image blocks using machine learning model

Abstract

Techniques for generating a scene based on image blocks using a machine learning model are disclosed. In various embodiments, examples of the present disclosure provide systems and methods for generating an image of a scene. A plurality of input image blocks based on at least one user input are obtained. A spatial location of each input image block included in the plurality of input image blocks relative to at least one other input image block included in the plurality of input image blocks is detected, and then a scene composition of the scene is determined based on the spatial location of each input image block included in the plurality of image blocks. A scene cue associated with the scene is obtained, and a machine learning model mixes the plurality of image tiles based on the scene cue to generate the scene.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application, entitled “ITERATIVE AND EXPRESSIVE PROMPTING FOR WORLD BUILDING USING GENERATIVE ARTIFICIAL INTELLIGENCE,” filed on August 11, 2023, and having Serial No. 63 / 519,203. The subject matter of this related application is hereby incorporated herein by reference. Background Art Technical Field

[0003] Various embodiments relate generally to computer-aided design and artificial intelligence, and more particularly, to techniques for using machine learning models to generate scenes based on input image patches.

[0004] Description of related technologies

[0005] Creating scenes or unique environments in fictional worlds generally requires a great deal of time and skill from artists. Creation generally involves an overall design process during which the artist first conceives the idea of ​​a scene in the fictional world, followed by a potentially time-consuming creation process of the scene. Some artists may have a level of expertise in utilizing computer-aided design tools in which electronic representations of scenes can be created. As is well known in practice, manually generating a large number of scenes to build a fictional world can be very labor-intensive and time-consuming. Because fictional worlds used in games and storytelling may require a large number of scenes, individual creators may spend a great deal of time creating a single fictional world, which may reduce the quality and / or output of the creator. Therefore, various conventional computer-aided design tools have been developed in an attempt to automate certain aspects of the creation process.

[0006] Some applications for automating some aspects of the creation process for creating scenes for fictional worlds involve domain-specific tools. For example, terrain generation tools can generate various terrains by procedures based on a fixed set of rules or grammars. With such tools, terrain can be automatically generated according to parameters or rules specified by the creator. However, such tools only generate terrain and only generate by procedures.

[0007] Other examples of tools available to creators include 3D world and game level creation tools. However, again, these tools operate by convention, preventing users from using higher-level semantic descriptions to define the fictional world. If users lack the basic knowledge required to utilize computer-aided design tools to create scenes for fictional worlds, artists may waste a lot of time and effort creating fictional worlds or may abandon the creation process altogether.

[0008] Generative tools can be used to create images based on natural language text prompts, thereby semantically guiding the image generation process. However, most generative models are limited to one-time use or "click once" text prompt interfaces, and the text prompt interface output model is trained to be believed to be the content of the finished product. This approach requires the artist to provide a complete and accurate semantic description of the required scene in advance. However, fictional world building is usually an iterative process, and the artist's creativity is expressed in a step-by-step creation process. The creator of the scene in the fictional world usually expresses creativity through sketches or storyboards, and then creates the scene based on the initial idea.

[0009] As described above, there is a need in the art for more efficient techniques for generating scenes of fictional worlds. Summary of the invention

[0010] One embodiment describes a computer-implemented method for generating a scene. In some embodiments, the method includes obtaining a plurality of input image patches based on at least one user input, detecting a spatial location of the plurality of input image patches relative to each other, determining a scene composition of the scene based on the spatial location of the plurality of image patches relative to each other, obtaining scene cues associated with the scene, and executing a first machine learning model to blend the plurality of image patches to form the scene based on the scene cues.

[0011] At least one technical advantage of the disclosed technology over the prior art is that the disclosed technology enables users with limited time or limited knowledge of computer-aided design tools or generation tools to more effectively and efficiently generate scenes for fictional worlds. In this regard, the disclosed technology provides an automated process for obtaining multiple image patches that can be used as a basis for generating scenes for fictional worlds. The image patches can be provided by a user or created using a machine learning model that is provided with user input such as text prompts, sketches, zone maps based on user-selected areas of the image patches, or any combination thereof. The machine learning model can generate one or more image patches based on user input from the spatial relationships of the image patches relative to each other and text scene prompts obtained from the user.

[0012] Therefore, using the disclosed technology, it is easier to generate and produce scenes that are more in line with the user's artistic intentions.These technical advantages provide one or more technical improvements over prior art methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to be able to understand in detail the manner of the above features of various embodiments, the inventive concept briefly summarized above may be described in more detail by reference to various embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the drawings only show typical embodiments of the inventive concept and therefore should not be considered to limit the scope in any way, and there are other equally effective embodiments.

[0014] Figure 1 is a conceptual illustration of a system configured to implement one or more aspects of various embodiments.

[0015] Figures 2 to 10 is an example diagram of a graphical user interface depicting how to generate a scene using a scene creation application, according to various embodiments.

[0016] Fig.11 A flow chart illustrating method steps for generating a scenario using a scenario creation application according to various embodiments is set forth. DETAILED DESCRIPTION

[0017] In the following description, numerous specific details are set forth to provide a more thorough understanding of various embodiments. However, it will be apparent to one skilled in the art that the inventive concept can be practiced without one or more of these specific details. For purposes of explanation, multiple instances of similar objects are symbolized using a reference numeral identifying the object and a parenthetical number identifying the instance.

[0018] Systematic review

[0019] Figure 1 A system 100 configured to implement one or more aspects of various embodiments is shown. The system may include a computing device 101 and a remote computing device 130 communicating via a network 120. The computing device 101 includes a desktop computer, a laptop computer, a smart phone, a personal digital assistant (PDA), a tablet computer, or any other type of computing device configured to receive input, process data, and optionally display images, and is suitable for practicing one or more embodiments. The computing device 101 is configured to run a scene creation application 124 resident in a memory 116.

[0020] Note that the computing devices described herein are illustrative, and any other technically feasible configurations fall within the scope of the present disclosure. For example, multiple instances of the scenario creation application 124 may be executed on a set of nodes in a distributed and / or cloud computing system to implement the functionality of the computing device 101. In another example, the scenario creation application 124 may be executed on a variety of hardware collections, device types, or environments to adapt the scenario creation application 124 to different use cases or applications.

[0021] In one embodiment, the computing device 101 includes, but is not limited to, an interconnect (bus) 112 connecting one or more processors 102, an input / output (I / O) device interface 104 coupled to one or more input / output (I / O) devices 108, a memory 116, a storage device 114, and a network interface 106. The processor 102 may be any suitable processor implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), an artificial intelligence (AI) accelerator, any other type of processing unit, or a combination of different processing units, such as a CPU configured to operate in conjunction with a GPU. In general, the processor 102 may be any technically feasible hardware unit capable of processing data and / or executing software applications. In addition, in the context of the present disclosure, the computing elements shown in the computing device 101 may correspond to a physical computing system (e.g., a system in a data center) or may be a virtual computing instance executed within a computing cloud.

[0022] I / O devices 108 include devices capable of providing input, such as a keyboard, a mouse, a touch-sensitive screen, and the like, and devices capable of providing output, such as a display device. In addition, I / O devices 108 may include devices capable of both receiving input and providing output, such as a touch screen, a universal serial bus (USB) port, and the like. I / O devices 108 may be configured to receive various types of input from an end user (e.g., a designer) of computing device 101, and also provide various types of output to the end user of computing device 101, such as displayed digital images or digital video or text. In some embodiments, one or more of I / O devices 108 are configured to couple computing device 101 to network 120.

[0023] The network 120 is any technically feasible type of communication network that allows data to be exchanged between the computing device 101 and an external entity or device (such as a network server or another networked computing device). For example, the network 120 may include a wide area network (WAN), a local area network (LAN), a wireless (WiFi) network, and / or the Internet, etc.

[0024] The storage device 114 includes non-volatile storage devices for applications and data, and may include fixed or removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other magnetic, optical or solid-state storage devices. The scene creation application 124 can be stored in the storage device 114 and loaded into the memory 116 when executed.

[0025] The memory 116 includes random access memory (RAM) modules, flash memory cells, or any other type of memory cells or combinations thereof. The processor 102, the I / O device interface 104, and the network interface 106 are configured to read data from and write data to the memory 116. The memory 116 includes various software programs executable by the processor 102 and application data associated with the software programs including the scene creation application 124.

[0026] In general, computing device 101 is configured to implement one or more software applications. For purposes of explanation only, each software application is described as residing in memory 116 of computing device 101 and executing on processor 102 of computing device 101. In some embodiments, any number of instances of any number of software applications may reside in memory 116 and any number of other memories associated with any number of other computing instances, and the any number of instances are executed in any combination on processor 102 of computing device 101 and any number of other processors associated with any number of other computing instances. In the same or other embodiments, the functionality of any number of software applications may be distributed across any number of other software applications residing in memory 116 and any number of other memories associated with any number of other computing instances, and the functionality is executed in any combination on processor 102 and any number of other processors associated with any number of other computing instances. In addition, subsets of the functionality of multiple software applications may be merged into a single software application.

[0027] In particular, the computing device 101 is configured to generate scenes based on user input provided by a user. As previously described herein, a conventional method for generating scenes involves a manual process with limited automation and limited support for the iterative creation process that creators typically prefer. In operation, embodiments of the present disclosure may automatically generate scenes of a fictional world based on input image blocks provided by a user or created using a generative machine learning model based on user input. Embodiments of the present disclosure may allow a user to define the spatial relationship between image blocks relative to each other and provide additional text prompts or scene text prompts. Using image blocks, the spatial relationship of image blocks relative to each other, and scene text prompts as input, a machine learning model that generates scenes may be executed.

[0028] Remote computing device 130 represents one or more computing devices that provide computing services that computing device 101 can access via network 120. Remote computing device 130 can be implemented as a computing device including, but not limited to, one or more processors and a network interface coupled to network 120. In the context of the present disclosure, the elements shown in remote computing device 130 may correspond to a physical computing system (e.g., a system in a data center) or may be a virtual computing instance executed within a computing cloud.

[0029] The remote computing device 130 may provide access to a generative machine learning model 132. The generative machine learning model 132 may include a machine learning model that can generate one or more images based on inputs (such as prompts) that may be provided to the model. The generative machine learning model 132 may generate images based on text prompts, image prompts, or a combination of text and images provided as inputs to the model. The generative machine learning model 132 may include a deep learning text-to-image model that can apply various techniques to generate images based on text inputs. For example, the generative machine learning model 132 may employ a diffusion model that utilizes a deep generative artificial neural network to generate images based on text input or based on images provided as input. For example, the generative machine learning model 132 may generate an image based solely on text or modify an image provided as input to the generative machine learning model 132. In addition, the generative machine learning model 132 may modify or transform an image provided as input based on a text prompt provided in combination with the input image. In some implementations, the generative machine learning model 132 may utilize a transformer architecture to generate an image based on a text prompt or to modify an image based on a text prompt provided with an input image.

[0030] For example, the generative machine learning model 132 may include a machine learning model that has been trained on a relatively large amount of existing text data and a relatively large amount of existing image data to generate new image data in response to a text prompt that is optionally associated with any number and / or type of image prompts. And because the generative machine learning model 132 generates new image data, the generative machine learning model 132 may be referred to as a generative prompt-to-image machine learning model.

[0031] In some embodiments, the generative machine learning model 132 includes a specialized version of the GPT-3 model known as the "DALL-E2" model. Techniques for implementing and using different multimodal ML models to generate new image data in response to text prompts that are optionally associated with image prompts are well known in the art. For example, see https: / / openai.com / dall-e-2 for information about implementing and using DALL-E2. In some other embodiments, the generative machine learning model 132 is trained to generate new image data in response to text prompts but does not process image prompts, and the techniques described herein are modified accordingly. Another example of a generative machine learning model 132 may include a stable diffusion machine learning model. For example, see https: / / github.com / Stability-AI / generative-models.

[0032] It should be appreciated that in some implementations, more than one type of generative machine learning model 132 may be utilized to generate one or more images based on input provided to a remote computing device 130. Additionally, more than one remote computing device 130 may be utilized to generate images based on text or image prompts provided as input. In the context of the present disclosure, as described in more detail herein, a scene creation application 124 may provide text or image-based input to a generative machine learning model 132, which may generate one or more output images that may be used to generate a scene on behalf of a user of the scene creation application 124. Additionally, in some implementations, the generative machine learning model 132 may be implemented on the same computing device 101 that executes the scene creation application 124. For example, the generative machine learning model 132 may be implemented as a module or functionality embedded within the scene creation application 124.

[0033] It should be understood that the system 100 shown herein is illustrative and that variations and modifications are possible. For example, the functionality provided by the scenario creation application 124 and the generative machine learning model 132 as described herein can be integrated into or across any number and / or type of software applications (including one) and any number of component distributions of the system 100. Furthermore, the system 100 may be modified as desired. Figure 1 The connection topology between the various units in.

[0034] Note that the techniques described herein are illustrative rather than restrictive and may be varied without departing from the broader spirit and scope of the embodiments. Many modifications and variations to the functionality of the generative machine learning model 132 and the scenario creation application 124 as described herein will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0035] Similarly, the storage, organization, amount and / or type of data described herein are illustrative rather than limiting and may be changed without departing from the broader spirit and scope of the embodiments. In this regard, many modifications to the data displayed within the various GUIs as described herein will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0036] Combination Figures 2 to 10 The scenario creation application 124 is described in more detail.

[0037] Generate image patches based on user input

[0038] To address the above issues, the system 100 includes, but is not limited to, a scene creation application 124. As described in more detail below, the scene creation application 124 interacts with a user via a graphical user interface (GUI) to obtain or generate one or more image tiles that can be used to define a scene composition. Based on the scene composition identified by the scene creation application 124, a generative machine learning model 132 can be used to create a scene. In other words, the scene creation application 124 can cause the generative machine learning model 132 to be executed to generate an output image representing a generated scene based on the input image tiles and text scene prompts provided as input to the generative machine learning model 132.

[0039] To generate a scene on behalf of a user, the scene creation application 124 obtains one or more image tiles from which the scene composition is determined. Figure 2 GUI 200 shows an example of how a user may generate or provide image patches as input to the generative machine learning model 132.

[0040] GUI 200 and any subsequent GUI depicted and described herein can be any type of user interface that allows a user to interact with one or more software applications via any number and / or type of GUI elements. GUI 200 can be displayed on any number and / or type of independent display devices, any number and / or type of display screens integrated into any number and / or type of user devices, or any combination thereof in any technically feasible manner. A user device can be any device that can display any number and / or type of media content on one or more associated display devices, one or more associated display screens, or any combination thereof. Some examples of user devices include desktop computers, laptop computers, smart phones, smart TVs, and tablet computers.

[0041] In the example shown in GUI 200, the scene creation application 124 may allow the user to enter a text mode for generating image tiles. In the text mode, the scene creation application 124 may prompt the user to provide a text prompt, and the generative machine learning model 132 may generate the image tiles according to the text prompt. The scene creation application 124 may generate the image tiles by providing the generative machine learning model 132 with the text prompt provided by the user.

[0042] Therefore, if Figure 2 As shown in , a user can provide an image block text hint 201 into a text input user interface element 203, and the scene creation application 124 can utilize the text hint as input to execute the generative machine learning model 132 to request one or more output images based on the text hint. The image block text hint 201 can describe the desired characteristics or features of the image block or text that describes the idea that the artist intends or desires to depict in the image block. Figure 2 In the example of , the image tile text hint 201 describes “an isometric view of a castle estate”, which is a text input that the scene creation application 124 can provide to the generative machine learning model 132. The scene creation application 124 can execute the generative machine learning model 132 or cause the generative machine learning model to execute to generate one or more output images that can be presented to the user as an image tile, and the scene creation application 124 can later create a scene based on the image tile.

[0043] continue Figure 2 Now referring to the example of GUI 300 Figure 3In the GUI 300, the scene creation application 124 has executed the generative machine learning model 132 to generate an image tile 305 based on the image tile text prompt 201 provided by the user. In some cases, the scene creation application 124 may request multiple output images, and the generative machine learning model 132 may generate multiple output images based on the image tile text prompt 201. The multiple images may be presented to the user via the GUI 300 as image tile candidates, from which the user may select to generate a particular image tile.

[0044] therefore, Figure 3 300 shows how the scene creation application 124 can execute the generative machine learning model 132 to generate an image tile 305 based on the text input user interface element 203 provided by the user in the GUI 200. In some implementations, the scene creation application 124 can allow the user to further edit the generated image tile 305 using image editing tools provided by the scene creation application 124. In other examples, the user can accept the generated image tile 305 as one of a plurality of image tiles that can be used as input to a process for generating a scene from the plurality of image tiles.

[0045] Based on Figures 2 to 3 The image block text prompt 201 shown in the figure is an example of a specific mode of image block creation, which is referred to as text mode in this article. In some cases, the scene creation application 124 can allow the user to start the text mode when the image block is previously created or the image block provided by the user. In this scene, the user can provide the image block text prompt 201 and the previously created image block. In this scene, the previously created image block and the image block text prompt 201 can be provided as input to the generative machine learning model 132 to create a new image block.

[0046] Examples of scene creation applications 124 may provide a multimodal system for generating image tiles and scenes. Figure 4 , which shows an example GUI 400. Figure 4 4, a region mode for creating image tiles that may be provided by the scene creation application 124 is shown.

[0047] exist Figure 4 In the example of , GUI 400 illustrates a zone mode mechanism for generating image tiles that may be provided by scene creation application 124. In the depicted GUI 400, scene creation application 124 may allow a user to input a zone mode for creating image tiles. In zone mode, a user may select one or more zones of a blank canvas. In an alternative example, a user may select one or more zones of a previously generated image tile or a user-provided image tile.

[0048] The scene creation application 124 may provide a user interface mechanism whereby a user may draw, sketch, or otherwise select an area of ​​a blank canvas or a previous image tile. Additionally, the scene creation application 124 may allow a user to provide a textual prompt associated with a selected area within an image tile. Figure 4 As shown in , the user can provide a text input user interface element 203 that can describe the artist's overall intent for the image block. Next, the scene creation application 124 can allow the artist to specify a region text hint 415 that specifies a text hint specific to the region represented by region 411 and region 413 in the image.

[0049] In one example user interaction, the scene creation application 124 may allow the user to outline or draw a zone (such as zone 411) in the image tile area 409, and then specify a zone text hint 415, where different portions of the zone text hint 415 are associated with specific zones 411 and 413. Figure 4 In the example shown in , the portion of the zone text prompt 415 associated with the zone 411 is underlined, and the portion of the zone text prompt 415 associated with the zone 413 is double underlined. In another example, a first zone outlined or identified by a user can be represented by a first color in the image block area 409, and a different zone outlined or identified by the user can be represented by a second color in the image block area 409. Correspondingly, the portion of the zone text prompt 415 corresponding to the first zone can be represented by the same color in the image block area 409. Similarly, the portion of the zone text prompt 415 corresponding to the second zone can be represented by the same color as the second zone in the image block area 409.

[0050] Back to Figure 4 400, once the user has outlined or identified the region with the image block area 409 and specified the region text prompt 415, the user can request the scene creation application 124 to generate the image block. Figure 5 , which shows a GUI 500 that can be provided by the scene creation application 124. The GUI 500 shows an example of an image block 510 that can be generated by the scene creation application 124. Figure 4 The image patch 510 is generated based on the region definition captured in the GUI 400, the user sketch in the image patch area 409, and the region text hint 415. Figure 4 The zone definitions captured in the GUI 400, the user sketch in the image block area 409, and the zone text prompt 415 can be provided to the generative machine learning model 132, which can generate one or more output images that can be presented to the user via the GUI 500.

[0051] Next reference Figure 6 , a GUI 600 is shown. The GUI 600 may be provided by the scene creation application 124 to provide an alternative block creation mode referred to herein as a sketch mode. Figure 6 In the example of , the scene creation application 124 may allow a user to provide a sketch, and the scene creation application 124 may execute the generative machine learning model 132 based on the sketch to generate one or more image patches that can be used to generate the scene later. Figure 6 As shown in , the user can enter a sketch into the image tile area 609. In addition, the GUI 600 can allow the user to provide an image tile text hint 601 using a text input user interface element 603. The scene creation application 124 can use the image tile text hint 601 to generate an image tile based on the sketch entered by the user into the image tile area 609.

[0052] Reference now Figure 7 , the figure continues Figure 6 and illustrates an example of how the scene creation application 124 may generate multiple image patch candidates from which a user may select. Figure 7 GUI 700 that may be generated by scene creation application 124 is shown. Scene creation application 124 may generate a plurality of image block candidates in any of the scene block creation modes, such as sketch mode, region mode, or text mode. The image block candidates may be presented in an image block selector within the GUI provided by the corresponding scene block creation mode, such as in Figure 7 In the example shown in .

[0053] exist Figure 7 In the example of , the user can select one of the image block candidates in the image block selector 713, which can cause the scene creation application 124 to display the image block 710 in the GUI 700. The image block candidate shown in the image block selector 713 can be generated by the scene creation application 124. The scene creation application 124 can provide Figure 6 The GUI 600 obtains the sketch and executes the generative machine learning model 132, thereby providing the image block text prompt 601 and the sketch as input. The generative machine learning model 132 can generate one or more output images based on the input, and the one or more output images can be presented as image block candidates in the image block selector 713.

[0054] It should be appreciated that the scene creation application 124 may present multiple image block candidates in the image block selector 713 in any of the image block creation modes. Additionally, the image block creation modes may be presented in a combined manner such that a user may enter text prompts, define areas, and enter sketches into a single user interface from which the scene creation application 124 may capture user input to be provided to the generative machine learning model 132 to generate image blocks. Additionally, in some scenarios, image blocks may be uploaded or provided by a user and may include previously generated images that are associated with an archive stored on the computing device 101 or accessible to the scene creation application 124. The archive may be associated with a user or a session of a user in which the user has generated or provided image blocks to be incorporated into a scene.

[0055] For scene descriptions without additional input, such as in the depicted text mode, the scene creation application 124 can generate an image from random noise based on the provided text input. If the user chooses to utilize the sketch mode or zone mode provided by the scene creation application 124 by inputting a sketch such as an RGB sketch, the scene creation application 124 can utilize the sketch along with Gaussian noise to generate an input image for the generative machine learning model 132 that matches the user sketch. The zones defined or specified by the user can be provided to the generative machine learning model 132 as an array, where each entry contains a binary mask image and a corresponding text input for the corresponding zone.

[0056] In sketch or zone mode, the scene creation application 124 can extract multiple binary masks from the user-provided zone segmentation, where pixels of a first color correspond to a unique zone color, and the remaining area is selected with a second color. A separate binary image mask can be generated, where the size matches the output image, and each word is in the input text prompt. Notably, the words in the user-provided text input exert variable effects on different parts of the image, where pixels of the first color serve as indicators of a higher probability for the element to appear in the assigned segment.

[0057] In some examples, before selecting image tiles to form a scene using the scene creation application 124, the user may modify or further refine the corresponding image tiles generated using the scene creation application 124. For example, if the image tiles generated by the scene creation application 124 using the generative machine learning model 132 do not fully meet the artist's expectations, the artist may select the image tiles and provide additional text prompts, user sketches, and / or region selections on the image tiles, which are user inputs that may be provided by the scene creation application 124 to the generative machine learning model 132. The generative machine learning model 132 may modify the image tiles based on the user input to generate new image tiles.

[0058] Additionally, the scene creation application 124 may maintain a history of previously selected and generated image tiles, such that a user may have access to a history of image tiles generated for a given session using the scene creation application 124 or a given user's history with the scene creation application 124. The user may select or modify historical image tiles to create new image tiles for a scene that the user wishes to generate using the scene creation application 124.

[0059] Generate scenes from image patches

[0060] Reference now Figure 8 , shows an example GUI 800 that may be generated by the scene creation application 124. The GUI 800 may be referred to herein as a global image patch view. Figure 8 In the example GUI 800 shown in , multiple image tiles are shown. The scene creation application 124 may require the user to select or generate multiple image tiles, and the scene creation application 124 may generate a scene on behalf of the user based on the multiple image tiles. In the depicted example, the scene creation application 124 may allow the user to select up to four image tiles, and the scene may be generated based on the four image tiles. However, in other implementations, the scene creation application 124 may allow or require the user to select or generate more or fewer image tiles, and the scene may be generated based on these image tiles.

[0061] The image tiles 801, 803, 805, and 807 may be provided by a user or generated by the scene creation application 124 based on text, region, or sketch prompts provided by the user, and then sequentially provided to the generative machine learning model 132 to generate the image tiles. In some examples, the user may upload or select one or more of the image tiles from a pre-existing image tile library. The pre-existing image tile library may be stored on the computing device 101 or obtained from a remote source, such as an image library accessible via the network 120.

[0062] The scene creation application 124 can determine the spatial relationship of the image tiles selected by the user for a given scene relative to each other and generate a scene based on the image tiles and the spatial relationship. In some examples, such as the example depicted in the GUI 800, the user can arrange the image tiles 801, 803, 805, and 807 in a grid format, where the image tiles are equally spaced. The user can click on each image tile in the GUI 800 and drag the image tile to a desired position to define the spatial relationship of the image tiles 801, 803, 805, and 807 relative to each other. The spatial relationship of the image tiles 801, 803, 805, and 807 relative to each other also defines the scene composition of the resulting scene generated by the scene creation application 124.

[0063] The scene creation application 124 may also allow the user to provide a scene text hint 813 that describes the desired features or characteristics of the scene that the user wishes to create using the scene creation application 124. In the depicted example, the scene text hint 813 may be provided to the scene creation application 124 via the GUI 800. Upon providing the scene text hint 813 and the image tiles 801, 803, 805, and 807, the user may cause the scene creation application 124 to generate a scene based on the image tiles 801, 803, 805, and 807 and the scene text hint 813 by activating the blending tile user interface element 815.

[0064] When the hybrid block user interface element 815 is activated, the scene creation application 124 can provide the scene text prompt 813, the image blocks 801, 803, 805 and 807, and the spatial relationship of the image blocks relative to each other as input to the generative machine learning model 132. In other words, the scene creation application 124 can execute the generative machine learning model 132 or cause the generative machine learning model 132 to execute to generate an output image using the image blocks 801, 803, 805 and 807, the scene text prompt 813, and the spatial relationship of the image blocks relative to each other as input. For example, the spatial relationship of the image blocks relative to each other can be expressed as a bitmap that identifies the corresponding image blocks and the coordinate positions of the image blocks on a Cartesian grid. The generative machine learning model 132 can generate an output image that can be provided to the scene creation application 124, which can present the output image as a scene generated by the scene creation application 124 on behalf of the user.

[0065] The scene creation application 124 can generate an output image corresponding to the scene by filling the empty spaces between the image blocks 801, 803, 805, and 807 and blending the blocks together. To blend the image blocks, the scene creation application 124 can, for example, obtain a binary mask with black pixels for the image blocks and white pixels for the empty spaces. A Gaussian blur can be applied to the mask to soften the edges of the black blocks to achieve a smooth blend. The final image corresponding to the scene can be generated by inputting the mask and the user-created image blocks 801, 803, 805, and 807 as input to the generative machine learning model 132.

[0066] Next reference Fig. 9 , shows an alternative example of a GUI 900 that may be generated by the scenario creation application 124. Fig. 9 In the example GUI 900 of , the user has rearranged the image tiles 801, 803, 805, and 807 within the GUI 900 to change the spatial orientation of the image tiles relative to each other. By rearranging the image tiles, the scene composition of the resulting scene has been changed.

[0067] Similarly, a user can provide a scene text prompt 813 and request the scene creation application 124 to generate a scene based on the image blocks 801, 803, 805 and 807 and the scene composition by activating the hybrid block user interface element 815, which can cause the scene creation application 124 to execute the generative machine learning model 132 to generate an output image as a scene.

[0068] continue Fig. 9 For example, now refer to Fig.10 . Fig.10 GUI 1050 is shown showing an example scene that can be generated by scene creation application 124 based on image tiles 801, 803, 805, and 807, scene composition, and scene text hint 813. Scene 1051 can be generated by blending image tiles 801, 803, 805, and 807 and filling in the empty spaces between the image tiles using scene text hint 813. The spatial positioning of image tiles 801, 803, 805, and 807 can remain largely unchanged, while generative machine learning model 132 can fill in the empty spaces with image content and blend image tiles 801, 803, 805, and 807 with image content generated to fill in the empty spaces.

[0069] As will be appreciated by those skilled in the art, the techniques described herein are illustrative rather than limiting, and these techniques may be modified and applied in other contexts without departing from the broader spirit and scope of the inventive concepts described herein. For example, the techniques described herein may be modified and applied to generate any number of scenes from any number of input image blocks generated by the scene creation application 124 or provided by a user.

[0070] Fig.11 A flowchart of method steps for generating a scene based on user input according to various embodiments is set forth. Figure 1 Although the method steps are described with reference to a system, those skilled in the art will understand that any system configured to implement the method steps in any order falls within the scope of the embodiments.

[0071] As shown, method 1100 begins at step 1102, where the scene creation application 124 can obtain an input image block based on a user input. By utilizing the generative machine learning model 132, the input image block can be used to generate a scene on behalf of the user. The corresponding input image block can be uploaded to the user or provided by the user. With the assistance of the generative machine learning model 132, the scene creation application 124 can also be used to generate the corresponding image block.

[0072] The scene creation application 124 may obtain a text prompt, a user sketch, a selection of a blank canvas or a user sketch, or a combination thereof. These user inputs may be provided to a generative machine learning model 132, which may be executed by the scene creation application 124 to generate an output image that may be presented to the user as an image tile. The user may generate multiple image tiles, which may be used by the scene creation application 124 to generate a scene on behalf of the user.

[0073] At step 1104, the scene creation application 124 may determine the spatial relationship of the image tiles relative to each other. The scene creation application 124 may allow the user to resize, arrange, or move the image tiles identified at step 1102. The spatial relationship of the image tiles relative to each other may be identified, such as by mapping the coordinates of the image tiles relative to each other on a Cartesian map. For example, a bitmap may be generated that specifies the coordinates of the respective image tiles relative to each other.

[0074] At step 1106, a scene composition may be determined based on the spatial relationship of the image tiles relative to each other. The scene composition may include the spatial relationship of the image tiles relative to each other and the empty spaces between the corresponding image tiles. The scene creation application 124 may utilize the empty spaces to blend the image tiles together to generate the scene.

[0075] At step 1108, the scene creation application 124 may obtain a scene text hint associated with the resulting output scene. The scene text hint may be a descriptive natural language hint provided by a user that may specify desired features of the scene.

[0076] At step 1110, the scene creation application 124 may execute the generative machine learning model 132, providing the input image patches, scene composition, and scene textual cues as inputs to the generative machine learning model 132. The scene creation application 124 may request the generative machine learning model 132 to generate one or more output images based on the provided inputs. The one or more output images may be candidates for a scene generated based on the provided inputs.

[0077] The scene creation application 124 can generate an output image corresponding to the scene by filling the empty spaces between the image blocks and blending the blocks together. To blend the image blocks, the scene creation application 124 can, for example, obtain a binary mask with black pixels for the image blocks and white pixels for the empty spaces. A Gaussian blur can be applied to the mask to soften the black block edges to achieve a smooth blend. A final image corresponding to the scene can be generated by inputting the mask and the image blocks created by the user as input to the generative machine learning model 132.

[0078] At step 1112 , the scene creation application 124 may obtain one or more output images from the generative machine learning model 132 as a scene generated by the scene creation application 124 .

[0079] Then, method 1100 terminates.

[0080] In summary, techniques are disclosed for generating scenes using computer-aided design tools, such as scenes associated with fictional worlds created by artists or authors. Input image blocks may be provided by a user or generated by a design tool based on a textual prompt provided by a user, a sketch provided by a user, or a combination thereof. A GUI may then be used to arrange the image blocks so that spatial relationships between the image blocks may be identified. The spatial relationships of the image blocks correspond to a scene composition. Based on the scene composition, the input image blocks, and the scene textual prompt, an output image may be generated by a generative machine learning model. The output image corresponds to a scene.

[0081] At least one technical advantage of the disclosed technology over the prior art is that the disclosed technology enables users with limited time or limited knowledge of computer-aided design tools or generation tools to more effectively and efficiently generate scenes for fictional worlds. In this regard, the disclosed technology provides an automated process for obtaining multiple image patches that can be used as a basis for generating scenes for fictional worlds. The image patches can be provided by a user or created using a machine learning model that is provided with user input such as text prompts, sketches, zone maps based on user-selected areas of the image patches, or any combination thereof. The machine learning model can generate one or more image patches based on user input from the spatial relationships of the image patches relative to each other and text scene prompts obtained from the user.

[0082] Any and all combinations of any of the claim elements of any of the claims and / or any elements described in this application in any manner fall within the contemplated scope of the invention and protection.

[0083] Descriptions of the various embodiments have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0084] Aspects of embodiments of the present invention may be embodied as systems, methods, or computer program products. Thus, aspects of the present disclosure may take the form of a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.), or an implementation combining software aspects with hardware aspects, which may all be generally referred to herein as "modules" or "systems." In addition, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied thereon.

[0085] Any combination of one or more computer readable media may be utilized. A computer readable medium may be a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media would include the following: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory, a flash memory, an optical fiber, a portable optical disc read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus.

[0086] Aspects of the present disclosure are described above with reference to the flowchart illustration and / or block diagram of the method, device (system) and computer program product according to the embodiments of the present disclosure. It should be understood that each frame in the flowchart illustration and / or block diagram and the combination of frames in the flowchart illustration and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine. The instructions enable the function / action specified in one or more frames of the flowchart and / or block diagram to be realized when executed via the processor of a computer or other programmable data processing device. Such processors can be, but are not limited to, general-purpose processors, special-purpose processors, special-purpose processors or field programmable gate arrays.

[0087] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a fragment or a portion of a code, and the code includes one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the box may not occur in the order indicated in the accompanying drawings. For example, the two boxes shown in succession may actually be executed substantially simultaneously, or the boxes may sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each box in the block diagram and / or flowchart illustration and the combination of boxes in the block diagram and / or flowchart illustration can be implemented by a system based on special-purpose hardware or a combination of special-purpose hardware and computer instructions that performs the specified function or action.

[0088] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope of the disclosure, and the scope of the disclosure is determined by the claims that follow.

Claims

1. A computer-implemented method for generating a scene, the method comprising: obtaining a plurality of input image blocks based on at least one user input; detecting a spatial location of each input image block included in the plurality of input image blocks relative to at least one other input image block included in the plurality of input image blocks; determining a scene composition of the scene based on the spatial positioning of each input image block included in the plurality of image blocks; Obtaining a scene prompt associated with the scene; as well as The machine learning model is caused to mix the multiple image blocks based on the scene prompt to generate the scene. 2 . The computer-implemented method of claim 1 , wherein the scene hint comprises a text hint indicating how the plurality of image tiles should be blended to generate the scene.

3. The computer-implemented method of claim 1 , wherein obtaining the plurality of input image patches based on the at least one user input comprises: obtaining an image block textual cue describing a desired feature of a corresponding image block included in the plurality of image blocks; Providing the image block text prompt to the machine learning model; as well as The machine learning model is executed to generate the corresponding image patch based on the text prompt.

4. The computer-implemented method of claim 1 , wherein obtaining the plurality of input image patches based on the at least one user input comprises: obtaining a sketch of a portion of the scene within a graphical user interface; providing the sketch to the machine learning model; as well as The machine learning model is executed to generate corresponding image patches based on the sketch.

5. The computer-implemented method of claim 1 , further comprising: obtaining a sketch of a portion of the scene within a graphical user interface; obtaining an image patch text hint associated with the sketch of the portion of the scene, wherein the image patch text hint specifies a desired feature of a corresponding image patch; as well as The machine learning model is executed to generate the corresponding image patch based on the sketch and the image patch text hint.

6. The computer-implemented method of claim 1 , wherein obtaining the plurality of input image patches based on the at least one user input comprises: obtaining, within the graphical user interface, a selection of a region corresponding to the image block; obtaining a region text cue corresponding to the region, wherein the region text cue specifies desired characteristics of the region within the corresponding image block; as well as The machine learning model is executed to generate the corresponding image block based on the region and the region text hint.

7. The computer-implemented method of claim 1 , wherein obtaining the plurality of input image patches based on at least one user input comprises: obtaining a selection of a previously generated image block included in a plurality of previously generated image blocks; generating new image patches based on user input; obtaining a textual cue specifying desired features for the combined image patch; as well as The machine learning model is executed to generate the combined image patch based on the desired feature, the previously generated image patch, and the new image patch.

8. The computer-implemented method of claim 1, further comprising: A spatial positioning of each input image block included in the multiple input image blocks relative to at least one other input image block included in the multiple input image blocks is obtained within a graphical user interface, wherein the spatial positioning includes a blank space between at least a first input image block included in the multiple image blocks and a second input image block included in the multiple input image blocks.

9. The computer-implemented method of claim 8, further comprising automatically adjusting an amount of blank space between each input image block included in the plurality of input image blocks relative to at least one other input image block included in the plurality of input image blocks.

10. The computer-implemented method of claim 8, wherein executing a first machine learning model to blend the plurality of image patches to form the scene based on the scene cue comprises filling the blank space between at least one of the plurality of image patches based on the scene cue.

11. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to generate a scenario by performing the following steps: obtaining a plurality of input image blocks based on at least one user input; detecting a spatial location of each input image block included in the plurality of input image blocks relative to at least one other input image block included in the plurality of input image blocks; determining a scene composition of the scene based on the spatial positioning of each input image block included in the plurality of image blocks; Obtaining a scene prompt associated with the scene; as well as The machine learning model is caused to mix the multiple image blocks based on the scene prompt to generate the scene.

12. The one or more non-transitory computer-readable media of claim 11, wherein the scene hint comprises a text hint indicating how the plurality of image tiles should be blended to generate the scene.

13. The one or more non-transitory computer-readable media of claim 11, wherein obtaining the plurality of input image blocks based on the at least one user input comprises: obtaining an image block textual cue describing a desired feature of a corresponding image block included in the plurality of image blocks; Providing the image block text prompt to the machine learning model; as well as The machine learning model is executed to generate the corresponding image patch based on the text prompt.

14. The one or more non-transitory computer-readable media of claim 11, wherein the method comprises: obtaining a sketch of a portion of the scene within a graphical user interface; providing the sketch to the machine learning model; as well as The machine learning model is executed to generate corresponding image patches based on the sketch.

15. The one or more non-transitory computer-readable media of claim 14, further comprising: obtaining a sketch of a portion of the scene within a graphical user interface; obtaining an image patch text hint associated with the sketch of the portion of the scene, wherein the image patch text hint specifies a desired feature of a corresponding image patch; as well as The machine learning model is executed to generate the corresponding image patch based on the sketch and the image patch text hint.

16. The one or more non-transitory computer-readable media of claim 11, wherein obtaining the plurality of input image blocks based on the at least one user input comprises: obtaining, within the graphical user interface, a selection of a region corresponding to the image block; obtaining a region text cue corresponding to the region, wherein the region text cue specifies desired characteristics of the region within the corresponding image block; as well as The machine learning model is executed to generate the corresponding image block based on the region and the region text hint.

17. The one or more non-transitory computer-readable media of claim 11, wherein obtaining the plurality of input image blocks based on at least one user input comprises: obtaining a selection of a previously generated image block included in a plurality of previously generated image blocks; generating new image patches based on user input; obtaining a textual cue specifying desired features for the combined image patch; as well as The machine learning model is executed to generate the combined image patch based on the desired feature, the previously generated image patch, and the new image patch.

18. The one or more non-transitory computer-readable media of claim 11, wherein the method comprises: A spatial positioning of each input image block included in the multiple input image blocks relative to at least one other input image block included in the multiple input image blocks is obtained within a graphical user interface, wherein the spatial positioning includes a blank space between at least a first input image block included in the multiple image blocks and a second input image block included in the multiple input image blocks.

19. One or more non-transitory computer-readable media as described in claim 18, wherein causing the machine learning model to blend the multiple image blocks to form the scene based on the scene prompt includes filling the blank space between at least one image block of the multiple image blocks based on the scene prompt.

20. A system comprising: one or more memories storing instructions; as well as One or more processors, the one or more processors being coupled to the one or more memories, the one or more processors performing the following steps when executed: obtaining a plurality of input image blocks based on at least one user input; detecting a spatial location of each input image block included in the plurality of input image blocks relative to at least one other input image block included in the plurality of input image blocks; determining a scene composition of a scene based on the spatial positioning of each input image block included in the plurality of image blocks; Obtaining a scene prompt associated with the scene; as well as The machine learning model is caused to mix the multiple image blocks based on the scene prompt to generate the scene.

Images