Redesigning images using diffusion models using textual conditions and depth maps

Abstract

A media application receives an initial image, a user input selecting one or more objects in the initial image, and a text request to generate an output image modifying the one or more selected objects in the initial image. The media application generates a user-selected mask including object pixels corresponding to the one or more selected objects. A diffusion model receives a text request to generate an output image, a depth map, and a user-selected mask, where the diffusion model is trained to generate output pixels that are not associated with a human subject. And the diffusion model outputs an output image meeting the text request.

Description

[0001] Cross-references to related applications

[0002] This application is a non-provisional application filed July 2, 2024, entitled “Generating Images with Uncrop and Recomposition,” pursuant to 35 USC § 119(e), the entire contents of which are incorporated herein by reference. Background Technology

[0003] Generative artificial intelligence (AI) can be used to generate images based on text prompts. It can also be used to create modified versions of pre-existing images based on text prompts. The results generated by AI can be problematic in some contexts, especially when the image includes people, as more detailed aspects may not be accurately represented. For example, generative AI remains imperfect when it comes to the complexity of capturing features such as fingers, eyes, and mouth in the generated image. Additionally, the generated images may lack realism.

[0004] The background description provided herein is for the purpose of presenting the overall context of this disclosure. The work of the currently attributed inventors (to the extent described in this background section) and aspects of the specification that may not be considered prior art at the time of filing are neither expressly nor impliedly acknowledged as prior art to this disclosure. Summary of the Invention

[0005] A computer-implemented method for generating an image based on a text request includes receiving: an initial image, user input selecting one or more objects in the initial image, and a text request for generating an output image that modifies the one or more selected objects in the initial image. The method includes generating a user-selected mask comprising object pixels corresponding to the one or more selected objects. The method further includes providing the text request for generating the output image, a depth map, and the user-selected mask as input to a diffusion model, wherein the diffusion model is trained to generate output pixels for the output image that are not associated with human subjects. The method further includes using the diffusion model to generate an output image that satisfies the text request.

[0006] In some embodiments, a depth map identifies the depth of image pixels in an initial image, and the output image retains the depth map of the initial image. In some embodiments, a classifier-free guide is used to control depth, and a higher conditional discard value preserves more of the structure of one or more selected objects in the initial image than a lower conditional discard value. In some embodiments, user input is provided from a user who performs one or more actions selected from the group consisting of: moving around one or more objects in the initial image, moving a finger over one or more objects in the image, tapping one or more objects in the initial image, providing text recognition for one or more objects, and combinations thereof. In some embodiments, the method further includes performing object recognition to identify one or more humans in the initial image, wherein the input to the diffusion model further includes one or more retention masks that identify human pixels corresponding to one or more humans in the initial image, the one or more retention masks being used by the diffusion model to prevent modification of human pixels. In some embodiments, the method further includes, in response to receiving user input, performing object recognition to identify one or more types of one or more selected objects, and providing one or more suggestions for modifying one or more selected objects based on the types of one or more objects. In some embodiments, the method further includes segmenting one or more selected objects in the initial image, and generating a segmentation mask that identifies one or more selected objects, wherein the input to the diffusion model further includes the segmentation mask.

[0007] A method for training a diffusion model includes generating training data comprising initial images of one or more selected objects and conditions, wherein for each initial image, the conditions include a text request, a depth map, and a user-selected mask. The method further includes training the diffusion model to output images that satisfy the conditions and do not include human pixels, wherein training comprises repeatedly generating output images until a comparison of the output image with a corresponding ground truth image satisfies a threshold loss value.

[0008] In some embodiments, the method further includes segmenting one or more selected objects in an initial image, and generating a segmentation mask that identifies the one or more selected objects, wherein the condition further includes the segmentation mask. In some embodiments, the depth map includes depth values ​​that identify the depth of image pixels in the initial image, and training the diffusion model includes training the output image to preserve the depth map associated with the initial image. In some embodiments, the method further includes training the diffusion model based on varying amounts of text requests and depth values ​​by running a first version of the diffusion model without text requests and depth values, running a second version of the diffusion model with text requests and no depth values, and running a third version of the diffusion model with text requests and depth values. In some embodiments, the condition further includes unclassifier guidance, and the unclassifier guidance amount is based on a higher conditional dropout value that preserves more of the structure of the one or more selected objects in the initial image than a lower conditional dropout value. In some embodiments, the condition further includes a preservation mask that identifies human pixels corresponding to one or more human subjects in the initial image, the preservation mask being used by the diffusion model to prevent modification of the human pixels during the generation of the output image. In some embodiments, the training data further includes pairs of ground truth images with corresponding images of randomly masked portions of the ground truth images. In some embodiments, the diffusion model is trained for use in the method for generating an image according to the first method claim.

[0009] A non-transitory computer-readable medium includes instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform operations. The operations include: receiving an initial image, user input selecting one or more objects in the initial image, and a text request for generating an output image modifying the one or more selected objects in the initial image; generating a mask comprising user-selected object pixels corresponding to the one or more selected objects; providing the text request, a depth map, and the user-selected mask as input to a diffusion model for generating the output image, wherein the diffusion model is trained to generate output pixels for the output image that are not associated with a human subject; and using the diffusion model to generate an output image that satisfies the text request.

[0010] In some embodiments, a depth map identifies the depth of image pixels in an initial image, and the output image retains the depth map of the initial image. In some embodiments, a classifier-free guide is used to control depth, and a higher conditional discard value preserves more of the structure of one or more selected objects in the initial image than a lower conditional discard value. In some embodiments, user input is provided from a user who performs one or more actions selected from the group consisting of: moving a finger around one or more objects in the initial image, moving a finger over one or more objects in the image, tapping one or more objects in the initial image, providing text recognition for one or more objects, and combinations thereof. In some embodiments, operation further includes performing object recognition to identify one or more humans in the initial image, wherein the input to the diffusion model further includes one or more retention masks that identify human pixels corresponding to one or more humans in the initial image, the one or more retention masks being used by the diffusion model to prevent modification of the human pixels. In some embodiments, operation further includes, in response to receiving user input, performing object recognition to identify one or more types of one or more selected objects, and providing one or more suggestions for modifying one or more selected objects based on the types of one or more objects. Attached Figure Description

[0011] Figure 1 This is a block diagram of an example network environment based on some embodiments described herein.

[0012] Figure 2 This is a block diagram of an example computing device according to some embodiments described herein.

[0013] Figure 3A Example user interfaces, including initial images, are illustrated according to some embodiments described herein.

[0014] Figure 3B Examples of embodiments described herein include text requests and requests with user input from... Figure 3A Example user interface with the initial image.

[0015] Figure 3C Examples of embodiments according to the description herein include satisfying the following: Figure 3B The example user interface for the output image of the text request provided in the document.

[0016] Figure 4 An example process is illustrated for training a diffusion model according to some embodiments described herein to generate an output image based on a text request and an initial image.

[0017] Figure 5 The architecture of an example diffusion model according to some embodiments described herein is illustrated.

[0018] Figure 6 This is a flowchart of an example method for training a diffusion model to generate an output image based on a text request, according to some embodiments described herein.

[0019] Figure 7 This is a flowchart of an example method for generating an output image based on a text request, according to some embodiments described herein. Detailed Implementation

[0020] Overview

[0021] Generative artificial intelligence (AI) models are used to generate images based on text prompts. Text prompts are user-input text that represents instructions / requests to the AI ​​model in text form for performing an action. In this disclosure, the action is the generation or modification of an image. However, existing generative AI techniques have various limitations, particularly when generating or modifying images that include human subjects. Current generative AI models frequently encounter difficulties in accurately representing the complex details of human features such as fingers, eyes, and mouths, often resulting in inaccurate or unrealistic depictions in the generated images. This problem becomes even more pronounced when a user attempts to modify specific aspects of an initial image depicting a human subject, causing unintended alterations or artifacts in the human elements.

[0022] Previous solutions for image modification using generative AI lacked mechanisms to consistently preserve the image's underlying structural characteristics, such as depth information of existing objects within the image, during modification operations. This can produce outputs that deviate significantly from the spatial composition of the original image, thus undermining the desired results of targeted modification.

[0023] Furthermore, current training methods for generative models are insufficient to address the specific constraints required for controlled image modifications, particularly those involving human subjects.

[0024] The technique described herein addresses the aforementioned problems by utilizing an initial image and conditionally trained diffusion models. Referring to the initial image, the conditions include a text request from a user to generate an output image modifying one or more selected objects in the initial image, a depth map, and a user-selected mask, wherein the user-selected mask includes object pixels corresponding to the one or more selected objects. In some embodiments, the conditions may also include a segmentation mask that identifies the one or more selected objects. This can be used as a fallback to ensure that the one or more objects selected by the user are accurately identified. The diffusion model is also trained to generate output pixels associated with non-human subjects. For example, the conditions may also include a reserved mask that identifies human pixels corresponding to one or more humans in the initial image.

[0025] The diffusion model described in this paper advantageously improves the quality of output images that include human subjects by using a unique combination of conditions. For example, using a depth map preserves the depth from the initial image; combining a text request with a user-selected mask ensures that the output image corresponds to the user's specifications regarding the attributes and content of the output image; and the absence of classifier guidance improves the overall quality of the output image. Training the diffusion model to generate output images that do not include human pixels improves the quality of the output image by reducing or eliminating illusions in the model's output.

[0026] Network environment

[0027] Figure 1 A block diagram illustrating an example network environment 100 is shown. In some embodiments, environment 100 includes a media server 101, user device 115a, and user device 115n coupled to network 105. Users 125a and 125n may be associated with corresponding user devices 115a and 115n. In some embodiments, environment 100 may include... Figure 1 Other servers or devices not shown in the diagram. Figure 1 In the other figures, the letter following the reference numeral (e.g., "115a") indicates a reference to an element having that particular reference numeral. Reference numerals in the text without a following letter (e.g., "115") indicate a general reference to an embodiment of the element having that reference numeral.

[0028] Media server 101 may include a processor, memory, and network communication hardware. In some embodiments, media server 101 is a hardware server. Media server 101 is communicatively coupled to network 105 via signal line 102. Signal line 102 may be a wired connection (such as Ethernet, coaxial cable, fiber optic cable, etc.) or a wireless connection (such as Wi-Fi®, Bluetooth®, or other wireless technologies). In some embodiments, media server 101 sends data to and receives data from one or more user devices 115a, 115n via network 105. Media server 101 may include media application 103a and database 199.

[0029] Database 199 can store machine learning models, training datasets, images, etc. Database 199 can also store social network data associated with user 125, user preferences of user 125, etc.

[0030] User device 115 may be a computing device including memory coupled to a hardware processor. For example, user device 115 may include a mobile device, tablet computer, mobile phone, wearable device, head-mounted display, mobile email device, portable game player, portable music player, e-reader device, or another electronic device capable of accessing network 105.

[0031] In the illustrated implementation, user device 115a is coupled to network 105 via signal line 108, and user device 115n is coupled to network 105 via signal line 110. Media application 103 may be stored on user device 115a as media application 103b and / or on user device 115n as media application 103c. Signal lines 108 and 110 may be wired connections (such as Ethernet, coaxial cable, fiber optic cable, etc.) or wireless connections (such as Wi-Fi®, Bluetooth®, or other wireless technologies). User devices 115a and 115n are accessed by users 125a and 125n, respectively. Used by way of example. Figure 1 User units 115a and 115n are included. Although Figure 1 Two user units 115a and 115n are illustrated, but this disclosure applies to system architectures having one or more user units 115.

[0032] Media application 103 may be stored on media server 101 or user device 115. In some embodiments, the operations described herein are performed on media server 101 or user device 115. For example, media application 103b on user device 115a may receive an initial image captured by user device 115a and generate an output image. In some embodiments, some operations may be performed on media server 101 and some operations may be performed on user device 115. For example, an initial image may be captured by user device 115a and transmitted to media application 103a on media server 101 along with user input and text requests, which generates an output image that is transmitted to media application 103b on user device 115a for display.

[0033] The execution of operations is based on user settings. For example, user 125a may specify a setting to perform operations on their respective device 115a instead of on media server 101. With such a setting, the operations described herein are performed entirely on user device 115a, and no operations are performed on media server 101. Furthermore, user 125a may specify that the user's images and / or other data will be stored locally only on user device 115a and not on media server 101. With such a setting, no user data is transferred to or stored on media server 101. The transmission of user data to media server 101, any temporary or permanent storage of such data by media server 101, and the execution of operations on such data by media server 101 are only performed if the user has consented to the transmission, storage, and operation by media server 101. Users are provided with the option to change settings at any time, such as enabling or disabling the use of media server 101.

[0034] If a machine learning model (e.g., a diffusion model or other type of model) is used for one or more operations, the machine learning model is stored and utilized locally on user device 115, subject to specific user permissions. Server-side models are used only with user permission. Furthermore, a trained model can be made available for use on user device 115. During such use, on-device training of the model can be performed if user 125 permits. Updated model parameters can be transferred to media server 101, for example, to implement federated learning, if user 125 permits. The model parameters do not include any user data.

[0035] Media application 103 receives an initial image, user input selecting one or more objects in the initial image, and a text request to generate an output image that modifies the one or more selected objects in the initial image. For example, the user may circle an object in the initial image and provide a text request to change that object to a different object, add features to the object, etc. Media application 103 generates a mask that includes the user's selection of object pixels corresponding to the one or more selected objects.

[0036] Media application 103 includes a diffusion model that receives a text request, a depth map, and a user-selected mask to generate an output image with modifications to an initial image. The diffusion model is trained to generate output pixels that are not associated with human subjects. The diffusion model may also receive a preserving mask that identifies human pixels corresponding to one or more humans in the input image. The diffusion model generates an output image that satisfies the text request.

[0037] In some embodiments, media application 103 may be implemented using hardware including a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a machine learning processor / coprocessor, any other type of processor, or a combination thereof. In some embodiments, media application 103a may be implemented using a combination of hardware and software.

[0038] Computing device

[0039] Figure 2 This is a block diagram of an example computing device 200 that can be used to implement one or more features described herein. The computing device 200 can be any suitable computer system, server, or other electronic or hardware device. In one example, the computing device 200 is a media server 101 for implementing media application 103a. In another example, the computing device 200 is a user device 115.

[0040] In some embodiments, the computing device 200 includes a processor 235, a memory 237, an input / output (I / O) interface 239, a display 241, a camera 243, and a storage device 245, all coupled via a bus 218. The processor 235 may be coupled to the bus 218 via signal line 222, the memory 237 may be coupled to the bus 218 via signal line 224, the I / O interface 239 may be coupled to the bus 218 via signal line 226, the display 241 may be coupled to the bus 218 via signal line 228, the camera 243 may be coupled to the bus 218 via signal line 230, and the storage device 245 may be coupled to the bus 218 via signal line 232.

[0041] Processor 235 may be one or more processors and / or processing circuits for executing program code and controlling the basic operations of computing device 200. "Processor" includes any suitable hardware system, mechanism, or component that processes data, signals, or other information. A processor may include systems having: a general-purpose central processing unit (CPU) with one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., in a multiprocessor configuration), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a dedicated circuit system for implementing functionality, a dedicated processor for implementing processing based on neural network models, neural circuits, a processor optimized for matrix computations (e.g., matrix multiplication), or other systems. In some embodiments, processor 235 may include one or more coprocessors implementing neural network processing. In some embodiments, processor 235 may be a processor that processes data to produce probabilistic outputs; for example, the output produced by processor 235 may be inaccurate or may be accurate within a range of the expected output. Processing is not necessarily limited to a specific geographical location or has time constraints. For example, a processor may perform its functions in real-time, offline, in batch processing mode, etc. The different parts of the process can be executed at different times and in different locations by different (or the same) processing systems. The computer can be any processor that communicates with memory.

[0042] Memory 237 is typically provided in computing device 200 for access by processor 235 and can be any suitable processor-readable storage medium suitable for storing instructions executable by a processor or set of processors and located separately from and / or integrated with processor 235, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc. Memory 237 may store software operated on computing device 200 by processor 235, including media application 103.

[0043] The memory 237 may include an operating system 262, other applications 264, and application data 266. Other applications 264 may include, for example, image library applications, image management applications, image gallery applications, communication applications, web hosting engines or applications, media sharing applications, etc. One or more methods disclosed herein can operate in various environments and platforms, for example, as a standalone computer program that can run on any type of computing device, as a web application with web pages, as a mobile application (“app”) running on a mobile computing device, etc.

[0044] Application data 266 may be data generated by other applications 264 or hardware of computing device 200. For example, application data 266 may include images used by an image gallery application and user actions identified by other applications 264 (e.g., a social networking application).

[0045] I / O interface 239 provides functionality to enable computing device 200 to interface with other systems and devices. Devices connected via the interface may be included as part of computing device 200, or may be separate and communicate with computing device 200. For example, network communication devices, storage devices (e.g., memory 237 and / or storage device 245), and input / output devices may communicate via I / O interface 239. In some embodiments, I / O interface 239 may be connected to interface devices such as input devices (keyboard, pointing device, touchscreen, microphone, scanner, sensor, etc.) and / or output devices (display device, speaker device, printer, monitor, etc.).

[0046] Some examples of docked devices that can be connected to I / O interface 239 may include display 241, which can be used to display content (e.g., images, videos, and / or user interfaces for output applications as described herein) and receive touch (or gesture) input from a user. For example, display 241 may be used to display a user interface including graphical guidance on a viewfinder. Display 241 may include any suitable display device, such as a liquid crystal display (LCD), a light-emitting diode (LED) or plasma display, a cathode ray tube (CRT), a television, a monitor, a touch screen, a 3D display, or other visual display device. For example, display 241 may be a flat panel display provided on a mobile device, multiple displays embedded in a shape-factor eyeglass or head-mounted device, or a monitor screen for a computer device.

[0047] Camera 243 can be any type of image capture device capable of capturing images and / or video. In some embodiments, camera 243 captures images or video, and I / O interface 239 transmits the images or video to media application 103.

[0048] Storage device 245 stores data related to media application 103. For example, storage device 245 may store a training dataset, which includes labeled images, machine learning models, outputs from machine learning models, etc.

[0049] Figure 2 An example media application 103 stored in memory 237 is illustrated. The example media application includes a user interface module 202, a splitter 204, and a diffusion module 206.

[0050] User interface module 202 generates graphical data for displaying a user interface including images. User interface module 202 receives an initial image. The initial image can be received from the camera 243 of computing device 200 or from media server 101 via I / O interface 239. The initial image can also be provided by the user, for example, via an upload enabled by user interface module 202.

[0051] Before the initial image is processed, the user interface presents a request to the user for their consent to modify the image. In some embodiments, media application 103 may obtain such consent all at once for all future images. The user is provided with options to withdraw such one-time consent and to request consent for each image individually. User interface module 202 does not collect or utilize user information unless the user provides consent.

[0052] Users may be provided with controls that allow them to choose whether and when the system, program, or feature described herein enables the collection of user information (e.g., information about the user's captured photos or other images, social networks, social actions or activities, occupation, user preferences, or the user's current location) and whether to send content or communications to the user from the server. Additionally, certain data may be processed in one or more ways before being stored or used, thereby removing personally identifiable information. For example, a user's identity may be processed to the point that their personally identifiable information cannot be determined, or, if location information is available, the user's geographic location may be generalized (e.g., to the city, zip code, or state level), making it impossible to determine the user's specific location. Therefore, users can control what information about themselves is collected, how that information is used, and what information is provided to them.

[0053] The initial image includes one or more objects. In some embodiments, the initial image also includes one or more human subjects. The user interface module 202 receives user input that selects one or more objects in the initial image. The user input may include moving a finger around one or more objects in the initial image (e.g., by drawing a circle or other shape around an object), moving a finger over one or more objects (e.g., holding down for one or more seconds, having a dragging gesture on the pixels of the image depicting one or more objects), tapping one or more objects in the initial image once or multiple times (e.g., double-tapping to indicate selection), providing text recognition for one or more images (e.g., "the tree on the right"), etc.

[0054] In some embodiments, the user interface may highlight one or more selected objects in response to receiving user input. In some embodiments, where a tap can be associated with multiple objects, different numbers of taps may cause the user interface to highlight different objects. For example, in an initial image that is a beach scene with a bucket in front of a sandcastle, a first tap on the bucket / sandcastle area highlights the bucket first, a second tap highlights the sandcastle, and a third tap highlights both the bucket and the sandcastle. In this way, by mapping tap counts to individual objects or sets of two or more objects in the initial image, it becomes possible to select individual objects in the initial image that may be close to or partially overlap each other.

[0055] User interface module 202 generates a user-selected mask comprising object pixels corresponding to one or more selected objects. In some embodiments, user interface module 202 generates the user-selected mask by identifying all pixels associated with the user selection as belonging to the user-selected mask. In some embodiments, such as when the user input includes one or more objects surrounding an initial image, user interface module 202 generates the user-selected mask by performing object recognition to identify one or more objects surrounded by the user input and identifying pixels corresponding to one or more identified objects as part of the user-selected mask. User interface module 202 provides the user-selected mask to diffusion module 206.

[0056] In some embodiments, the user interface module 202 identifies objects (e.g., by performing object recognition) to identify the type of one or more objects in the initial image. The user interface module 202 may generate graphical data used to update the user interface to provide suggestions for modifying or replacing objects selected by the user. For example, if the user selects a mountain in the initial image, the user interface may include suggestions to change the mountain to include snow, make it greener, include animals on the mountain, etc. If the object is a human subject, the suggestions may include different types of clothing for the human subject. The suggestions may be based on objects that are generally close to the identified object, on the object most frequently requested for modification based on object type, or a combination of both.

[0057] The user interface module includes options for providing a text request associated with one or more selected objects in the initial image. For example, the user interface may include a text field for the user to directly input the text request, an audio button for providing audio input that is converted into a text request, etc. In some embodiments, the user interface may be updated using auto-completion suggestions when the user provides a text request. For example, for an outdoor scene, if the text field includes "change to m", the user interface module 202 may add "mountains" as an auto-completion suggestion. In some embodiments, the text request includes text associated with a suggestion displayed in the user interface and selected by the user.

[0058] In some embodiments, the user interface module 202 receives a text request from a user to generate an output image, rather than receiving user input to select objects in an initial image. For example, the initial image may be paired with a text request to change the sky in the initial image from noon to dawn.

[0059] In some embodiments, the user interface module 202 generates graphical data for displaying the output image. The user interface may also include options for editing the output image, sharing the output image, adding the output image to a photo album, etc.

[0060] user interface

[0061] Figure 3A An example user interface 300, including an initial image 302, is illustrated according to some embodiments described herein. The initial image 302 includes a human subject 304 and a tree 306. The user interface 300 also includes a share button 308, an edit button 310, and a trash can button 312. The user can select the edit button 310, which allows the user to select one or more objects in the user interface 300.

[0062] Figure 3BAn example user interface 325 according to some embodiments described herein is illustrated, the example user interface including a text request 335 and an initial image 327 with user input (as shown in the image from...). Figure 3A (The initial image 302 is the same). In this example, the user provides input (e.g., touch input) around the tree 329, and the user interface module 202 updates the initial image 327 to highlight the tree with a line 331 to indicate that the tree has been selected. The user interface 325 also includes a text field 333 in which the user enters the following text request 335: “light snow on pine tree”. In this example, the user input indicates that the tree 306 (selected by the user) in the initial image will be replaced by a pine tree with light snow on it.

[0063] In some embodiments, the user interface module 202 provides suggestions for modifications to the initial image 327. For example, the suggestions may be based on different seasons (e.g., changing weather conditions in the initial image from summer to winter), different weather conditions (e.g., adding rain), and / or effects (e.g., adding flickering). The user interface 325 also includes suggested modifications for the selected tree 329. Figure 3B In the example, the suggestions include: snow 337, which can be added to tree 329; gazebo 339, which can replace tree 329; bird 341, which can be added to the tree; or dog 343, which can replace the tree. In some embodiments, selecting one of the suggested modifications causes the corresponding text request to be displayed in text field 333, and the user can further modify the text request (not shown). For example, the user can select snow 337 and then modify text request 335 to “light snow on pine tree”, as shown. Figure 3B As shown. Once the user is satisfied with the text request 335, the user can select the arrow button 345 to request the generation of the output image.

[0064] Figure 3C Examples of embodiments according to the description herein include satisfying the following: Figure 3B The example user interface 350 for the output image 352 provided in the text request is shown. The output image 352 includes a human subject 354 (not modified from the initial image 302) and a pine tree 356 with light snow (replacing the tree 306 in the initial image 302). The user can save a copy 358, undo changes 360, or select the "Done" button 362. Based on the segmenter 204, no... Figure 3C The human subject 354 in the segmenter is modified, and the segmenter generates a preservation mask to prevent human pixels associated with the human subject 354 from being modified by the diffusion module 206 during the generation of the output image 352, as described in detail below.

[0065] In some embodiments, segmenter 204 segments one or more objects selected by the user in an initial image. Segmenter 204 generates a segmentation mask based on the segmentation of the one or more objects, identifying object pixels associated with those objects. In some embodiments, the segmentation mask is used in conjunction with a user-selected mask to identify the one or more selected objects for modification.

[0066] Segmenter 204 identifies whether a human subject is present in the initial image. If one or more objects selected by the user include a human subject, segmenter 204 segments the human subject's face. Segmenter 204 generates a preservation mask for the face, which includes pixels corresponding to the position of the face in the initial image. Segmenter 204 segments the subject's face to generate the preservation mask, which is provided as input to diffusion module 206 and prevents diffusion module 206 from modifying the face during the generation of the output image. This preservation mask corresponds to the face to prevent modification of the subject's face when aspects such as the subject's hair, clothing, etc., are changed.

[0067] Segmenter 204 can also segment portions other than the face, such as segmenting the entire body to prevent modification of the entire body. Body segments include pixels corresponding to the position of the body in the initial image. Body segmentation can be used to prevent modification of the entire human subject's body when the rest of the image is modified (such as changes to the background of the initial image). In some embodiments, a preservation mask includes all aspects of the initial image except for the modified portions. For example, when the subject's clothing is modified, the preservation mask may cover the face, hair, and background.

[0068] Segmenter 204 can automatically or in response to user input to segment one or more objects in an initial image. For example, segmenter 204 segments objects when user interface module 202 generates suggestions for modifying, removing, and / or replacing objects in the initial image. In another example, the user interface receives user input identifying an object to be modified, removed, and / or replaced, and segmenter 204 segments the object in response to its selection. In some embodiments, segmenter 204 generates a segmentation map that associates identity with each pixel in the initial image belonging to a face, body, object, etc. The segmentation map can be used to construct segmentation masks for different objects within the initial image.

[0069] Segmenter 204 performs segmentation by detecting objects in the initial image. Objects can be people, animals, cars, buildings, etc. A person can be the main subject of the initial image, or not (e.g., a bystander captured in the initial image). Bystanders can include people walking, running, cycling, standing behind a subject, or otherwise located within the initial image. In different examples, bystanders can be in the foreground (e.g., a person walking in front of the camera), at the same depth as the subject (e.g., a person standing next to the subject), or in the background. In some examples, there may be more than one bystander in the initial image. Bystanders can be in any pose (e.g., standing, sitting, crouching, lying down, jumping, etc.). Bystanders can face the camera, be at an angle to the camera, or have their backs to the camera.

[0070] Segmenter 204 can detect the type of an object by performing object recognition, comparing it with prior objects such as people, vehicles, and buildings, to identify the expected shape of the object and thus determine whether a pixel is associated with the selected object or with the background. Segmenter 204 can generate a region of interest for the selected object, such as a bounding box with x, y coordinates and a scale.

[0071] Segmenter 204 generates a retainable mask that at least covers the face of the subject. For example, the retainable mask for the face may include pixels corresponding to pixels of facial segments in the initial image. In some embodiments, the retainable mask includes additional or different body parts of the human subject, such as the subject's entire head, hands, body, etc.

[0072] In some embodiments, segmenter 204 generates a depth map of the initial image. The depth map is a representation of distance or depth information for each pixel in the initial image. The depth map may be a two-dimensional array where each pixel contains a value representing the distance from a camera (e.g., camera 243 if computing device 200 captured the initial image) to a corresponding point in the scene. The depth map provides a continuous representation of the depth information of the scene captured in the initial image. The depth map can be generated using a depth sensor (either as metadata generated during image capture if available in the initial image, or by deriving depth from pixel values ​​using depth estimation techniques).

[0073] The segmenter 204 can generate a preservation mask by clustering detections based on depth, using superpixels of the generated image and matching superpixel centroids with depth map values. More specifically, depth ranges can be determined using depth values ​​in the masked regions, and superpixels falling within these depth ranges can be identified. Another technique for generating the preservation mask includes weighting depth values ​​based on their proximity to the preservation mask, where the weights are represented by a distance transform map.

[0074] In some embodiments, the segmenter 204 may specify a circuit configuration (e.g., for a programmable processor, for a field-programmable gate array (FPGA), etc.) that enables the processor 235 to implement a machine learning model. In some embodiments, the segmenter 204 may include software instructions, hardware instructions, or a combination thereof. In some embodiments, the segmenter 204 may provide an application programming interface (API) that can be invoked by the operating system 262 and / or other applications 264 to, for example, apply a machine learning model to application data 266 to output a preserved mask.

[0075] The segmenter 204 uses training data to generate a trained machine learning model. For example, the training data may include pairs of initial images with one or more subjects and output images with one or more segmentation masks or preservation masks, depending on whether the training is for generating segmentation masks or preserving masks.

[0076] Training data can be obtained from any source, such as data repositories specifically tagged for training, data for which licenses are provided for use as training data for machine learning, etc. In some embodiments, training may occur on media server 101 that provides training data directly to user device 115, training may occur locally on user device 115, or a combination of both.

[0077] In some embodiments, segmenter 204 uses weights obtained from another application that have not been edited / transmitted. For example, in these embodiments, a trained model may be generated, for example, on a different device and provided as part of segmenter 204. In various embodiments, the trained model may be provided as a data file including a model structure or form (e.g., defining the number and type of neural network nodes, the connectivity between nodes, and the organization of nodes into multiple layers) and associated weights. Segmenter 204 may read the data file of the trained model and implement a neural network with node connectivity, layers, and weights based on the model structure or form specified in the trained model.

[0078] A trained machine learning model may include one or more model forms or structures. For example, a model form or structure may include any type of neural network, such as a linear network, a deep learning neural network that implements multiple layers (e.g., "hidden layers" between the input and output layers, where each layer is a linear network), a convolutional neural network (e.g., a network that splits or partitions input data into multiple parts or tiles, uses one or more neural network layers to process each tile individually and aggregates the results of processing from each tile), a sequence-to-sequence neural network (e.g., a network that takes sequential data such as words in a sentence or frames in a video as input and produces a sequence of results as output), and so on.

[0079] The model form or structure can specify the connectivity between various nodes and the organization of nodes into layers. For example, nodes in a first layer (e.g., an input layer) may receive data as input data or application data. For example, when a trained model is used for analysis of, for example, an initial image, such data may include, for example, one or more pixels per node. Subsequent intermediate layers may receive the outputs of nodes from previous layers as input, according to the connectivity specified in the model form or structure. These layers may also be referred to as hidden layers. For example, a first layer may output a segmentation between foreground and background. The final layer (e.g., an output layer) produces the output of the machine learning model. For example, an output layer may receive a segmentation of the initial image to foreground and background, and output pixels that are part of a preservative mask. In some embodiments, the model form or structure also specifies the number and / or type of nodes in each layer.

[0080] In various embodiments, the trained model may include one or more models. The one or more models in the model may include multiple nodes arranged in layers according to a model structure or form. In some embodiments, a node may be a computational node without memory, configured, for example, to process an input unit to produce an output unit. Computations performed by a node may include, for example, multiplying each node input in a plurality of node inputs by weights to obtain a weighted sum, and adjusting the weighted sum using bias or intercept values ​​to produce a node output. In some embodiments, computations performed by a node may also include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a nonlinear function. In various embodiments, such computations may include operations such as matrix multiplication. In some embodiments, computations performed by multiple nodes may be performed in parallel, for example, using multiple processor cores of a multi-core processor, an individual processing unit using a graphics processing unit (GPU), or a dedicated neural circuit system. In some embodiments, a node may include memory, for example, the ability to store one or more earlier inputs and use one or more earlier inputs when processing subsequent inputs. For example, a node with memory may include a Long Short-Term Memory (LSTM) node. LSTM nodes can use memory to maintain the "state" of the permitted node, which acts like a finite state machine (FSM).

[0081] In some embodiments, the trained model may include embeddings or weights for individual nodes. For example, the model may be started as a plurality of nodes organized into layers as specified by the model form or structure. During initialization, appropriate weights may be applied to the connections between each pair of nodes connected in the model form (e.g., nodes in consecutive layers of a neural network). For example, the appropriate weights may be randomly assigned or initialized to default values. The model may then be trained, for example, using training data to produce results.

[0082] Training may include the application of supervised learning techniques. In supervised learning, training data may include multiple inputs (e.g., images, segmentation maps, segmentation masks, preservation masks, etc.) and corresponding ground truth outputs for each input (e.g., ground truth segmentation masks that correctly identify pixels corresponding to selected objects and / or ground truth preservation masks that correctly identify portions of a subject in each image (such as the subject's face)). Based on a comparison of the model's output with the ground truth outputs, the values ​​of the weights are automatically adjusted, for example, in a way that increases the probability that the model will produce ground truth outputs for the images.

[0083] In various embodiments, the trained model includes a set of weights or embeddings corresponding to the model structure. In some embodiments, the trained model may include a fixed set of weights (e.g., downloaded from a server providing weights). In various embodiments, the trained model includes a set of weights or embeddings corresponding to the model structure. In embodiments where data is omitted, the segmenter 204 may generate a trained model based on prior training performed, for example, by the developer of segmenter 204, a third party, etc. In some embodiments, the trained model may include a fixed set of weights (e.g., downloaded from a server providing weights).

[0084] In some embodiments, a trained machine learning model receives an initial image with one or more selected objects. In some embodiments, the trained machine learning model outputs one or more segmentation masks that identify object pixels associated with one or more objects in the initial image. In some embodiments, if one or more selected objects include human subjects, the trained machine learning model generates one or more retainable masks corresponding to the one or more human subjects. For example, the one or more retainable masks may include image pixels corresponding to the faces of one or more subjects and exclude other pixels in the image.

[0085] The diffusion module 206 trains and implements a diffusion model to receive: an initial image and a text request for generating an output image; a preserving mask and / or a segmentation mask as input; and a depth map generated by the segmenter 204. In some embodiments, the initial image is described by red, green, and blue (RGB) color channels for each pixel, wherein the value in each color channel ranges from 0 to 255.

[0086] The diffusion model generates an output image that satisfies the text request and does not include object pixels associated with human subjects. In some embodiments, the diffusion model receives an empty mask as input, which identifies all pixels in the initial image as not associated with humans (regardless of whether the initial image includes humans). As a result of using the empty mask, the diffusion module 206 generates an output image that does not include human pixels.

[0087] In some embodiments where the initial image includes a human subject (either as a selected object or present in the image), the diffusion model also receives a retention mask from segmenter 204. The retention mask is used to prevent modifications to the human subject by the diffusion model during the generation of the output image.

[0088] In some embodiments, the diffusion module 206 trains a diffusion model using a two-step process to generate an output image. First, the diffusion model is trained to perform a forward diffusion process on an initial image, where Gaussian noise with variance is added to obtain a noisy image. Gaussian noise with variance is added again to obtain an image with progressively increasing noise until the final noisy image is achieved. Second, the diffusion model is trained to perform a reverse diffusion process, which uses a convolutional neural network (CNN) to transform the final noisy image into a meaningful output (e.g., an output image).

[0089] The diffusion module 206 trains a diffusion model to perform forward diffusion using training data including the initial image. The diffusion module 206 converts the initial image into a tensor. A tensor is an array of bytes with any number of dimensions. Tensors can be described as having arbitrary shapes because they can have any number of dimensions. The diffusion module 206 parses the bytes in the tensor to convert them into pixel data for the RGB color channels.

[0090] The diffusion module 206 can sample noise to match the shape (dimensions) of the initial image. The diffusion module 206 can sample random diffusion times and use these random diffusion times to generate noise and signal rates according to a diffusion timetable. The diffusion module 206 applies weights to the initial image to generate a noisy image. In some embodiments where the diffusion model is used to generate an output image from text, each forward diffusion step predicts noise from the noisy image and text embeddings generated from the text.

[0091] The diffusion module 206 calculates the loss (e.g., mean absolute error) between the predicted noise and the noise from the ground truth image, and takes a gradient step with respect to this loss function. After the gradient step, the neural network weights of the (training) diffusion model are updated to a weighted average of the existing weights and the trained neural network weights.

[0092] The diffusion module 206 can train a diffusion model to perform inverse diffusion and denoise noisy images, such that it predicts noise via an instruction neural network and then undoes the noise addition operation using the noise rate and signal rate to satisfy the text request. The diffusion model includes a CNN comprising convolutional layers, where the output of one layer is used as the input to subsequent layers. The convolutional layers include downsampling blocks and upsampling blocks, in which the initial image is spatially compressed but channel-wise expanded, and in the upsampling blocks, the representation is spatially expanded while reducing the number of channels.

[0093] The diffusion module 206 provides noise variance and a noisy image described by a tensor as input to the first convolutional layer in the CNN to increase the number of channels. The noise variance and the noisy image are cascaded across channels. In some embodiments, the diffusion module 206 includes skip connections between the outputs of convolutional layers performing downsampling and upsampling to achieve an equivalent spatial shaping layer in the network. The final convolutional layer reduces the number of channels to three RGB channels.

[0094] During training for the reverse diffusion process, the diffusion module 206 predicts noise to remove noise from the noisy image to achieve an initial image. The diffusion module 206 performs predictions in multiple steps, and these multiple steps may differ from those used during training for the forward diffusion process.

[0095] Figure 4 An example process 400 is illustrated, illustrating the training of a diffusion model according to some embodiments described herein to generate an output image 430 in response to a text request 420 and an initial image 405. The diffusion model includes a diffusion process 410 for performing forward diffusion and a CNN 425 for performing reverse diffusion.

[0096] An initial image 405 is provided as input to a diffusion process 410, which generates a corresponding noisy image 415. The initial image 405 is an image of a girl next to a tree. The noisy image 415 and a text request 420 (“light snow onpine tree”) are provided as input to a CNN 425. In some embodiments, a user-selected mask is also received, which identifies pixels associated with one or more selected objects in the initial image 405. The CNN 425 performs a reverse diffusion process to generate an output image 430 that satisfies the text request.

[0097] The architecture of a diffusion model can include different components. When a diffusion model is used to generate an output image based on an initial image and a text request, it includes an image encoder, a text encoder, and a CNN. A diffusion model can be started with the U-Net architecture, a specialized type of CNN, which can be modified to improve efficiency and facilitate more realistic output images.

[0098] Figure 5 An architecture of an example diffusion model 500 according to some embodiments described herein is illustrated. The diffusion model 500 is trained using training data including an initial image 502 and a condition 505. In some embodiments, the training data further includes ground truth output images, such as output images that satisfy text requests. In some embodiments, the training data further includes pairs of ground truth images with corresponding images of randomly masked portions of the ground truth images.

[0099] Condition 505 includes a text encoder 507, a temporal encoder 509, a user-selected mask 511, a depth map 513, an optional retention mask 514, an optional segmentation mask 515, and a classifier-free guide 516. The text encoder 507 encodes the text request (i.e., the text condition) by converting the text into tokens used to represent the text request in a vector space (embedding space). The temporal encoder 509 encodes the diffusion timestamp using positional encoding.

[0100] The user-selected mask 511 identifies object pixels associated with one or more objects in the initial image. During inference (i.e., during the generation of the output image), the user-selected mask 511 identifies regions in the output image to be modified. The user-selected mask 511 can identify object pixels associated with one or more selected objects.

[0101] Depth map 513 identifies the depth of one or more image pixels in the initial image. Depth map 513 is provided as input to CNN 512 to preserve the relative depths of various objects in the initial image in the output image. For example, if the selected image includes a door with a handle, depth map 513 is used to preserve the structure of the door and retain the handle in the output image.

[0102] The retention mask 514 identifies pixels corresponding to the human subject in the initial image and those to be retained during the generation of the output image 557. For example, the retention mask may include the human subject's hair (if the user instructs the hair to remain the same (or more generally, if no changes to the hair are specified in condition 505)), the human subject's fingers, the subject's entire body (to prevent excessive modification of the pet in the case of the subject being a pet), etc. In some embodiments where the output image modifies the human subject's clothing, the retention mask excludes pixels of the human subject's clothing and instead includes the remaining pixels associated with the human subject to prevent modifications to the human subject by the diffusion model 500. In some embodiments, multiple different generative machine learning diffusion models may be trained and made available for image generation, such as shape-preserving models, structure-preserving models, etc.

[0103] The segmentation mask 515 identifies one or more selected objects. The segmentation mask 515 can be used to improve the recognition of the user-selected mask 511.

[0104] In some embodiments, instead of using a reserved mask 514, condition 505 may include an empty mask that identifies all pixels in the initial image 502 as not associated with humans.

[0105] In some embodiments, a classifier-free guide 516 is used to control the depth in the output image. A classifier guide controls the categories generated by a classification model. The classifier-free guide 516 trains a diffusion model on conditions with conditional dropout, i.e., conditions are removed at a certain percentage of time. In some embodiments, the removed conditions are replaced with special input values ​​representing the absence of conditional information. Higher conditional dropout values ​​preserve more structure of one or more objects in the initial image than lower conditional dropout values. A drawback of higher conditional dropout values ​​is that the increased structure may come at the cost of reduced diversity in the output image.

[0106] An initial image 502 is provided as input to the first layer of CNN 512, and conditions 505 are provided as input to each block within CNN 512. CNN 512 includes: encoder blocks 517, 520, 525, and 530; intermediate block 535; and decoder blocks 540, 545, 550, and 555 with skip connections. In some embodiments, the model is a diffusion model 500 and contains 25 blocks, of which 8 blocks are downsampled convolutional layers or upsampled convolutional layers. Although Figure 5 Four encoder blocks and four decoder blocks are shown, but in various embodiments, fewer or more encoder blocks and / or decoder blocks may be used (and the number of encoder blocks and the number of decoder blocks may be different).

[0107] The denoising process can occur in the pixel space or in the latent space of the diffusion model. In some embodiments, during training, the diffusion module 206 performs preprocessing on the initial image 502 to transform the initial image 502 from a pixel-space image to a latent space (e.g., a vector representation of the image in a high-dimensional vector space). The diffusion module 206 performs training by transforming one or more conditions in condition 505 from the input size to a feature space vector that matches the size of the CNN 512.

[0108] The diffusion module 206 trains a diffusion model to receive an initial image 502 and progressively adds noise to the initial image 502 with each iteration of the diffusion model to produce a noisy image. Considering that the set of conditions 505 includes the time generated by the temporal encoder 509, the text request encoded by the text encoder 507, and other task-specific conditions (e.g., user-selected mask 511, depth map 513, preservation mask 514, segmentation mask 515, and classifier-free guidance 516), the image diffusion model is trained to predict the noise to be added to the noisy image. The diffusion module 206 trains the diffusion model to generate multiple output images that satisfy the text request and do not include human pixels by progressively removing noise (via a denoising process). In some embodiments, the denoising during training includes approximately 10,000 optimization steps to minimize the loss between the generated output images and the ground truth output images.

[0109] In some embodiments, the diffusion module 206 trains the diffusion model using three different versions of varying amounts of text requests and depth values. For example, the diffusion module 206 may run a first version of the diffusion model without text requests and depth values, a second version of the diffusion model with text requests and no depth values, and a third version of the diffusion model with text requests and depth values. Training each version of the diffusion model may include multiple iterations.

[0110] Once the diffusion model is trained, it receives a text request to generate an output image, a corresponding depth map, and a user-selected mask. The diffusion model is trained to generate output pixels that are not associated with human subjects. The diffusion model performs a diffusion process on an initial image to generate a noisy image based on the initial image. In some embodiments, the diffusion model performs an inverse diffusion process, such as DDIM inversion, to generate an output image from the noisy image, wherein the output image is generated according to condition 505. The diffusion model performs inverse diffusion by predicting noise to be added to the noisy image and generating an output image that satisfies the text request.

[0111] method

[0112] Figure 6 An example method 600 is illustrated for training a diffusion model to generate output images based on text requests. Method 600 can be derived from... Figure 2 The method 600 is executed by the computing device 200. In some embodiments, the method 600 is executed by the user device 115, the media server 101, or is executed partly on the user device 115 and partly on the media server 101.

[0113] Figure 6 Method 600 may begin at box 602. At box 602, training data is generated, which includes an initial image with one or more selected objects and conditions. For each image, the conditions include a text request, a depth map, and a user-selected mask. In some embodiments, the training data further includes pairs of ground truth images with corresponding images of masked portions (e.g., randomly masked portions) of the ground truth images. The depth map may include depth values ​​that identify the depth of image pixels in the initial image, wherein training the diffusion model includes training the output image to retain the depth map associated with the initial image.

[0114] The condition may further include a retention mask that identifies human pixels corresponding to one or more human subjects in the initial image, the retention mask being used by a diffusion model to prevent modification of the human pixels during the generation of the output image. Method 600 may further include segmenting one or more selected objects in the initial image and generating a segmentation mask, wherein the condition further includes the segmentation mask.

[0115] The condition may further include classifier-free guidance of the depth map, such that higher values ​​preserve more of the structure of one or more objects in the initial image than lower values. Box 602 may be followed by box 604.

[0116] At box 604, the diffusion model is trained to output an image that meets the condition and does not include human pixels. The training involves repeatedly generating the output image until the comparison between the output image and the corresponding ground truth image meets the threshold loss value.

[0117] Training the diffusion model is based on varying text requests and depth values. Training may include: a first run of the diffusion model without text requests and depth values, a second run of the diffusion model with text requests and no depth values, and a third run of the diffusion model with text requests and depth values.

[0118] Figure 7 An example method 700 for generating an output image based on a text request is illustrated. Method 700 can be derived from... Figure 2 The method 700 is executed by the computing device 200. In some embodiments, the method 700 is executed by the user device 115, the media server 101, or is executed partly on the user device 115 and partly on the media server 101.

[0119] Figure 7 Method 700 may begin at box 702. At box 702, an initial image, user input selecting one or more objects in the initial image, and a text request for generating an output image that modifies one or more selected objects in the initial image are received. User input may be provided from the user, who performs one or more actions selected from the following group: around one or more objects in the initial image, moving a finger on one or more objects in the image, tapping one or more objects in the initial image, providing text recognition for one or more objects, and combinations thereof.

[0120] In some embodiments, the method further includes, in response to receiving user input, performing object identification to identify one or more types of one or more objects, and providing one or more suggestions for modifying one or more objects based on the types of one or more objects. Box 702 may be followed by box 704.

[0121] At box 704, determine whether permission to modify the original image has been obtained. If permission has not been obtained, box 706 can follow box 704. If permission has been obtained, box 708 can follow box 704.

[0122] At box 708, one or more objects in the initial image may be optionally segmented. A segmentation mask (object mask) may be generated for one or more objects in the initial image, wherein each mask identifies pixels in the initial image belonging to the corresponding object. Box 708 may be followed by box 710.

[0123] At box 710, a user-selected mask is generated, comprising object pixels associated with one or more objects. In some embodiments, the user-selected mask is generated based on user input (e.g., tapping, circling, or otherwise selecting an object) and on segmenting one or more objects (e.g., matching the user input with previously segmented objects from box 708). Box 710 may be followed by box 712.

[0124] At box 712, the diffusion model receives a text request, a depth map, and a user-selected mask as input to generate the output image. The pre-trained diffusion model generates output pixels that are not associated with human subjects and respond to the text request and the user-selected mask, wherein the output image conforms to the depth map (e.g., generated objects added to the output image are at similar depths to the objects they replace). The depth map identifies the depth of image pixels in the initial image, and the output image retains the depth map of the initial image. Depth can be controlled using a classifier-free approach, and higher conditional dropout values ​​preserve more of the structure of one or more objects in the initial image than lower conditional dropout values. The input to the diffusion model may further include a segmentation mask.

[0125] In some embodiments, the method further includes performing object recognition to identify one or more objects and one or more humans in the initial image, wherein the input to the diffusion model further includes one or more preservation masks that identify human pixels corresponding to one or more humans (human faces and / or other parts of the body, such as limbs, hair, torso, etc.) in the initial image, the one or more preservation masks being used by the diffusion model to prevent modification of the human pixels. Box 712 may be followed by box 714.

[0126] At box 714, the diffusion model outputs an output image that satisfies the text request. The output image is provided to user interface module 202 for display in the user interface. Users can use the output image as input for further modification, save the output image, share the output image with others, etc. In various embodiments where the output image is shared with others, the output image may include metadata (or embedded pixel-level features) that enable the output image to be recognized as having been modified using generative AI.

[0127] In various embodiments, text requests from users may be subject to one or more filters to ensure that the resulting output image conforms to applicable rules and standards. For example, filters may detect text requests that prevent certain modifications to the image (e.g., adding objects of prohibited categories, changing objects in the image that conform to certain standards, etc.). In response to such detections, the user is provided with guidance on types of text requests that are not permitted. Additionally, the user may be provided with guidance on constructing a text request specifying their requirements for the output image.

[0128] In the foregoing description, numerous specific details have been set forth for purposes of explanation in order to provide a thorough understanding of this specification. However, it will be apparent to those skilled in the art that this disclosure may be practiced without these specific details. In some instances, structures and apparatuses have been shown in block diagram form to avoid obscuring descriptions. For example, embodiments may be described above primarily with reference to user interfaces and specific hardware. However, embodiments are applicable to any type of computing device that receives data and commands, as well as any peripheral devices that provide services.

[0129] The references to "some embodiments" or "some examples" in this specification mean that a particular feature, structure, or characteristic described in connection with an embodiment or example may be included in at least one implementation of this description. The appearance of the phrase "some embodiments" in various places in the specification does not necessarily refer to the same embodiment.

[0130] Some parts of the detailed description above are presented in terms of algorithms and symbolic representations of operations on data bits in computer memory. These algorithmic descriptions and representations are the means by which those skilled in the art of data processing most effectively communicate the substance of their work to others skilled in the art. An algorithm herein and generally is considered to be a self-consistent sequence of steps that produce a desired result. These steps are those that require physical manipulation of physical quantities. Typically, although not necessary, these quantities are usually in the form of electrical or magnetic data that can be stored, transmitted, combined, compared, and otherwise manipulated. It has been found that, primarily for reasons of general use, these data may sometimes be appropriately referred to as bits, values, elements, symbols, characters, items, numbers, etc.

[0131] However, it should be remembered that all these and similar terms will be associated with appropriate physical quantities and are merely convenient labels applied to those quantities. Unless otherwise specifically indicated, as will be apparent from the following discussion, it should be understood that throughout the description, the use of terms including “processing” or “computing” or “operation” or “determining” or “displaying” refers to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities in the registers and memory of the computer system and transforms that data into other data similarly represented as physical quantities in the computer system's memory or registers or other such information storage, transmission, or display devices.

[0132] Embodiments of this specification may also relate to a processor for performing one or more steps of the methods described above. The processor may be a dedicated processor that is selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory computer-readable storage medium, including but not limited to any type of disk, including optical discs, ROMs, CD-ROMs, magnetic disks, RAM, EPROMs, EEPROMs, magnetic cards or optical cards, flash memory (including USB flash drives with non-volatile memory), or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0133] The specification may take the form of some entirely hardware embodiments, some entirely software embodiments, or some embodiments that include both hardware and software elements. In some embodiments, this specification is implemented in software, including but not limited to firmware, resident software, microcode, etc.

[0134] Furthermore, this description may take the form of a computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by or in conjunction with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable medium may be any device capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, device, or apparatus.

[0135] A data processing system suitable for storing or executing program code will include at least one processor directly or indirectly coupled to memory elements via a system bus. Memory elements may include local memory, mass storage, and cache memory used during the actual execution of the program code, the cache memory providing temporary storage for at least some of the program code to reduce the number of times code must be retrieved from mass storage during execution.

Claims

1. A computer-implemented method to generate an image based on a text request, the method comprising: receiving an initial image, user input selecting one or more objects in the initial image, and a text request to generate an output image that modifies the one or more selected objects in the initial image; generating a user-selected mask that includes object pixels corresponding to the one or more selected objects; providing the text request, a depth map, and the user-selected mask as inputs to a diffusion model to generate the output image, wherein the diffusion model is trained to generate output pixels for the output image that are not associated with a human subject; and generating the output image that satisfies the text request using the diffusion model.

2. The method of claim 1, wherein: the depth map identifies depths of image pixels in the initial image; and the output image preserves the depth map of the initial image.

3. The method of claim 2, wherein, depths are controlled using classifier-free guidance, and higher conditional dropout values preserve more structure of the one or more selected objects in the initial image than lower conditional dropout values.

4. The method of claim 1, the user input is provided from a user performing one or more actions selected from the group of: moving a finger around the one or more objects in the initial image, tapping the one or more objects in the image, providing text recognition of the one or more objects, and combinations thereof.

5. The method of claim 1, further comprising: performing object recognition to identify one or more humans in the initial image; wherein the inputs to the diffusion model further include one or more preservation masks that identify human pixels corresponding to the one or more humans in the initial image, the one or more preservation masks used by the diffusion model to prevent modification of the human pixels.

6. The method of claim 1, further comprising: in response to receiving the user input, performing object recognition to identify one or more types of the one or more selected objects; and based on the one or more types of objects, providing one or more suggestions for modifying the one or more selected objects.

7. The method of claim 1, further comprising: segmenting the one or more selected objects in the initial image; and generating a segmentation mask, wherein the inputs to the diffusion model further include the segmentation mask.

8. A computer-implemented method to train a diffusion model, the method comprising: generating training data that includes initial images having one or more selected objects and conditions, for each initial image, the conditions including a text request, a depth map, and a user-selected mask; and training the diffusion model to output an image that satisfies the condition and does not include human pixels, wherein the training includes repeatedly generating the output image until a comparison of the output image to a corresponding ground truth image satisfies a threshold loss value.

9. The method of claim 8, further comprising: segmenting the one or more selected objects in the initial image; and generating a segmentation mask, wherein the condition further includes the segmentation mask.

10. The method of claim 8, wherein: the depth map includes depth values that identify depths of image pixels in the initial image; and training the diffusion model includes training the output image to preserve the depth map associated with the initial image.

11. The method of claim 10, further comprising: training the diffusion model based on varying amounts of the textual request and the depth values by running a first version of the diffusion model that does not have the textual request and does not have depth values, running a second version of the diffusion model that has the textual request and does not have depth values, and running a third version of the diffusion model that has the textual request and has the depth values.

12. The method of claim 10, wherein: the condition further includes no classifier guidance; and an amount of no classifier guidance preserves structures of the one or more selected objects in the initial image more than a lower condition dropout value.

13. The method of claim 8, wherein, the condition further includes a preservation mask that identifies human pixels corresponding to one or more human subjects in the initial image, the preservation mask used by the diffusion model to prevent modification of human pixels during generation of the output image.

14. The method of claim 8, wherein, the training data further includes pairs of a ground truth image and a corresponding image that has a random mask of portions of the ground truth image.

15. The method of claim 8, wherein, the diffusion model is trained for use in the method to generate an image of claim 1.

16. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform operations comprising: receiving an initial image, user input that selects one or more objects in the initial image, and a textual request to generate an output image that modifies one or more selected objects in the initial image; generating a user-selected mask that includes object pixels corresponding to the one or more selected objects; providing the textual request to generate the output image, a depth map, and the user-selected mask as inputs to a diffusion model, wherein the diffusion model is trained to generate output pixels for the output image that are not associated with human subjects; and generating the output image that satisfies the textual request using the diffusion model.

17. The non-transitory computer-readable medium of claim 16, wherein: the depth map identifies depths of image pixels in the initial image; and the diffusion model is trained to preserve the depths of the image pixels in the output image. The output image preserves the depth map of the initial image.

18. The non-transitory computer-readable medium of claim 16, wherein, The user input is provided from a user performing one or more actions selected from the group of moving a finger around the one or more objects in the initial image, moving a finger over the one or more objects in the image, tapping the one or more objects in the initial image, providing text recognition of the one or more objects, and combinations thereof.

19. The non-transitory computer-readable medium of claim 16, wherein, The operations further include: performing object recognition to identify one or more humans in the initial image; wherein the input to the diffusion model further includes one or more preservation masks that identify human pixels corresponding to the one or more humans in the initial image, the one or more preservation masks used by the diffusion model to prevent modification of the human pixels.

20. The non-transitory computer-readable medium of claim 16, wherein, The operations further include: in response to receiving the user input, performing object recognition to identify one or more types of the one or more selected objects; and providing one or more suggestions for modifying the one or more selected objects based on the type of the one or more objects.

Images