Hint-driven image editing using machine learning

Abstract

A media application receives an initial image and a text request to alter the initial image, the initial image including a subject having a face. The media application generates, from the initial image, a retention mask corresponding to the face of the subject. The media application provides the text request, the initial image, and the retention mask as input to the diffusion model. The diffusion model outputs a denoised initial image based on the initial image; performing text conditionalization and forward diffusion of the text request to generate a noisy transformed image that satisfies the text request; and outputting a denoised converted image based on the noisy converted image, the extracted features, and the self-attention map. The media application mixes the denoised initial image, the retention mask, and the denoised converted image to form an output image, wherein the retention mask prevents modification of the face from the initial image.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 465,226, filed on May 9, 2023, entitled “Prompt-Driven Image Editing Using Machine-Learning,” which is incorporated herein in its entirety. Background Art

[0002] Generative artificial intelligence (AI) can be used to generate images based on text prompts. For example, a user can request an image of an avocado chair, and generative AI will create the image. The results are often problematic, especially when the image includes people, as more detailed aspects may not be correctly rendered. For example, generative AI is still in the development stage when it comes to capturing complex features such as fingers, eyes, and mouths.

[0003] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the presently named inventors (to the extent that it is described in this background section) and aspects of the specification that might not have been identified as prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art of the present disclosure. Summary of the invention

[0004] A computer-implemented method includes receiving an initial image and a text request to modify the initial image, the initial image including a subject having a face. The method further includes generating a retention mask corresponding to the face of the subject. The method further includes providing the text request, the initial image, and the retention mask as inputs to a diffusion model. The method further outputs a denoised initial image based on the initial image using the diffusion model. The method further includes performing text conditioning on the text request and forward diffusion using the diffusion model to generate a noisy transformed image that satisfies the text request. The method further includes outputting a denoised transformed image based on the noisy transformed image, extracted features, and a self-attention map using the diffusion model. The method further includes blending the denoised initial image, the retention mask, and the denoised transformed image to form an output image, wherein the retention mask prevents modification of the face from the initial image.

[0005] In some embodiments, outputting the denoised initial image includes: performing inverse diffusion of the initial image using a diffusion model to generate a noisy initial image based on the initial image; and providing the noisy initial image to a first convolutional neural network (CNN) and outputting the denoised initial image; and outputting the denoised transformed image includes: providing the noisy transformed image to a second CNN; injecting the extracted features and the self-attention map during diffusion; and outputting the denoised transformed image. In some embodiments, the inverse diffusion is a denoised diffusion implicit model (DDIM) inversion.

[0006] In some embodiments, the method further includes: receiving a selection of a first object in the initial image, wherein the text request includes a comment to replace the first object in the initial image with a second object. In some embodiments, the method further includes: identifying an area in the background to be replaced or modified from the initial image, and providing a suggestion to replace or modify the background, wherein the text request is associated with the suggestion. In some embodiments, the method further includes: identifying an object to be removed from the background from the initial image, and providing a suggestion to remove the object from the background. In some embodiments, the method further includes: identifying one or more objects to be replaced from the initial image, and providing a suggestion to replace the object.

[0007] In some embodiments, the text request is to change the background of the initial image, and the retention mask further includes one or more parts of the subject other than the subject's face. In some embodiments, the text request further includes at least one selection from a set of global presets, a menu of options, a library of pre-made prompts, or a combination thereof.

[0008] A non-transitory computer-readable medium includes instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations. The operations include: receiving an initial image and a text request to modify the initial image, the initial image including a subject having a face; generating a retention mask corresponding to the face of the subject; providing the text request, the initial image, and the retention mask as inputs to a diffusion model; using the diffusion model to output a denoised initial image based on the initial image; using the diffusion model to perform text conditioning and forward diffusion on the text request to generate a noisy transformed image that satisfies the text request; using the diffusion model to output a denoised transformed image based on the noisy transformed image, extracted features, and a self-attention map; and blending the denoised initial image, the retention mask, and the denoised transformed image to form an output image, wherein the retention mask prevents modification of the face from the initial image.

[0009] In some embodiments, outputting the denoised initial image includes: performing inverse diffusion of the initial image using a diffusion model to generate a noisy initial image based on the initial image; and providing the noisy initial image to a first convolutional neural network (CNN) and outputting the denoised initial image; and outputting the denoised transformed image includes: providing the noisy transformed image to a second CNN; injecting the extracted features and the self-attention map during diffusion; and outputting the denoised transformed image. In some embodiments, the inverse diffusion is a DDIM inversion. In some embodiments, the operation further includes: receiving a selection of a first object in the initial image, wherein the text request includes an annotation to replace the first object in the initial image with a second object. In some embodiments, the operation further includes: identifying an area in the background to be replaced or modified from the initial image, and providing a suggestion to replace or modify the background, wherein the text request is associated with the suggestion. In some embodiments, the operation further includes: identifying an area in the background to be replaced or modified from the initial image, and providing a suggestion to replace or modify the background, wherein the text request is associated with the suggestion. In some embodiments, the operation further includes: identifying an area in the background to be replaced or modified from the initial image, and providing a suggestion to replace or modify the background, wherein the text request is associated with the suggestion.

[0010] A system includes a processor and a memory coupled to the processor, the memory having instructions stored thereon, which when executed by the processor causes the processor to perform operations. The operations include: receiving an initial image and a text request to modify the initial image, the initial image including a subject having a face; generating a retention mask corresponding to the face of the subject; providing the text request, the initial image, and the retention mask as inputs to a diffusion model; using the diffusion model to output a denoised initial image based on the initial image; using the diffusion model to perform text conditioning and forward diffusion on the text request to generate a noisy transformed image that satisfies the text request; using the diffusion model to output a denoised transformed image based on the noisy transformed image, extracted features, and a self-attention map; and blending the denoised initial image, the retention mask, and the denoised transformed image to form an output image, wherein the retention mask prevents modification of the face from the initial image.

[0011] In some embodiments, outputting the denoised initial image includes: performing inverse diffusion of the initial image using a diffusion model to generate a noisy initial image based on the initial image; and providing the noisy initial image to a first convolutional neural network (CNN) and outputting the denoised initial image; and outputting the denoised transformed image includes: providing the noisy transformed image to a second CNN; injecting the extracted features and the self-attention map during diffusion; and outputting the denoised transformed image. In some embodiments, the inverse diffusion is a DDIM inversion. In some embodiments, the operation further includes: receiving a selection of a first object in the initial image, wherein the text request includes an annotation to replace the first object in the initial image with a second object. In some embodiments, the operation further includes: identifying an area in the background to be replaced or modified from the initial image, and providing a suggestion to replace or modify the background, wherein the text request is associated with the suggestion. In some embodiments, the operation further includes: identifying an area in the background to be replaced or modified from the initial image, and providing a suggestion to replace or modify the background, wherein the text request is associated with the suggestion. In some embodiments, the operation further includes: identifying an area in the background to be replaced or modified from the initial image, and providing a suggestion to replace or modify the background, wherein the text request is associated with the suggestion. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram of an example network environment according to some embodiments described herein.

[0013] Figure 2 is a block diagram of an example computing device according to some embodiments described herein.

[0014] Figure 3A An example initial image according to some embodiments described herein is illustrated.

[0015] Figure 3B An example output image is illustrated in which a shirt is altered and an object is replaced with another object according to some embodiments described herein.

[0016] Figure 3C Illustrated are example output images in which a user's outfit, hair, and skin are altered according to some embodiments described herein.

[0017] Figure 3D An example output image in which the background is altered is illustrated according to some embodiments described herein.

[0018] Figure 4 An example user interface including options for selecting different areas of an image to be altered, global presets to be applied, fields for providing text, and an example output image is illustrated according to some embodiments described herein.

[0019] Figure 5An example user interface including an option for receiving user input for selecting an object to be replaced by a text request is illustrated in accordance with some embodiments described herein.

[0020] Figure 6 An example user interface including a menu of options and a library of pre-made prompts is illustrated according to some embodiments described herein.

[0021] Figure 7 An example architecture for generating a self-attention map and using the self-attention map during diffusion according to some embodiments described herein is illustrated.

[0022] Figure 8 An example architecture for generating images using a diffusion process with inversion and context conditioning according to some embodiments described herein is illustrated.

[0023] Fig. 9 is a block diagram of an example architecture for generating an image in conjunction with a text request according to some embodiments described herein.

[0024] Fig.10 An example method for generating an output image according to a text request according to some embodiments described herein is illustrated.

[0025] Fig.11 Another example method to generate an output image according to a text request according to some embodiments described herein is illustrated. DETAILED DESCRIPTION

[0026] Generative artificial intelligence (AI) can be used to generate images based on text prompts. For example, a user can request an image of an avocado chair, and generative AI will create the image. The results are often problematic, especially when the image includes people, as more detailed aspects may not be correctly rendered. For example, generative AI is still in the development stage when it comes to capturing complex features such as fingers, eyes, and mouths.

[0027] The technology described below includes a media application that receives an initial image and a text request to change the initial image. The initial image includes a subject with a face (such as an initial image of a family on a mountaintop) and a text request to replace the family's clothing (including anorak) with summer clothing. The text request can be received directly from a user or selected from a library of pre-made prompts or a menu of options. The media application generates a retention mask corresponding to the face.

[0028] A text request, an initial image, and a retention mask are provided as inputs to a diffusion model. The diffusion model outputs a denoised initial image based on the initial image, performs text conditioning and forward diffusion on the text request to generate a noisy transformed image that satisfies the text request, and outputs a denoised transformed image based on the noisy transformed image, the extracted features, and the self-attention map. The denoised initial image, the retention mask, and the denoised transformed image are mixed to form an output image, wherein the retention mask prevents modification of faces from the initial image. The extracted features and the self-attention map are used to maintain the structure of the initial image and make changes to the initial image in a faster process than would be assumed to occur if the output image was generated without reference to the initial image.

[0029] The media application may perform additional steps, such as identifying areas to replace or modify. Continuing with the above example, the media application may suggest making the background less cloudy. The media application may also identify objects for removal. For example, the media application may suggest removing other people and hiking gear from the background of the initial image.

[0030] Sample Environment 100

[0031] Figure 1 A block diagram of an example environment 100 is illustrated. In some embodiments, the environment 100 includes a media server 101 coupled to a network 105, a user device 115a, and a user device 115n. Users 125a, 125n may be associated with respective user devices 115a, 115n. In some embodiments, the environment 100 may include Figure 1 Other servers or devices not shown in FIG. Figure 1 In the illustrative drawings and the remaining drawings, a letter following a reference number (e.g., "115a") indicates a reference to the element having that specific reference number. A reference number in the text without a following letter (e.g., "115") indicates a general reference to an embodiment of the element with that reference number.

[0032] The media server 101 may include a processor, a memory, and network communication hardware. In some embodiments, the media server 101 is a hardware server. The media server 101 is communicatively coupled to the network 105 via a signal line 102. The signal line 102 may be a wired connection (such as Ethernet, coaxial cable, fiber optic cable, etc.) or a wireless connection (such as Wi-Fi®, Bluetooth® or other wireless technologies). In some embodiments, the media server 101 sends data to one or more user devices in the user devices 115a, 115n via the network 105 and receives data from the one or more user devices. The media server 101 may include a media application 103a and a database 199.

[0033] The database 199 may store machine learning models, training data sets, images, etc. The database 199 may also store social network data associated with the user 125, user preferences of the user 125, etc.

[0034] User device 115 may be a computing device including a memory coupled to a hardware processor. For example, user device 115 may include a mobile device, a tablet computer, a mobile phone, a wearable device, a head mounted display, a mobile email device, a portable game player, a portable music player, a reader device, or another electronic device capable of accessing network 105.

[0035] In the illustrated implementation, user device 115a is coupled to network 105 via signal line 108, and user device 115n is coupled to network 105 via signal line 110. Media application 103 may be stored on user device 115a as media application 103b and / or stored on user device 115n as media application 103c. Signal lines 108 and 110 may be wired connections (such as Ethernet, coaxial cable, fiber optic cable, etc.) or wireless connections (such as Wi-Fi®, Bluetooth® or other wireless technologies). User devices 115a, 115n are accessed by users 125a, 125n, respectively. By way of example, Figure 1 Although Figure 1 Two user devices 115 a and 115 n are illustrated, but the present disclosure is applicable to a system architecture having one or more user devices 115 .

[0036] The media application 103 may be stored on the media server 101 or the user device 115. In some embodiments, the operations described herein are performed on the media server 101 or the user device 115. In some embodiments, some operations may be performed on the media server 101 and some operations may be performed on the user device 115. The execution of the operations is based on user settings. For example, the user 125a may specify a setting that an operation is to be performed on its corresponding device 115a instead of on the media server 101. With such settings, the operations described herein are performed entirely on the user device 115a, and no operations are performed on the media server 101. Further, the user 125a may specify that the user's images and / or other data will only be stored locally on the user device 115a and not on the media server 101. With such settings, no user data is transmitted to or stored on the media server 101. The transmission of user data to the media server 101, any temporary or permanent storage of such data by the media server 101, and the performance of operations on such data by the media server 101 are only performed if the user has consented to the transmission, storage and performance of operations by the media server 101. The user is provided with the option to change the settings at any time, for example so that they can enable or disable the use of the media server 101.

[0037] If a machine learning model (e.g., a neural network or other type of model) is used for one or more operations, it is stored and utilized locally on the user device 115 with specific user permission. Server-side models are used only with user permission. Further, trained models can be provided for use on the user device 115. During such use, if the user 125 permits, on-device training of the model can be performed. If the user 125 permits, updated model parameters can be transmitted to the media server 101, for example, to implement federated learning. The model parameters do not include any user data.

[0038] The media application 103 receives an initial image and a text request to modify the initial image, the initial image including a subject having a face. For example, the media application 103 receives the initial image from a camera as part of the user device 115, or the media application 103 receives the initial image through the network 105. The media application 103 generates a retention mask corresponding to the face of the subject from the initial image. The media application 103 provides the text request, the initial image, and the retention mask as input to the diffusion model. The diffusion model outputs a denoised initial image based on the initial image; performs text conditioning and forward diffusion on the text request to generate a noisy converted image that satisfies the text request; and outputs a denoised converted image based on the noisy converted image, the extracted features, and the self-attention map. The media application 103 mixes the denoised initial image, the retention mask, and the denoised converted image to form an output image, wherein the retention mask prevents modification of the face from the initial image. The output image satisfying the text request corresponds to the initial image, wherein in the output image, pixels of the subject's face are pixels of the subject's face in the initial image, and pixels that do not define a face are pixels of the initial image modified by a diffusion process according to the text request.

[0039] In some embodiments, media application 103 can be implemented using hardware, including a central processing unit (CPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a machine learning processor / coprocessor, any other type of processor, or a combination thereof. In some embodiments, media application 103a can be implemented using a combination of hardware and software.

[0040] Example computing device 200

[0041] Figure 2 2 is a block diagram of an example computing device 200 that can be used to implement one or more features described herein. The computing device 200 can be any suitable computer system, server, or other electronic or hardware device. In one example, the computing device 200 is a media server 101 for implementing a media application 103a. In another example, the computing device 200 is a user device 115.

[0042] In some embodiments, computing device 200 includes processor 235, memory 237, input / output (I / O) interface 239, display 241, camera 243, and storage device 245, all coupled via bus 218. Processor 235 may be coupled to bus 218 via signal line 222, memory 237 may be coupled to bus 218 via signal line 224, I / O interface 239 may be coupled to bus 218 via signal line 226, display 241 may be coupled to bus 218 via signal line 228, camera 243 may be coupled to bus 218 via signal line 230, and storage device 245 may be coupled to bus 218 via signal line 232.

[0043] The processor 235 may be one or more processors and / or processing circuits for executing program code and controlling the basic operation of the computing device 200. A "processor" includes any suitable hardware system, mechanism or component for processing data, signals or other information. The processor may include a system having: a general-purpose central processing unit (CPU) with one or more cores (e.g., in a single-core, dual-core or multi-core configuration), multiple processing units (e.g., in a multi-processor configuration), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a dedicated circuit system for implementing functionality, a dedicated processor for implementing processing based on a neural network model, a neural circuit, a processor optimized for matrix calculations (e.g., matrix multiplication), or other systems. In some embodiments, the processor 235 may include one or more coprocessors for implementing neural network processing. In some embodiments, the processor 235 may be a processor that processes data to generate a probabilistic output, for example, the output generated by the processor 235 may be inaccurate, or may be accurate within a range from the expected output. Processing does not have to be limited to a specific geographic location or have time constraints. For example, the processor may perform its functions in real time, offline, in batch mode, etc. Portions of the processing may be performed at different times and in different locations by different (or the same) processing systems.A computer may be any processor in communication with a memory.

[0044] Memory 237 is typically provided in computing device 200 for access by processor 235 and may be any suitable processor-readable storage medium suitable for storing instructions for execution by a processor or set of processors and located separately from and / or integrated with processor 235, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc. Memory 237 may store software operated by processor 235 on computing device 200, including media application 103.

[0045] The memory 237 may include an operating system 262, other applications 264, and application data 266. The other applications 264 may include, for example, an image library application, an image management application, an image gallery application, a communication application, a web hosting engine or application, a media sharing application, etc. One or more methods disclosed herein may operate in a number of environments and platforms, such as, for example, as a stand-alone computer program that can run on any type of computing device, as a web application with a web page, as a mobile application ("app") running on a mobile computing device, etc.

[0046] Application data 266 may be data generated by other applications 264 or hardware of computing device 200. For example, application data 266 may include images used by an image library application, user actions recognized by other applications 264 (eg, a social networking application), and the like.

[0047] The I / O interface 239 may provide functionality to enable the computing device 200 to interface with other systems and devices. The interfaced device may be included as part of the computing device 200, or may be separate and in communication with the computing device 200. For example, a network communication device, a storage device (e.g., memory 237 and / or storage device 245), and an input / output device may communicate via the I / O interface 239. In some embodiments, the I / O interface 239 may be connected to an interface device, such as an input device (keyboard, pointing device, touch screen, microphone, scanner, sensor, etc.) and / or an output device (display device, speaker device, printer, monitor, etc.).

[0048] Some examples of docked devices that may be connected to I / O interface 239 may include a display 241 that may be used to display content (e.g., images, video, and / or a user interface of an output application as described herein) and to receive touch (or gesture) input from a user. For example, display 241 may be used to display a user interface including a graphical guide on a viewfinder. Display 241 may include any suitable display device, such as a liquid crystal display (LCD), a light emitting diode (LED), or a plasma display screen, a cathode ray tube (CRT), a television, a monitor, a touch screen, a three-dimensional display screen, or other visual display device. For example, display 241 may be a flat display screen provided on a mobile device, multiple display screens embedded in an eyeglass form factor or a head-mounted device, or a monitor screen for a computer device.

[0049] Camera 243 can be any type of image capture device that can capture images and / or videos. In some embodiments, camera 243 captures images or videos, and I / O interface 239 transmits the images or videos to media application 103.

[0050] The storage device 245 stores data related to the media application 103. For example, the storage device 245 may store a training data set (which includes labeled images), a machine learning model, outputs from the machine learning model, and the like.

[0051] Figure 2 An example media application 103 is illustrated as stored in memory 237 and includes a user interface module 202 , a segmenter 204 , a repairer module 206 , and a diffusion module 208 .

[0052] The user interface module 202 generates graphic data for displaying a user interface including an image. The user interface module 202 receives an initial image. The initial image may be received from the camera 243 of the computing device 200 or from the media server 101 via the I / O interface 239. The initial image includes a subject having a face, such as a person or an animal.

[0053] The user interface module includes an option for providing a text request associated with the initial image. For example, the user interface may include a text field for the user to directly enter the text request, an audio button for providing an audio input that is converted to a text request, etc. In some embodiments, when the user provides the text request, the user interface may be updated with auto-complete suggestions. For example, for an outdoor scene, when the text field includes "change to m", the user interface module 202 may add "mountains" as an auto-complete suggestion. The user interface also includes an option for receiving user input for selecting objects and areas to be modified and / or replaced.

[0054] The user interface module 202 may identify one or more objects to be replaced in the initial image. In some embodiments, the user interface module 202 performs object recognition to identify objects in the initial image and provides suggestions for replacing the objects. For example, the user interface may highlight the object and include a menu of options or a library of pre-made prompts to select from. The user interface may include a library of pre-made prompts for how to change the scene, change the clothing of people in the image, etc. For example, a user may select a pre-made prompt to change an initial image of a beach scene with a house in the background to an output image in which the house is replaced with a sand castle.

[0055] In some embodiments, the user interface includes options for selecting various people, objects, portions of people or objects, or backgrounds in the initial image to be modified or changed based on the request. For example, the user can select a person by clicking on the person, circling the object, etc. In some embodiments, the user interface generates recommendations for modifying the image, such as highlighting the boots of the person in the image and asking the user if he wants to change the boots of the person.

[0056] In some embodiments, the text request is related to replacing a first object in the initial image with a second object. For example, a user may select a first object and provide text input regarding replacing the selected object with a second object corresponding to the text request.

[0057] User interface module 202 can identify the area in the background of the initial image to be replaced or modified, and provide suggestions for replacing or modifying the background. For example, the suggestions can include a global preset, such as a list that can be selected to change the theme of the initial image.

[0058] The user interface module 202 may identify an object to be removed from the background of the initial image. For example, the object may be identified by the user interface module 202 in response to performing object recognition. The user interface may provide suggestions for removing the object from the background. In some embodiments, the user interface includes a text field for receiving a text request from a user for replacing an object based on the text request.

[0059] In some embodiments, the user interface module 202 generates graphical data for displaying the output image. The user interface may also include options for editing the output image, sharing the output image, adding the output image to an album, and the like.

[0060] Go to Figure 3A , according to some embodiments described herein, illustrates an example initial image 300. The initial image 300 includes a woman 302 as a subject, a shirt 307 worn by the woman 302, and a suitcase 304. The initial image 300 is displayed as part of a user interface (not shown) that includes a means for a user to provide user input. For example, the user interface may include a text field that receives text input, the user interface may recognize a finger or mouse (or other pointer) that selects an object by clicking on the object, circling the object, moving back and forth over the object to highlight the object, etc.

[0061] Figure 3B An example output image 310 is illustrated in which a shirt 312 is removed from a Figure 3A 307 changes in shirts, and Figure 3A The suitcase 304 in the Figure 3B In order to achieve Figure 3B The output image 310 in Figure 3A The request provided by the initial image 300 may include the science shirt 312 and the Figure 3A The suitcase 304 in is replaced by the turtle 314.

[0062] Figure 3CAn example output image 320 is illustrated in which a subject's shirt 322 and hair 324 are altered and a tattoo 326 is added to the subject's arm, according to some embodiments described herein. Figure 3C In the output image 320, the user may have Figure 3A The initial image 300 in provides a textual request to make the subject more punk rock style, a punk rock theme is selected from a library of pre-made prompts, each object in the input image 300 is individually changed, etc.

[0063] Figure 3D An example output image 330 is illustrated in which the background is changed according to some embodiments described herein. Figure 3D The output image 330 in Figure 3A The text request provided in the initial image 300 may include a request to add a background of a waterfall. In some embodiments, Figure 3D The retention mask covers the entirety of the subject (rather than, for example, just the face) because the retention mask prevents the subject from being modified when the background is replaced.

[0064] As discussed in more detail below, diffusion module 208 uses Figure 3B , 3C and a 3D preservation mask including at least a face of the subject to prevent the face of the subject from being modified during blending with the composite image.

[0065] Figure 4 Example user interfaces 400, 425, 450 including options for selecting different areas of an image to be altered, global presets to be applied, fields for providing text, and example output images are illustrated according to some embodiments described herein. Specifically, the first user interface 400 automatically provides a global preset 405 for the user to select to alter the input image 401 to look like an oil painting, a surreal world, or a nostalgic scene. The user can select an option (such as oil painting, surreal world, or nostalgic scene) from the global preset 405 to temporarily alter the input image, and can revert to the original input image by selecting the none option from the global preset 405.

[0066] The first user interface 400 also includes circles 410, 411, 412 representing identifications of different areas in the initial image 401. The user can specify changes to the sky by tapping the first circle 410, to the bridge by tapping the second circle 411, and to the person by tapping the third circle 412.

[0067] In response to the user selecting one of the circles 410, 411, 412, the user interface may update the display to provide a menu of options (not shown). For example, selecting the first circle 410 may cause the user interface to display suggestions, such as changing a cloudy sky to a clear sky. Selecting the second circle 411 may cause the user interface to display suggestions, such as an option to remove the bridge associated with the second circle 411, an option to replace the bridge with a different type of bridge or boat, etc. Selecting the third circle 412 may cause the user interface to display a suggestion to remove a person.

[0068] The second user interface 425 includes an input image 426 and a text input field 430 in which the user can specify the changes they want to make. The user can include a description that is specific enough to cover the object the user wants to change (e.g., change the boots to colorful glitter boots), or the user can select the object the user wants to change in the second user interface 425 and then describe the specific changes to make. For example, the user can select the object by tapping on the object, circling the object, drawing on the object, etc. In this case, the user selects the boots 427 on the subject.

[0069] The third user interface 450 includes an output image 451 in which the text request 452 "colorful glitter boots" is satisfied. The boots 453 are changed to glittering colorful stars. The user interface also includes an option that allows the user to undo 454 the changes made to the initial image.

[0070] Figure 5 Example user interfaces 500, 525, 550 including options for receiving user input for selecting an object to be replaced based on a text request according to some embodiments described herein are illustrated. The first user interface 500 includes an initial image 501 and suggested global presets 505. In some embodiments, the user interface module 202 performs object recognition on the initial image 501 to determine the object in the initial image 501, and provides the suggested global presets 505 based on the determined object. For example, the user interface module 202 may suggest the global presets 505 stylized, sketch, and vintage for outdoor scenes because these presets are particularly suitable for outdoor scenes.

[0071] The second user interface 525 includes an initial image 526 in which the user has circled a concrete object in the initial image 526. The second user interface 525 highlights the selected object 527 with an outline. The text input field 530 includes text in which the user has entered "rock" after the initial prompt "change to" to indicate that the user wants to replace the highlighted portion with rock. The second user interface 525 also includes suggested objects 532, such as rock, water, and shrubs. In some embodiments, the suggested objects are provided based on the user interface module 202 performing object recognition and suggesting objects that often appear with other objects that appear in the image, such as streams, rocks, and mountains.

[0072] The third user interface 550 includes an output image 551 generated by the user interface module 202 to replace the object 527 selected in the second user interface 525 with a rock as specified by the user. The third user interface 550 also includes options to save a copy of the output image 552 and undo changes to the output image 553.

[0073] Figure 6 An example user interface 600 including a menu of options 605 and a library 610 of pre-made prompts according to some embodiments described herein is illustrated. In this example, the menu of options 605 includes options for modifying the subject's clothes, the scenery in the input image 601, and free text options for providing more specific changes. The library 610 of pre-made prompts includes different themes to be applied to the input image 601, including sea adventurer, ancient warrior, space crusader, wise mage, aristocrat, and space mission.

[0074] In some embodiments, the user interface module 202 generates a user interface that includes options for modifying user preferences. For example, the user interface may include user preferences for specifying a level of stochasticity including how much noise the user wants to see in the output image (e.g., the degree to which the output image differs from the initial image) and the degree to which a reality seed is used in the output image (e.g., the degree to which the output image differs from reality). For example, where the input image is a boy drinking liquid from a mug through a straw, a spectrum of increasing stochasticity causes the output image to change from small changes (such as differences in the type of mug and a background that looks slightly different) to full changes (where the mug is different, the background is completely different, the boy's clothing is different, and the table on which the mug rests is also different). In another example, where the extent of the seed type is increased, a spectrum of increasing differences in the seed type causes the output image to change from replacing the mug with a recognizable mug and changing the background from a kitchen counter to another room in the house to the mug becoming an unrecognizable mug and changing the background to a room on a spaceship.

[0075] In some embodiments, the user interface module 202 generates a graphical user interface that includes an option for receiving user input from a user that can be converted into an image. For example, the user can sketch the outline of a dinosaur, and the diffusion module 208 can generate an output image including the dinosaur based on the sketch. In some embodiments, the user interface receives user input on an initial image. For example, the user can sketch a hat onto the initial image of a child, and the user interface module 202 updates the user interface to include an output image generated by the diffusion module 208, which includes a rendering of the sketched hat.

[0076] Segmenter 204 segments one or more objects including the face of the subject from the initial image. The face segment includes pixels corresponding to the location of the face in the initial image. Segmenter 204 segments the face of the subject to generate a preservation mask that diffusion module 208 uses to prevent modification of the face during generation of the output image. Face segmentation can be used to prevent modification of the face of the subject when changing the subject's hair, clothing, etc.

[0077] Segmenter 204 may also segment portions other than the face, such as the entire body in the case where the entire body is prevented from being modified. The body segment includes pixels corresponding to the location of the body in the original image. Body segmentation may be used to prevent modification of the entire body when the rest of the image is modified, such as changes to the background of the original image. In some embodiments, the retention mask includes all aspects of the original image except for the portions that are modified. For example, when the subject's clothing is modified, the retention mask may cover the face, hair, and background.

[0078] The segmenter 204 may segment other objects in the initial image automatically or in response to user input. For example, when the user interface module 202 generates a suggestion for modifying, removing and / or replacing an object in the initial image, the segmenter 204 segments the object. In another example, the user interface receives user input identifying an object to be modified, removed and / or replaced, and the segmenter 204 segments the object in response to the object being selected. In some embodiments, the segmenter 204 generates a segmentation map that associates an identity with each pixel in the initial image as belonging to a face, a body, an object, etc.

[0079] In some embodiments, the segmenter 204 uses the alpha map as part of a technique for distinguishing the foreground and background of the initial image during segmentation. The segmenter 204 may also identify the texture of the selected object in the foreground of the initial image.

[0080] The segmenter 204 may perform segmentation by detecting objects in the initial image. The object may be a person, an animal, a car, a building, etc. The person may be the subject of the initial image, or may not be the subject of the initial image (i.e., a bystander). Bystanders may include people walking, running, riding a bicycle, standing behind the subject, or otherwise within the initial image. In different examples, the bystander may be in the foreground (e.g., a person passing in front of the camera), at the same depth as the subject (e.g., a person standing next to the subject), or in the background. In some examples, there may be more than one bystander in the initial image. Bystanders may be people in any posture (e.g., standing, sitting, squatting, lying, jumping, etc.). Bystanders may be facing the camera, may be at an angle to the camera, or may have their backs to the camera.

[0081] The segmenter 204 may detect the type of the object by performing object recognition, comparing the object to object priors such as people, vehicles, buildings, etc., to identify the expected shape of the object, thereby determining whether the pixel is associated with the selected object or the background. The segmenter 204 may generate a region of interest for the selected object, such as a bounding box with x, y coordinates and a scale.

[0082] The segmenter 204 generates a retention mask that covers at least the face of the subject. For example, the retention mask for the face may include pixels corresponding to pixels of the face segment in the initial image. In some embodiments, the retention mask includes additional or different body parts, such as the entire head, hands, body, etc. of the subject. In some embodiments, the retention mask is generated based on: generating superpixels of the image, and matching the superpixel centroids to depth map values ​​(e.g., obtained by the camera 243 using a depth sensor or by deriving the depth from the pixel values) to cluster the detections based on depth. More specifically, the depth values ​​in the masked area can be used to determine a depth range, and superpixels that fall within the depth range can be identified. Another technique for generating a mask includes weighting the depth values ​​based on their proximity to the mask, where the weights are represented by a distance transform map.

[0083] In some embodiments, the partitioner 204 may specify a circuit configuration (e.g., for a programmable processor, for a field programmable gate array (FPGA), etc.) that enables the processor 235 to apply the machine learning model. In some embodiments, the partitioner 204 may include software instructions, hardware instructions, or a combination. In some embodiments, the partitioner 204 may provide an application programming interface (API) that may be used by the operating system 262 and / or other applications 264 to call the partitioner 204, for example, to apply the machine learning model to the application data 266 to output a retention mask.

[0084] The segmenter 204 uses the training data to generate a trained machine learning model. For example, the training data may include pairs of an initial image with one or more subjects and an output image with one or more retention masks.

[0085] The training data may be obtained from any source, such as a data repository specifically labeled for training, data for which permission has been provided for use as training data for machine learning, etc. In some embodiments, training may occur on a media server 101 that provides training data directly to a user device 115, training may occur locally on the user device 115, or a combination of both.

[0086] In some embodiments, the segmenter 204 uses weights that are obtained from another application and are not edited / transferred. For example, in these embodiments, the trained model may be generated, for example, on a different device, and provided as part of the segmenter 204. In various embodiments, the trained model may be provided as a data file that includes a model structure or form (e.g., the model structure or form defines the number and type of neural network nodes, the connectivity between nodes, and the organization of nodes into multiple layers) and associated weights. The segmenter 204 may read the data file of the trained model and implement a neural network with node connectivity, layers, and weights based on the model structure or form specified in the trained model.

[0087] The trained machine learning model may include one or more model forms or structures. For example, the model form or structure may include any type of neural network, such as a linear network, a deep learning neural network that implements multiple layers (e.g., a "hidden layer" between an input layer and an output layer, where each layer is a linear network), a convolutional neural network (e.g., a network that splits or partitions input data into multiple parts or tiles, uses one or more neural network layers to process each tile separately, and aggregates the results from the processing of each tile), a sequence-to-sequence neural network (e.g., a network that receives sequential data such as words in a sentence, frames in a video, etc. as input and produces a sequence of results as output), etc.

[0088] The model form or structure may specify the connectivity between various nodes and the organization of nodes into layers. For example, the nodes of the first layer (e.g., the input layer) may receive data as input data or application data. For example, when the trained model is used for analysis of, for example, an initial image, such data may include, for example, one or more pixels per node. Subsequent intermediate layers may receive the outputs of the nodes of the previous layers as inputs according to the connectivity specified in the model form or structure. These layers may also be referred to as hidden layers. For example, the first layer may output the segmentation between the foreground and the background. The last layer (e.g., the output layer) generates the output of the machine learning model. For example, the output layer may receive the segmentation of the initial image into the foreground and the background, and output whether the pixel is part of the retention mask. In some embodiments, the model form or structure also specifies the number and / or type of nodes in each layer.

[0089] In various embodiments, the trained model may include one or more models. One or more models in the model may include multiple nodes, which are arranged in layers according to the model structure or form. In some embodiments, the node may be a computing node without memory, which is configured to process an input unit to generate an output unit, for example. The calculation performed by the node may include, for example, multiplying each node input in multiple node inputs by a weight, thereby obtaining a weighted sum, and using a bias or intercept value to adjust the weighted sum to generate a node output. In some embodiments, the calculation performed by the node may also include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a nonlinear function. In various embodiments, such calculations may include operations such as matrix multiplication. In some embodiments, the calculations performed by multiple nodes may be performed in parallel, for example, using multiple processor cores of a multi-core processor, using individual processing units of a graphics processing unit (GPU), or a dedicated neural circuit system. In some embodiments, the node may include memory, for example, it may be able to store one or more earlier inputs and use the one or more earlier inputs to process subsequent inputs. For example, a node with memory may include a long short-term memory (LSTM) node. LSTM nodes can use memory to maintain a "state" that allows the node to act like a finite state machine (FSM).

[0090] In some embodiments, a trained model may include embeddings or weights for individual nodes. For example, a model may be initiated as a plurality of nodes organized into layers as specified by the model form or structure. Upon initialization, corresponding weights may be applied to connections between each pair of nodes connected in the model form (e.g., nodes in successive layers of a neural network). For example, corresponding weights may be randomly assigned, or initialized to default values. The model may then be trained (e.g., using training data) to produce results.

[0091] Training may include applying supervised learning techniques. In supervised learning, the training data may include multiple inputs (e.g., images, retention masks, etc.) and corresponding true-value outputs for each input (e.g., a true-value mask that correctly identifies a portion of a subject (such as the subject's face) in each image). Based on a comparison of the output of the model with the true-value output, the values ​​of the weights are automatically adjusted, for example, in a manner that increases the probability that the model will produce a true-value output for an image.

[0092] In various embodiments, the trained model includes a set of weights or embeddings corresponding to the model structure. In some embodiments, the trained model may include a fixed set of weights (e.g., downloaded from a server that provides weights). In various embodiments, the trained model includes a set of weights or embeddings corresponding to the model structure. In embodiments where data is omitted, the segmenter 204 may generate a trained model based on a priori training performed, for example, by a developer of the segmenter 204, by a third party, etc. In some embodiments, the trained model may include a fixed set of weights (e.g., downloaded from a server that provides weights).

[0093] In some embodiments, the trained machine learning model receives an initial image having one or more subjects. In some embodiments, the trained machine learning model outputs one or more retention masks corresponding to the one or more subjects. For example, the one or more retention masks may be for the faces of the one or more subjects.

[0094] For the situation of removing the object from the initial image, the repairer module 206 generates the repaired image in which the object pixels corresponding to one or more objects are replaced with background pixels. The background pixels can be based on the pixels of the reference image without the object from the same position. Alternatively, the repairer module 206 can identify the background pixels to replace the removed object based on the proximity of the background pixels to other pixels around the object. The repairer module 206 can use the gradient of the neighborhood pixels to determine the attributes of the background pixels. For example, when a bystander is standing on the ground, the repairer module 206 replaces the background pixels with the pixels of the ground. Other repair techniques are possible, including the repairer technique based on machine learning that outputs the background pixels based on the training data of the image including similar structures.

[0095] In embodiments where the user chooses to erase the selected object, the user interface module 202 may display an inpainted image in which the selected object is removed and the selected object pixels are replaced with background pixels.

[0096] The diffusion model receives as input a request (e.g., a text request provided directly by the user, a selection of a pre-made prompt, a selection of a global preset, a selection of an option from a menu, etc.), an initial image, and a retention mask. The diffusion model encodes the image in latent space, performs diffusion, and decodes it back to pixel space.

[0097] The diffusion module 208 performs text conditioning on the request. Text conditioning describes the process of generating an image that is conditioned on (e.g., aligned with) a text prompt. For example, if the text request is for replacing a red shirt that the subject is wearing in the initial image with a blue shirt, the diffusion module 208 performs text conditioning by generating an output image of a blue shirt.

[0098] In some embodiments, the diffusion module 208 uses two types of training data to train the diffusion model. The first type of training data includes image pairs, where the pairs may include synthetic pairs generated by a prompt-to-prompt generative machine learning model. The prompt-to-prompt generative machine learning model is a diffusion model that receives a text prompt and uses self-attention to extract keys and values ​​from the text prompt and transforms portions of an attention map previously generated for an input image based on the input text prompt to output an output image that matches the text prompt.

[0099] The prompt-to-prompt generative machine learning model generates a self-attention map. Self-attention computes the interactions between different elements of the input sequence (e.g., different words in a text request). This is in contrast to cross-attention, where the interactions are between two different input sequences (e.g., how the text request relates to the original prompt).

[0100] The self-attention map describes the structure and different semantic regions in the image. For example, in the self-attention map, an image described as "pepperoni pizza next to orange juice" includes how the pixels on the pizza crust pay attention to other pixels on the crust. Conversely, in the cross-attention map, the pixels on the pizza crust pay attention to the orange juice.

[0101] Go to Figure 7 , according to some embodiments described herein, illustrates an example architecture 700 for generating a self-attention map and using the self-attention map during diffusion. Figure 7 Two ways to describe the process are included: text-to-image self-attention 705 and self-attention control 740.

[0102] During the text-to-image self-attention 705, the diffusion module 208 generates an attention map by extracting pixel features 710 from the noisy input image. The pixel features 710 from the noisy input image are projected to a pixel query matrix 715. The text embeddings are projected to a key matrix, which is in the form of word-meta keys from the initial image prompt 720. The initial image prompt may be "A cat with a hat lying on a beach chair".

[0103] The pixel query 715 is multiplied with the word-gram key from the initial image hint 720 to produce a self-attention map 725 for different layers. The different layers are used to focus on different features in the input image at different levels of abstraction. The self-attention map 725 contains rich semantic relationships that have a crucial impact on the generated image. In the self-attention map 725, each cell defines the weight of the value of a specific word-gram at a specific pixel based on the potential projection dimension of the word-gram key from the initial image hint 720 and the pixel query 715.

[0104] The self-attention map 725 is used to create an output image with modifications to the initial image. Continuing with the above example, the initial image prompt can be modified to "A cat with a three-musketeers hat lying on a beach chair". The diffusion model creates word-unit values ​​based on the request prompt 730. The self-attention map 725 is multiplied with the word-unit values ​​from the request prompt 730 to produce the self-attention output 735, where the weight is the attention map that is related to the similarity between the pixel query 714 and the word-unit key 720 from the initial image prompt. The self-attention output 735 is used to create the output image.

[0105] The self-attention control 740 illustrates how to modify the self-attention map based on the difference between the initial image prompt and the request prompt. For example, "A cat on a chair" can be replaced with "A dog on a chair". The main challenge is to preserve the original composition while resolving the content of the new prompt. In the case of replacing the words in the initial image prompt, the diffusion model performs a word swap 745 from the self-attention map 725 to the modified self-attention map 750. The modified self-attention map 750 replaces the word-gram for "cat" with the word-gram for "dog". If a different number of word-grams are used to replace the words, the diffusion model can copy the self-attention map 725 or average the self-attention map 725 to obtain the modified self-attention map 750.

[0106] Additional words may be added to the initial image prompt. For example, "A cat on a chair" may be replaced with "A black cat on a white chair". The diffusion model performs prompt refinement 750 by adding self-attention maps for new words and creating a refined self-attention map 755 that incorporates the new self-attention map for the new words. To preserve common details, the diffusion model applies attention injection to common word-grams from two prompts (e.g., a cat on a chair). In some embodiments, the diffusion model uses an alignment function that receives a word-gram index from a target prompt and outputs a corresponding word-gram index.

[0107] The self-attention map is used in a text-conditional diffusion model to use the structure and different semantic regions in the input image to change one or more word-meta values ​​while fixing the self-attention map to preserve scene composition. In some embodiments, the diffusion model adds new words to the prompt and freezes the attention on the previous word-meta while allowing new attention to flow to the new word-meta. This results in a global edit or modification of specific objects in the input image to match the text request.

[0108] Each diffusion step predicts noise and text embeddings from the noisy image. At the final step, the process produces a generated image. The interaction between the textual cues and the image occurs during the noise prediction, where the embeddings of visual and textual features are fused using a self-attention layer that produces a spatial attention map for each text token.

[0109] The second type of training data includes pairs with real images and synthetic images. The real images are received by a diffusion model, such as a denoising diffusion implicit model (DDIM). The diffusion model uses an inversion method to output a synthetic image based on the real image and instructions on how to edit the input image. The diffusion module 208 trains the diffusion model to generate an output image upon request using a forward process in which the diffusion model adds noise to the data and a reverse process in which the diffusion model learns to recover the data from the noise.

[0110] Figure 8 An example transformation 800 including inversion 805 and a text-conditioned diffusion process 815 is illustrated according to some embodiments described herein. Inversion 805 starts with an input image 810 with a text prompt "A cat with a hat is lying on a beachchair". Inversion gradually adds noise to the input image 810.

[0111] The text-conditioned diffusion process 815 conditions the image based on the text request. In this example, "musketeers" is added to the text request associated with the input image 810, so that the text request is for an image including "A cat with a musketeers hat is lying on a beach chair". The diffusion model combines the text condition of the text request with the noisy image in the forward diffusion.

[0112] The diffusion module 208 trains the diffusion model to maintain photo-realism and preserve the identity of the person shown in the image. During training, the diffusion model receives editing instructions and modifies the editing instructions based on a language model, such as a large language model, to create corresponding prompts. For example, the diffusion module 208 uses the language model to convert the editing instruction "make person look like an astronaut" into prompts describing various aspects of what the clothing of the space suit will look like.

[0113] The diffusion model creates a set of input and output image pairs from the generated prompt pairs, where each prompt can generate N images (using different seeds). The diffusion module 208 filters out certain images from the image pairs, such as image transformations that do not match a given edit instruction, image transformations that do not produce well-aligned images, and unmatched pairs. In some embodiments, the diffusion module 208 also filters images based on an edit alignment score that reflects the alignment between the image-to-image transformation and the original edit instruction text, and an image-text alignment score that reflects the alignment between the input / output image and the corresponding input / output prompt. In some embodiments, the diffusion module 208 trains the diffusion model by generating one or more loss functions based on the images filtered out from the image pairs.

[0114] The diffusion model is trained to generate images by gradually adding noise to the image, and then learns how to gradually remove the noise. The diffusion model applies the denoising process to random seeds to generate realistic images. By simulating diffusion, the diffusion model generates one or more noisy images.

[0115] Once the diffusion model is trained, the diffusion model receives an input image and performs an inverse diffusion process on the initial image to generate a noisy image based on the initial image. In some embodiments, the diffusion module 208 performs inverse diffusion using DDIM inversion.

[0116] The diffusion model provides a noisy image to a first CNN with features and self-attention mechanism. The first CNN samples the input image and extracts features from the input image. The first CNN injects the extracted features and self-attention map directly into the second CNN. The first CNN performs forward diffusion of the noisy initial image, which is a process of progressively denoising the noisy image using sampling to output a denoised initial image.

[0117] The text request and the noisy image are provided as input to the second CNN. The second CNN uses a self-attention map to align the semantic features of the text request with the structure of the noisy image to generate a noisy transformed image. The second CNN performs forward diffusion of the noisy transformed image to output a denoised transformed image.

[0118] The denoised original image is combined with the denoised transformed image and the retained mask. This advantageously prevents modification of the face that might otherwise be modified in a way that produces unrealistic features. In some embodiments, the diffusion module 208 performs blending by using a mask smoothing algorithm and Poisson blending.

[0119] In some embodiments, the preservation mask includes other parts of the subject, such as the subject's hair (in cases where the user wants their hair to remain unchanged), the subject's fingers (because fingers are often modified in unrealistic ways by machine learning models), the subject's entire body (where the subject is a pet) to prevent the pet from being overly modified, etc. In some embodiments where the output image modifies the subject's clothing, the preservation mask may include everything except the subject's clothing, so that the body (minus the clothing) and the background of the original image are preserved.

[0120] The combined denoised image and retained mask are blended with the denoised transformed image to form an output image that satisfies the text request.

[0121] Go to Fig. 9 , illustrates a block diagram of an example architecture 900 for generating an output image incorporating a text request. An initial image 905 is provided to a diffusion model, where a denoised diffusion implicit model (DDIM) inversion 910 is performed on the input image 705 to output a noisy image 915 obtained by inverting the input image. During DDIM inversion, features are extracted from the input image.

[0122] The noisy image 915 is provided as input to a first CNN 920 with a self-attention mechanism. The CNN utilizes an aggregation function over a local receptive field based on the convolution filter weights shared by the feature map. The self-attention map applies a weighted average operation based on the context of the input features, where the attention weights are dynamically calculated using a similarity function between related pixel pairs. The example architecture 900 uses a combination of two types of feature extraction to output a denoised initial image. The first CNN 920 outputs a denoised initial image 925.

[0123] Text conditioning is performed on the text input 935, specifically, "Add a hat to the photo" is used to predict the category that best matches the output image corresponding to the text input 935. The text conditioning is used to generate a noisy text-guided converted image 940, which is provided to the second CNN 945. The second CNN 945 receives the extracted features and self-attention map from the first CNN 920, and uses the extracted features and self-attention map to align the noisy text-guided converted image 940 with the structure of the initial image 905.

[0124] The second CNN 945 outputs a denoised converted image, which is blended with the denoised initial image 925 combined with the retention mask 930 to form an output image 950 that satisfies the text request of the subject wearing a hat. In this example, the retention mask 930 covers the face to prevent modification of the face from the initial image 905 during blending. At each blending step, the initial latent image within the mask is used instead of the denoised converted image to preserve the original features.

[0125] Example Method

[0126] Fig.10 An example flow chart of a method 1000 for generating an output image is illustrated. The method 1000 may be performed by Figure 2 In some embodiments, the method 1000 is performed by the user device 115, the media server 101, or partially performed on the user device 115 and partially performed on the media server 101.

[0127] Fig.10Method 1000 may begin at block 1002. At block 1002, an initial image and a text request to change the initial image are received, wherein the initial image includes a subject having a face. The text request may include a request to change the subject of the initial image, the attributes of the background of the initial image, and / or a selection of a first object, and a request to change the first object of the initial image to a second object. In some embodiments, the request is at least one selection from a set of global presets, a menu of options, and / or a library of pre-made prompts.

[0128] The method may further include: receiving a selection of a first object in the initial image, wherein the text request includes an annotation to replace the first object in the initial image with a second object. The method may further include: identifying an area of ​​a background from the initial image to be replaced or modified; and providing a suggestion to replace or modify the background, wherein the text request is associated with the suggestion. The text request may include a request to change the background of the initial object.

[0129] In addition to the text request, the method may further include: identifying an object to be removed from the background of the initial image; and providing a suggestion to remove the object from the background. In some embodiments, the method includes: identifying one or more objects to be replaced from the initial image; and providing a suggestion to replace the object. Block 1002 may be followed by block 1004.

[0130] At block 1004, a retention mask corresponding to at least the face of the subject is generated. Segmenter 204 may also be applied to other portions of the initial image, depending on where the modification is intended. For example, in instances where the text request includes a request to change the background of the initial image, the retention mask includes one or more portions of the subject in addition to the face of the subject. Block 1004 may be followed by block 1006.

[0131] At block 1006 , the text request, the initial image, and the retention mask are provided as inputs to a diffusion model. Block 1006 may be followed by block 1008 .

[0132] At block 1008 , the diffusion model performs inverse diffusion on the initial image to generate a noisy initial image based on the initial image. Block 1008 may be followed by block 1010 .

[0133] At block 1010 , a noisy initial image is provided to a first CNN and a denoised initial image is output. Block 1010 may be followed by block 1012 .

[0134] At block 1012 , the diffusion model performs text conditioning on the text request and forward diffusion to generate a noisy translation image that satisfies the text request. Block 1012 may be followed by block 1014 .

[0135] At block 1014 , the noisy converted image is provided to a second CNN and a denoised converted image is output, wherein the second CNN injects the extracted features and the self-attention map to output a denoised converted image. Block 1014 may be followed by block 1016 .

[0136] At block 1016, the denoised original image, the preservation mask, and the denoised converted image are blended to form an output image, wherein the preservation mask prevents modification of faces from the original image.

[0137] Fig.11 An example flow chart of a method 1100 for generating an output image is illustrated. The method 1100 may be performed by Figure 2 In some embodiments, the method 1100 is performed by the user device 115, the media server 101, or partially performed on the user device 115 and partially performed on the media server 101.

[0138] Fig.11 Method 1100 may begin at block 1102. At block 1102, an initial image and a text request to modify the initial image are received, wherein the initial image includes a subject having a face. Block 1102 may be followed by block 1104.

[0139] At block 1104, a retention mask corresponding to the subject's face is generated. Block 1104 may be followed by block 1106.

[0140] At block 1106 , the text request, the initial image, and the retention mask are provided as inputs to a diffusion model. Block 1106 may be followed by block 1108 .

[0141] At block 1108 , the diffusion model outputs a denoised initial image based on the initial image. Block 1108 may be followed by block 1110 .

[0142] At block 1110 , the diffusion model performs text conditioning on the text request and forward diffusion to generate a noisy translated image that satisfies the text request. Block 1110 may be followed by block 1112 .

[0143] At block 1112 , the diffusion model outputs a denoised converted image based on the noisy converted image, the extracted features, and the self-attention map. Block 1112 may be followed by block 1114 .

[0144] At block 1114, the denoised original image, the preservation mask, and the denoised converted image are blended to form an output image, wherein the preservation mask prevents modification of faces from the original image.

[0145] In addition to the above, controls may be provided to the user that allow the user to make choices about whether and when the systems, programs, or features described herein may enable the collection of user information (e.g., information about the user's social network, social actions, or activities, occupation, the user's preferences, or the user's current location) and whether to send content or communications from a server to the user. In addition, before certain data is stored or used, it may be processed in one or more ways to remove personally identifiable information. For example, the user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location for which location information is obtained may be generalized (such as to a city, zip code, or state level) so that the user's specific location cannot be determined. Thus, the user can control what information is collected about the user, how the information is used, and what information is provided to the user.

[0146] In the above description, many specific details are set forth for the purpose of explanation in order to provide a thorough understanding of the present specification. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In some instances, structures and devices are shown in block diagram form to avoid obscuring the present description. For example, the above may describe embodiments primarily with reference to user interfaces and specific hardware. However, embodiments may be applicable to any type of computing device that receives data and commands, and any peripheral device that provides services.

[0147] References in the specification to "some embodiments" or "some examples" mean that a particular feature, structure, or characteristic described in conjunction with the embodiments or examples may be included in at least one implementation of the specification. The appearance of the phrase "in some embodiments" in various places in the specification does not necessarily all refer to the same embodiment.

[0148] Some portions of the above detailed description are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the art of data processing to most effectively convey the substance of their work to others skilled in the art. An algorithm is here and generally considered to be a self-consistent sequence of steps that produces a desired result. The steps are those that require physical manipulation of physical quantities. Typically, although not necessarily, these quantities are typically in the form of electrical or magnetic data that can be stored, transferred, combined, compared, and otherwise manipulated. Mainly for general reasons, it has sometimes proven convenient to refer to these data as bits, values, elements, symbols, characters, terms, numbers, etc.

[0149] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically noted otherwise, as will be apparent from the following discussion, it should be understood that throughout this specification, discussions utilizing terms including "processing" or "computing" or "calculating" or "determining" or "displaying" and the like refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities within the computer system's registers and memories and transforms that data into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission or display devices.

[0150] The embodiments of the specification may also relate to a processor for performing one or more steps of the above method. The processor may be a special-purpose processor selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory computer-readable storage medium, including but not limited to any type of disk, including an optical disk, ROM, CD-ROM, magnetic disk, RAM, EPROM, EEPROM, magnetic card or optical card, flash memory (including USB disk with non-volatile memory) or any type of medium suitable for storing electronic instructions, each of which is coupled to a computer system bus.

[0151] The specification may take the form of some complete hardware embodiments, some complete software embodiments, or some embodiments containing both hardware elements and software elements. In some embodiments, the specification is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0152] Furthermore, the present description may take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of the present description, a computer-usable or computer-readable medium may be a device that can contain, store, communicate, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or device.

[0153] A data processing system suitable for storing or executing program code will include at least one processor coupled directly or indirectly to a memory element via a system bus. The memory element may include local memory, bulk storage, and cache memory used during the actual execution of the program code, which provides temporary storage of at least some program code to reduce the number of times that code must be retrieved from bulk storage during execution.

Claims

1. A computer-implemented method comprising: receiving an initial image and a text request to modify the initial image, the initial image including a subject having a face; generating a retention mask corresponding to the face of the subject; providing the text request, the initial image, and the retention mask as input to a diffusion model; outputting a denoised initial image based on the initial image using the diffusion model; performing text conditioning and forward diffusion on the text request using the diffusion model to generate a noisy transformed image satisfying the text request; outputting a denoised transformed image based on the noisy transformed image, the extracted features and the self-attention map using the diffusion model; as well as The denoised initial image, the retention mask, and the denoised converted image are blended to form an output image, wherein the retention mask prevents modification of the face from the initial image.

2. The method of claim 1, wherein: Outputting the denoised initial image includes: performing inverse diffusion on the initial image using the diffusion model to generate a noisy initial image based on the initial image; and Providing the noisy initial image to a first convolutional neural network (CNN) and outputting the denoised initial image; and Outputting the denoised transformed image includes: Providing the noisy transformed image to a second CNN; injecting the extracted features and the self-attention map during diffusion; and The denoised transformed image is output.

3. The method of claim 2, wherein the inverse diffusion is a denoised diffusion implicit model (DDIM) inversion.

4. The method of claim 1, further comprising: A selection of a first object in the initial image is received, wherein the text request includes an annotation to replace the first object in the initial image with a second object.

5. The method of claim 1, further comprising: identifying, from the initial image, an area of ​​the background to be replaced or modified; as well as A suggestion is provided to replace or modify the background, wherein the text request is associated with the suggestion.

6. The method of claim 1, further comprising: identifying objects to be removed from the background in the initial image; as well as A suggestion is provided to remove the object from the background.

7. The method of claim 1, further comprising: identifying one or more objects to be replaced from the initial image; as well as Suggestions are provided for replacing the object.

8. The method of claim 1, wherein: The textual request is to change the background of the initial image; and The retention mask further includes one or more portions of the subject other than the face of the subject.

9. The method of claim 1, wherein the text request further comprises selecting from at least one of a set of global presets, a menu of options, a library of pre-made prompts, and combinations thereof.

10. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform operations comprising: receiving an initial image and a text request to modify the initial image, the initial image including a subject having a face; generating a retention mask corresponding to the face of the subject; providing the text request, the initial image, and the retention mask as input to a diffusion model; outputting a denoised initial image based on the initial image using the diffusion model; performing text conditioning and forward diffusion on the text request using the diffusion model to generate a noisy transformed image satisfying the text request; outputting a denoised transformed image based on the noisy transformed image, the extracted features and the self-attention map using the diffusion model; as well as The denoised initial image, the retention mask, and the denoised converted image are blended to form an output image, wherein the retention mask prevents modification of the face from the initial image.

11. The non-transitory computer readable medium of claim 10, wherein the operations further comprise: Outputting the denoised initial image includes: performing inverse diffusion on the initial image using the diffusion model to generate a noisy initial image based on the initial image; and Providing the noisy initial image to a first convolutional neural network (CNN) and outputting the denoised initial image; and Outputting the denoised transformed image includes: Providing the noisy transformed image to a second CNN; injecting the extracted features and the self-attention map during diffusion; and The denoised transformed image is output.

12. The non-transitory computer readable medium of claim 11, wherein the inverse diffusion is a denoised diffusion implicit model (DDIM) inversion.

13. The non-transitory computer readable medium of claim 10, wherein the operations further comprise: A selection of a first object in the initial image is received, wherein the text request includes an annotation to replace the first object in the initial image with a second object.

14. The non-transitory computer readable medium of claim 10, wherein the operations further comprise: identifying, from the initial image, an area of ​​the background to be replaced or modified; as well as A suggestion is provided to replace or modify the background, wherein the text request is associated with the suggestion.

15. The non-transitory computer readable medium of claim 10, wherein the operations further comprise: identifying, from the initial image, an area of ​​the background to be replaced or modified; as well as A suggestion is provided to replace or modify the background, wherein the text request is associated with the suggestion.

16. A system comprising: processor; as well as A memory coupled to the processor, the memory storing instructions, the instructions causing the processor to perform operations when executed by the processor, the operations comprising: receiving an initial image and a text request to modify the initial image, the initial image including a subject having a face; generating a retention mask corresponding to the face of the subject; providing the text request, the initial image, and the retention mask as input to a diffusion model; outputting a denoised initial image based on the initial image using the diffusion model; performing text conditioning and forward diffusion on the text request using the diffusion model to generate a noisy transformed image satisfying the text request; outputting a denoised transformed image based on the noisy transformed image, the extracted features and the self-attention map using the diffusion model; and The denoised initial image, the retention mask, and the denoised converted image are blended to form an output image, wherein the retention mask prevents modification of the face from the initial image.

17. The system of claim 16, wherein the operations further comprise: Outputting the denoised initial image includes: performing inverse diffusion on the initial image using the diffusion model to generate a noisy initial image based on the initial image; and Providing the noisy initial image to a first convolutional neural network (CNN) and outputting the denoised initial image; and Outputting the denoised transformed image includes: Providing the noisy transformed image to a second CNN; injecting the extracted features and the self-attention map during diffusion; and The denoised transformed image is output.

18. The system of claim 17, wherein the inverse diffusion is a denoised diffusion implicit model (DDIM) inversion.

19. The system of claim 16, wherein the operations further comprise: A selection of a first object in the initial image is received, wherein the text request includes an annotation to replace the first object in the initial image with a second object.

20. The system of claim 16, wherein the operations further comprise: identifying, from the initial image, an area of ​​the background to be replaced or modified; as well as A suggestion is provided to replace or modify the background, wherein the text request is associated with the suggestion.

Images