Re-illumination of outdoor images using machine learning

Abstract

A media application provides, as inputs to a diffusion model, an initial image and a request to change illumination in the initial image, wherein the initial image includes a subject and the sky. The media application outputs an output image that satisfies the request using the diffusion model. The media application determines a sky section and a body section from the initial image. The media application generates a sky mask corresponding to the sky section and a body mask corresponding to the body section. The media application modifies the shading of the initial image to match the shading of the output image. The media application mixes the modified initial image with the output image to form a mixed image while using the subject mask to prevent modification of the subject from the modified initial image and using the sky mask to prevent modification of the sky from the output image during mixing.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 465,224, entitled “Relighting of Outdoor Images Using Machine Learning,” filed on May 9, 2023, the entire contents of which are incorporated herein. Background Art

[0003] The machine learning model can generate outdoor scenes; however, the scenes are often not realistic enough. For example, the machine learning model can generate nighttime scenes with shadows. In addition, the machine learning model can generate outdoor scenes with people that are not realistic enough, where the more detailed aspects of the people may not be properly represented. For example, the people may look like they were photographed indoors, while the background matches the sunset.

[0004] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the presently named inventors (to the extent that it is described in this background section) and aspects of the specification that might not have been considered prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art to the present disclosure. Summary of the invention

[0005] A computer-implemented method includes providing an initial image and a request to change lighting in the initial image as input to a diffusion model, wherein the initial image includes a subject and a sky. The method also includes outputting an output image that satisfies the request using the diffusion model. The method also includes determining a sky segment and a subject segment from the initial image. The method also includes generating a sky mask corresponding to the sky segment and a subject mask corresponding to the subject segment. The method also includes modifying the shading of the initial image to match the shading of the output image. The method also includes blending the modified initial image with the output image to form a blended image, while during blending, using the subject mask to prevent modification of the subject from the modified initial image, and using the sky mask to prevent modification of the sky from the output image.

[0006] In some embodiments, modifying the colorization of the initial image includes: performing bilateral grid upsampling (BGU) that identifies a local color transform between the initial image and the output image, and applying the local color transform to the initial image while using a sky mask to prevent colorization of the sky. In some embodiments, the method also includes: generating a super-resolution version of at least a portion of the output image from the output image, wherein blending the modified initial image with the output image includes blending the super-resolution version of at least the portion of the output image while, during blending, using a subject mask to prevent modification of a subject from the modified initial image, and using a sky mask to prevent modification of a sky from the super-resolution version of at least the portion of the output image. In some embodiments, the output image includes one or more shadows corresponding to one or more objects in the output image, and the method also includes: determining from the output image shadow segments corresponding to the one or more shadows in the output image, and generating a shadow mask corresponding to the shadow segments, wherein blending the output image with the modified initial image includes: during blending, using the shadow mask to prevent modification of the one or more shadows from the output image.

[0007] In some embodiments, the request to change the lighting includes a user providing a text request including an attribute selected from the group consisting of: light level, amount of clouds in the sky, color of the sky, and combinations thereof. In some embodiments, the request to change the lighting is selected from the group consisting of: region suggestions associated with one or more regions of the initial image, a global preset, an options menu, a library of pre-made text requests, and combinations thereof. In some embodiments, the method further includes determining that the initial image includes an outdoor scene, and providing a suggestion to the user to modify the lighting.

[0008] A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations. The operations include: providing an initial image and a request to change lighting in the initial image as input to a diffusion model, wherein the initial image includes a subject and a sky; outputting an output image that satisfies the request using the diffusion model; determining a sky segment and a subject segment from the initial image; generating a sky mask corresponding to the sky segment and a subject mask corresponding to the subject segment; modifying the shading of the initial image to match the shading of the output image; and blending the modified initial image with the output image to form a blended image, while during blending, using the subject mask to prevent modification of the subject from the modified initial image and using the sky mask to prevent modification of the sky from the output image.

[0009] In some embodiments, modifying the colorization of the initial image includes: performing a BGU that identifies a local color transform between the initial image and the output image, and applying the local color transform to the initial image while using a sky mask to prevent colorization of the sky. In some embodiments, the operation also includes: generating a super-resolution version of at least a portion of the output image from the output image, wherein blending the modified initial image with the output image also includes blending the modified initial image with the output image includes blending the super-resolution version of at least the portion of the output image while, during blending, using a subject mask to prevent modification of a subject from the modified initial image, and using a sky mask to prevent modification of a sky from the super-resolution version of at least the portion of the output image. In some embodiments, the output image includes one or more shadows corresponding to one or more objects in the output image, and the operation also includes: determining from the output image shadow segments corresponding to the one or more shadows in the output image, and generating a shadow mask corresponding to the shadow segments, wherein blending the output image with the modified initial image includes: during blending, using the shadow mask to prevent modification of the one or more shadows from the output image. In some embodiments, the request to change the lighting includes a user providing a text request including an attribute selected from the group consisting of: light level, amount of clouds in the sky, color of the sky, and combinations thereof. In some embodiments, the request to change the lighting is selected from the group consisting of: region suggestions associated with one or more regions of the initial image, a global preset, an options menu, a library of pre-made text requests, and combinations thereof. In some embodiments, the operations further include determining that the initial image includes an outdoor scene, and providing a suggestion to the user to modify the lighting.

[0010] A system includes a processor and a memory coupled to the processor, the memory having instructions stored thereon, which when executed by the processor cause the processor to perform operations. The operations include: providing an initial image and a request to change lighting in the initial image as input to a diffusion model, wherein the initial image includes a subject and a sky; outputting an output image satisfying the request using the diffusion model; determining a sky segment and a subject segment from the initial image; generating a sky mask corresponding to the sky segment and a subject mask corresponding to the subject segment; modifying the shading of the initial image to match the shading of the output image; and blending the modified initial image with the output image to form a blended image, while during blending, using the subject mask to prevent modification of the subject from the modified initial image and using the sky mask to prevent modification of the sky from the output image.

[0011] In some embodiments, modifying the colorization of the initial image includes: performing a BGU that identifies a local color transform between the initial image and the output image, and applying the local color transform to the initial image while using a sky mask to prevent colorization of the sky. In some embodiments, the operation also includes: generating a super-resolution version of at least a portion of the output image from the output image, wherein blending the modified initial image with the output image includes blending the super-resolution version of at least the portion of the output image while, during blending, using a subject mask to prevent modification of a subject from the modified initial image, and using a sky mask to prevent modification of a sky from the super-resolution version of at least the portion of the output image. In some embodiments, the output image includes one or more shadows corresponding to one or more objects in the output image, and the operation also includes: determining a shadow segment corresponding to the one or more shadows in the output image from the output image, and generating a shadow mask corresponding to the shadow segment, wherein blending the output image with the modified initial image includes: during blending, using the shadow mask to prevent modification of the one or more shadows from the output image. In some embodiments, the request to change the lighting includes a user providing a text request including an attribute selected from the group consisting of: light level, amount of clouds in the sky, color of the sky, and combinations thereof. In some embodiments, the request to change the lighting is selected from the group consisting of: region suggestions associated with one or more regions of the initial image, a global preset, an options menu, a library of pre-made text requests, and combinations thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram of an example network environment according to some embodiments described herein.

[0013] Figure 2 is a block diagram of an example computing device according to some embodiments described herein.

[0014] Figure 3 An example user interface is shown including options for selecting different aspects of an image to change, fields for providing text, and an example output image according to some embodiments described herein.

[0015] FIG. 4A to FIG. 4B An example user interface is shown that includes options for selecting different types of modifications for changing lighting in an image according to some embodiments described herein.

[0016] Figure 5 is a block diagram of an example architecture for generating a blended image incorporating a text request according to some embodiments described herein.

[0017] Figure 6An example flow chart of a method for generating a blended image with modified lighting according to some embodiments described herein is shown. DETAILED DESCRIPTION

[0018] The machine learning model can generate outdoor scenes; however, the scenes are often not realistic enough. For example, the machine learning model can generate nighttime scenes with shadows. In addition, the machine learning model can generate outdoor scenes with people that are not realistic enough, where the more detailed aspects of the people may not be properly represented. For example, the people may look like they were photographed indoors, while the background matches the sunset.

[0019] The technology described below describes a media application that advantageously modifies the lighting of an input image by utilizing a diffusion model to output an output image that satisfies a request to modify the lighting. For example, a user may provide a text request to change an image of a person captured outdoors on a sunny day to an image of the person under a moonlit sky.

[0020] The media application generates an output image. For example, the media application may use a diffusion model to generate a synthetic moonlit sky. The media application determines a sky segment and a subject segment from the initial image. The media application generates a subject mask corresponding to the subject segment and a sky mask corresponding to the sky segment.

[0021] The media application modifies the shading of the initial image to match the shading of the output image. The shading ensures that the shading of the subject matches the changes to the output image. For example, replacing a sunny image with a moonlit image causes the color projected on a person to change from including all colors to primarily including shades of blue, purple, or black. The media application may modify the shading of the initial image by performing bilateral grid upsampling (BGU) to identify local color transformations between the initial image and the output image. The media application may also generate a super-resolution version of at least a portion of the output image (e.g., a synthesized sky portion of the output image) from the output image. The super-resolution version of the output image advantageously extracts more details from the lower resolution output image to improve the quality of the output image.

[0022] The media application blends a modified initial image (i.e., the initial image modified to match the shading of the output image) with a super-resolution version of at least the portion of the output image to form a blended image, while during blending, using a subject mask to prevent modification of subjects from in the modified image, and using a sky mask to prevent modification of the sky from the super-resolution version.

[0023] Sample Environment 100

[0024] Figure 11 shows a block diagram of an example environment 100. In some embodiments, the environment 100 includes a media server 101 coupled to a network 105, a user device 115a, and a user device 115n. Users 125a, 125n can be associated with respective user devices 115a, 115n. In some embodiments, the environment 100 can include Figure 1 Other servers or devices not shown in FIG. Figure 1 In the illustrative drawings and the remaining drawings, a letter following a reference number (e.g., "115a") indicates a reference to the element having that specific reference number. A reference number in the text without a following letter (e.g., "115") indicates a general reference to an embodiment of the element with that reference number.

[0025] The media server 101 may include a processor, a memory, and network communication hardware. In some embodiments, the media server 101 is a hardware server. The media server 101 is communicatively coupled to the network 105 via a signal line 102. The signal line 102 may be a wired connection (such as Ethernet, coaxial cable, fiber optic cable, etc.) or a wireless connection (such as Wi-Fi®, Bluetooth® or other wireless technologies). In some embodiments, the media server 101 sends data to one or more user devices of the user devices 115a, 115n via the network 105 and receives data from the one or more user devices. The media server 101 may include a media application 103a and a database 199.

[0026] The database 199 may store machine learning models, training data sets, images, etc. The database 199 may also store social network data associated with the user 125, user preferences of the user 125, etc.

[0027] User device 115 may be a computing device including a memory coupled to a hardware processor. For example, user device 115 may include a mobile device, a tablet computer, a mobile phone, a wearable device, a head mounted display, a mobile email device, a portable game player, a portable music player, a reader device, or another electronic device capable of accessing network 105.

[0028] In the illustrated implementation, user device 115a is coupled to network 105 via signal line 108, and user device 115n is coupled to network 105 via signal line 110. Media application 103 may be stored on user device 115a as media application 103b and / or stored on user device 115n as media application 103c. Signal lines 108 and 110 may be wired connections (such as Ethernet, coaxial cable, fiber optic cable, etc.) or wireless connections (such as Wi-Fi®, Bluetooth® or other wireless technologies). User devices 115a, 115n are accessed by users 125a, 125n, respectively. By way of example, Figure 1 Although Figure 1 Two user devices 115 a and 115 n are shown, but the present disclosure is applicable to system architectures having one or more user devices 115 .

[0029] The media application 103 may be stored on the media server 101 or the user device 115. In some embodiments, the operations described herein are performed on the media server 101 or the user device 115. In some embodiments, some operations may be performed on the media server 101 and some operations may be performed on the user device 115. The execution of the operations is based on user settings. For example, the user 125a may specify a setting that an operation is to be performed on its corresponding device 115a instead of on the media server 101. With such settings, the operations described herein are performed entirely on the user device 115a, and no operations are performed on the media server 101. In addition, the user 125a may specify that the user's images and / or other data will only be stored locally on the user device 115a and not on the media server 101. With such settings, no user data is transmitted to or stored on the media server 101. The transmission of user data to the media server 101, any temporary or permanent storage of such data by the media server 101, and the performance of operations on such data by the media server 101 are only performed if the user has consented to the transmission, storage and performance of operations by the media server 101. The user is provided with the option to change the settings at any time, for example so that they can enable or disable the use of the media server 101.

[0030] If a machine learning model (e.g., a neural network or other type of model) is used for one or more operations, it is stored and utilized locally on the user device 115 with specific user permission. Server-side models are used only with user permission. In addition, trained models can be provided for use on the user device 115. During such use, if the user 125 permits, on-device training of the model can be performed. If the user 125 permits, updated model parameters can be transmitted to the media server 101, for example, to enable federated learning. The model parameters do not include any user data.

[0031] The media application 103 receives a request to change the lighting in the initial image. The request may include a text request, a selection of a suggestion, a selection of a preset, a selection of an option from a library of pre-made text requests, etc. The initial image includes a subject.

[0032] The media application 103 provides an initial image and a request as input to a diffusion model. The diffusion model outputs an output image that satisfies the request by including the features described in the request. For example, if the request requires that the input image be changed from a rainy day image with a cloudy sky to a clear sky at sunset, the output image includes a clear sky at sunset. Therefore, the output image corresponds to the initial image and represents a modified initial image, wherein the modification of the initial image is performed in response to the request, i.e., in response to the change requested in the request. Specifically, the output image is the initial image modified by implementing the change indicated in the request (e.g., a change in the lighting in the initial image).

[0033] The media application 103 determines a sky segment and a subject segment from the initial image. The media application 103 generates a sky mask corresponding to the sky segment and a subject mask corresponding to the subject segment. The media application 103 modifies the shading of the initial image to match the shading of the output image. The media application 103 blends the modified initial image with the output image to form a blended image, while during blending, using the subject mask to prevent modification of the subject from the modified initial image, and using the sky mask to prevent modification of the sky from the output image.

[0034] In some embodiments, media application 103 can be implemented using hardware, including a central processing unit (CPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a machine learning processor / coprocessor, any other type of processor, or a combination thereof. In some embodiments, media application 103a can be implemented using a combination of hardware and software.

[0035] Example computing device 200

[0036] Figure 2 is a block diagram of an example computing device 200 that can be used to implement one or more features described herein. The computing device 200 can be any suitable computer system, server, or other electronic or hardware device. In one example, the computing device 200 is a media server 101 for implementing a media application 103a. In another example, the computing device 200 is a user device 115.

[0037] In some embodiments, computing device 200 includes processor 235, memory 237, input / output (I / O) interface 239, display 241, camera 243, and storage device 245, all coupled via bus 218. Processor 235 may be coupled to bus 218 via signal line 222, memory 237 may be coupled to bus 218 via signal line 224, I / O interface 239 may be coupled to bus 218 via signal line 226, display 241 may be coupled to bus 218 via signal line 228, camera 243 may be coupled to bus 218 via signal line 230, and storage device 245 may be coupled to bus 218 via signal line 232.

[0038] Processor 235 may be one or more processors and / or processing circuits for executing program code and controlling the basic operations of computing device 200. "Processor" includes any suitable hardware system, mechanism or component for processing data, signals or other information. Processors may include systems having: a general-purpose central processing unit (CPU) with one or more cores (e.g., in a single-core, dual-core or multi-core configuration), multiple processing units (e.g., in a multi-processor configuration), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a dedicated circuit system for implementing functionality, a dedicated processor for implementing processing based on a neural network model, a neural circuit, a processor optimized for matrix calculations (e.g., matrix multiplication), or other systems. In some embodiments, processor 235 may include one or more coprocessors for implementing neural network processing. In some embodiments, processor 235 may be a processor that processes data to generate a probabilistic output, for example, the output generated by processor 235 may be imprecise, or may be accurate within a range from the expected output. Processing does not have to be limited to a specific geographic location or have time constraints. For example, a processor may perform its functions in real time, offline, in batch mode, etc. Portions of the processing may be performed at different times and in different locations by different (or the same) processing systems.A computer may be any processor in communication with a memory.

[0039] Memory 237 is typically provided in computing device 200 for access by processor 235 and may be any suitable processor-readable storage medium suitable for storing instructions for execution by a processor or processor group and located separately from and / or integrated with processor 235, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc. Memory 237 may store software operated by processor 235 on computing device 200, including media application 103.

[0040] The memory 237 may include an operating system 262, other applications 264, and application data 266. The other applications 264 may include, for example, an image library application, an image management application, an image gallery application, a communication application, a web hosting engine or application, a media sharing application, etc. One or more methods disclosed herein may operate in a number of environments and platforms, such as as a stand-alone computer program that can run on any type of computing device, as a web application with a web page, as a mobile application ("app") running on a mobile computing device, etc.

[0041] Application data 266 may be data generated by other applications 264 or hardware of computing device 200. For example, application data 266 may include images used by an image library application, user actions recognized by other applications 264 (eg, a social networking application), and the like.

[0042] The I / O interface 239 may provide functionality to enable the computing device 200 to interface with other systems and devices. The interfaced devices may be included as part of the computing device 200, or may be separate and in communication with the computing device 200. For example, network communication devices, storage devices (e.g., memory 237 and / or storage device 245), and input / output devices may communicate via the I / O interface 239. In some embodiments, the I / O interface 239 may be connected to interface devices such as input devices (keyboards, pointing devices, touch screens, microphones, scanners, sensors, etc.) and / or output devices (display devices, speaker devices, printers, monitors, etc.).

[0043] Some examples of devices that can be connected to the I / O interface 239 via the interface connection can include a display 241, which can be used to display content (e.g., images, videos, and / or user interfaces of output applications as described herein), and to receive touch (or gesture) input from a user. For example, the display 241 can be used to display a user interface including a graphical guide on a viewfinder. The display 241 can include any suitable display device, such as a liquid crystal display (LCD), a light emitting diode (LED), or a plasma display screen, a cathode ray tube (CRT), a television, a monitor, a touch screen, a three-dimensional display screen, or other visual display device. For example, the display 241 can be a flat display screen provided on a mobile device, multiple display screens embedded in a glasses form factor or a head-mounted device, or a monitor screen for a computer device.

[0044] Camera 243 can be any type of image capture device that can capture images and / or video. In some embodiments, camera 243 captures images or videos, and I / O interface 239 transmits the images or videos to media application 103.

[0045] Storage 245 stores data related to media application 103. For example, storage 245 may store a training data set (including labeled images), a machine learning model, output from a machine learning model, and the like.

[0046] Figure 2 An example media application 103 is shown stored in memory 237 and includes a user interface module 202 , a segmenter 204 , a diffusion module 206 , a resolution module 208 , a shading module 210 , and a blending module 212 .

[0047] The user interface module 202 generates graphical data for displaying a user interface including an image. In some embodiments, the user interface module 202 receives an initial image. The initial image may be received from the camera 243 of the computing device 200 or from the media server 101 via the I / O interface 239. The initial image includes a subject, such as a person, an animal, a tree, etc. The subject may include multiple subjects, such as a person and a dog, a series of buildings, etc.

[0048] The user interface module 202 includes an option for providing a request associated with the image. In some embodiments, the request is a text request received from a user. The user interface module 202 may include a text box where the user enters the text request. For example, the text request may include a request to change the light level (e.g., brighten the sky, change the sky to a sunset, darken the sky, create a moonlit night, etc.), the amount of clouds in the sky (e.g., make the sky cloudy, remove clouds, show rain, etc.), and / or the color of the sky (e.g., add purple and blue, include light orange and dark yellow, make the sky look like a rainbow, etc.).

[0049] In some embodiments, user interface module 202 may provide suggestions or presets to the user that form part of the request to change the lighting. The suggestions may include region suggestions associated with one or more regions of the initial image. For example, the initial image may include different regions with different attributes, such as the sky, clouds in the sky, the horizon, water, etc.

[0050] Suggestions or presets may also include global presets (e.g., changes that affect the entire image), options menus (e.g., changes for different parts of the image, different themes, etc.), and / or a library of pre-made text requests (e.g., sky options, golden hour options, etc.).

[0051] In some embodiments, the user interface module 202 may determine that the initial image includes an outdoor scene. For example, object recognition may be performed on the initial image to determine whether the initial image includes an outdoor object. The user interface module 202 provides a suggestion to the user to modify the lighting. The suggestion may take the form of a button that the user may select to generate a list of options, a menu of suggestions, a text field that reveals a place where the user may directly enter a request, etc.

[0052] Figure 3 Example user interfaces 300, 325, 350 are shown that include options for selecting different aspects of the lighting to be changed, fields for providing text, and example output images according to some embodiments described herein. Specifically, the first user interface 300 automatically provides global presets 305 for the user to select to change the lighting to look like a sunset, night time, a cloudy sky, etc. The first user interface 300 also includes circles 310, 315, 320 that represent regional suggestions for different zones, which allow the user to specify changes to be made to only the sky, fog, and water, respectively. For example, the user can tap circle 310 to change the sky, tap circle 315 to change the fog, or tap circle 320 to change the water.

[0053] The second user interface 325 includes a text input field 330 where the user can specify the changes they want to make. The user can include a description specific enough to cover what the user wants to change (e.g., change the water to be smoother), or the user can select the object the user wants to change in the second user interface 325 and then describe the specific changes to be made.

[0054] The third user interface 350 includes an output image in which the text request "make it almost night" has been fulfilled. The resulting image has both a darker background and a darker subject 355 because the subject 355 is modified to appear consistent with the modified lighting.

[0055] The user interface module 202 generates graphical data for displaying the output image. In some embodiments, the user interface may also include options for editing the output image, sharing the output image, adding the output image to an album, etc. In some embodiments, the output image is marked to identify that artificial intelligence was used to generate the image.

[0056] FIG. 4A to FIG. 4B Example user interfaces 400, 425, 450 are shown that include options for selecting different types of modifications for changing the lighting in an image according to some embodiments described herein. The first user interface 400 includes options for modifying different aspects of an input image 402. The user interface module 202 provides options for generating different skies 405 or golden hours 410. Other variations are possible.

[0057] The second user interface 425 shows an output image 427 generated by the blending module 212 in response to the user selecting the sky 405 option in the first user interface 400. The sky 405 option includes different types of skies with variations in clouds, shades of blue, light levels, etc. The user interface module 202 can provide the user with one or more output images to select an image from. The user can also select button 430 to obtain a new set of results with different types of skies in the output images.

[0058] The third user interface 450 shows output images 452 generated by the blending module 212 in response to the user selecting the golden hour 410 option in the first user interface 400. The golden hour 410 option includes images during the daytime shortly after sunrise or shortly before sunset, when the daylight is redder and softer than when the sun is higher in the sky. The user interface module 202 can provide the user with one or more output images to select an image from. The user can also select button 430 to obtain a new set of results with different types of golden hour output images.

[0059] In some embodiments, the user interface module 202 generates a user interface that includes options for modifying user preferences. For example, the user interface can include user preferences for specifying: a randomness level including how much noise the user wants to see in the output image (e.g., the degree to which the output image differs from the initial image), and the degree to which a seed is used in the output image (e.g., the degree to which the output image differs from the image captured by the camera). The randomness level and the degree to which a seed is used can be expressed using radio buttons with different levels (e.g., low, medium, high), a scale slider, a percentage text box, or other options.

[0060] The segmenter 204 segments the initial image. In some embodiments, the segmenter 204 determines a sky segment and a subject segment. The segmenter 204 may also generate a shadow segment corresponding to one or more shadows added to the output image corresponding to an object in the output image. The sky segment includes pixels corresponding to the location of the sky in the initial image. The subject segment includes pixels corresponding to a subject, where the subject may be a person, a dog, a building, etc. The shadow segment includes pixels associated with shadows in the output image. Additional segmentation may be applied, such as by generating one or more of a foreground segment, a product segment (cloth, shoes, bags, etc.), a power line segment, a skin segment, etc. In some embodiments, the segmenter 204 generates a segmentation map that associates an identification with each pixel in the initial image, such as belonging to the sky, one or more subjects, a shadow segment, etc.

[0061] The segmenter 204 may perform segmentation by performing object recognition on the initial image to identify objects in the initial image. For example, the segmenter 204 may compare the sky and one or more subjects in the initial image with object priors of the sky, a person, a shadow, a vehicle, a building, etc. to identify the expected shape of the object in order to determine whether a pixel is associated with the sky or subject in the initial image or with a shadow in the output image. In some embodiments, the segmenter 204 may separate the image into a foreground and a background before performing object recognition to help identify the sky and subject because the sky is in the background and the subject is in the foreground. The segmenter 204 may associate the segment with a location in the initial image, such as a bounding box with x, y coordinates and scale, coordinates of pixels associated with the segment, etc.

[0062] The segmenter 204 generates one or more retention masks. The segmenter 204 generates a sky mask that acts as a retention mask corresponding to the sky segment. For example, the sky mask includes pixels corresponding to pixels of the sky segment in the initial image. The segmenter 204 generates a body mask that acts as a retention mask corresponding to the body segment. For example, the body mask includes pixels corresponding to pixels of the body segment in the initial image. In some embodiments, the segmenter 204 generates a shadow mask that acts as a retention mask corresponding to shadows in the output image.

[0063] In some embodiments, a retention mask is generated based on generating superpixels for an image and matching superpixel centroids with depth map values ​​(e.g., obtained by camera 243 using a depth sensor or by deriving depth from pixel values) to cluster detections based on depth. More specifically, the depth values ​​in the masked region can be used to determine a depth range, and superpixels that fall within the depth range can be identified.

[0064] Another technique for generating a mask includes weighting the depth values ​​based on their proximity to a retained mask, where the weights are represented by a distance transform map.

[0065] In some embodiments, segmenter 204 uses a machine learning algorithm (such as a neural network) to segment the initial image and generate a retention mask. In some embodiments, segmenter 204 can specify a circuit configuration (e.g., for a programmable processor, for a field programmable gate array (FPGA), etc.) that enables processor 235 to apply a machine learning model. In some embodiments, segmenter 204 can include software instructions, hardware instructions, or a combination. In some embodiments, segmenter 204 can provide an application programming interface (API) that can be used by operating system 262 and / or other applications 264 to call segmenter 204, for example, to apply a machine learning model to application data 266 to output a retention mask.

[0066] The segmenter 204 uses the training data to generate a trained machine learning model. For example, the training data may include multiple pairs of initial images with sky and subject and output images with sky mask and subject mask.

[0067] The training data may be obtained from any source, such as a data repository specifically marked for training, data that is licensed for use as training data for machine learning, etc. In some embodiments, training may occur on a media server 101 that provides the training data directly to a user device 115, training may occur locally on the user device 115, or a combination of both.

[0068] In some embodiments, the segmenter 204 uses weights that are obtained from another application and not edited / transferred. For example, in these embodiments, the trained model can be generated, for example, on a different device, and provided as part of the segmenter 204. In various embodiments, the trained model can be provided as a data file that includes a model structure or form (e.g., the model structure or form defines the number and type of neural network nodes, the connectivity between nodes, and the organization of nodes into multiple layers) and associated weights. The segmenter 204 can read the data file of the trained model and implement a neural network with node connectivity, layers, and weights based on the model structure or form specified in the trained model.

[0069] The trained machine learning model may include one or more model forms or structures. For example, the model form or structure may include any type of neural network, such as a linear network, a deep learning neural network that implements multiple layers (e.g., a "hidden layer" between an input layer and an output layer, where each layer is a linear network), a convolutional neural network (e.g., a network that splits or partitions input data into multiple parts or tiles, uses one or more neural network layers to process each tile separately, and aggregates the results from the processing of each tile), a sequence-to-sequence neural network (e.g., a network that receives sequential data such as words in a sentence, frames in a video, etc. as input and produces a sequence of results as output), etc.

[0070] The model form or structure can specify the connectivity between various nodes and the organization of nodes into layers. For example, the nodes of the first layer (e.g., the input layer) can receive data as input data or application data. For example, when the trained model is used for analysis of, for example, an initial image, such data may include, for example, one or more pixels per node. Subsequent intermediate layers can receive the output of the nodes of the previous layer as input according to the connectivity specified in the model form or structure. These layers may also be referred to as hidden layers. For example, the first layer can output the segmentation between the foreground and the background. The last layer (e.g., the output layer) generates the output of the machine learning model. For example, the output layer can receive the segmentation of the initial image into the foreground and the background, and output whether the pixel is part of the retention mask. In some embodiments, the model form or structure also specifies the number and / or type of nodes in each layer.

[0071] In various embodiments, the trained model may include one or more models. One or more models in the model may include multiple nodes, which are arranged in layers according to the model structure or form. In some embodiments, the node may be a computing node without memory, for example, configured to process an input unit to generate an output unit. The calculation performed by the node may include, for example, multiplying each node input in multiple node inputs by a weight, thereby obtaining a weighted sum, and using a bias or intercept value to adjust the weighted sum to generate a node output. In some embodiments, the calculation performed by the node may also include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a nonlinear function. In various embodiments, such calculations may include operations such as matrix multiplication. In some embodiments, the calculations performed by multiple nodes may be performed in parallel, for example, using multiple processor cores of a multi-core processor, using individual processing units of a graphics processing unit (GPU), or a dedicated neural circuit system. In some embodiments, the node may include a memory, for example, it may be possible to store one or more earlier inputs and use the one or more earlier inputs to process subsequent inputs. For example, a node with a memory may include a long short-term memory (LSTM) node. LSTM nodes can use memory to maintain the "state" that allows the node to act like a finite state machine (FSM).

[0072] In some embodiments, a trained model may include embeddings or weights for individual nodes. For example, a model may be initiated as a plurality of nodes organized into layers as specified by the model form or structure. Upon initialization, corresponding weights may be applied to connections between each pair of nodes connected in the model form (e.g., nodes in successive layers of a neural network). For example, corresponding weights may be randomly assigned, or initialized to default values. The model may then be trained (e.g., using training data) to produce results.

[0073] Training can include applying supervised learning techniques. In supervised learning, the training data can include multiple inputs (e.g., images, retention masks, etc.) and corresponding true value outputs for each input (e.g., a true value mask of a subject that correctly identifies the subject, a true value mask of a sky that correctly identifies the sky in each image, a true value mask of a shadow that correctly identifies the shadow in each image, etc.). Based on a comparison of the output of the model with the true value output, the value of the weights is automatically adjusted, for example, in a manner that increases the probability that the model produces a true value output for the image.

[0074] In various embodiments, the trained model includes a set of weights or embeddings corresponding to the model structure. In some embodiments, the trained model may include a fixed set of weights (e.g., downloaded from a server that provides weights). In various embodiments, the trained model includes a set of weights or embeddings corresponding to the model structure. In embodiments where data is omitted, the segmenter 204 may generate a trained model based on a priori training performed, for example, by a developer of the segmenter 204, by a third party, etc. In some embodiments, the trained model may include a fixed set of weights (e.g., downloaded from a server that provides weights).

[0075] In some embodiments, the trained machine learning model receives an initial image having a sky and one or more subjects. In some embodiments, the trained machine learning model outputs a sky mask corresponding to the sky and one or more subject masks corresponding to the one or more subjects, wherein the sky mask and the subject mask are reserved masks. In some embodiments, the trained machine learning model receives an output image having one or more shadows and outputs a shadow mask.

[0076] In some embodiments, the machine learning model outputs a confidence value for each retention mask output by the trained machine learning model. The confidence value can be expressed as a percentage, a number from 0 to 1, etc. For example, the machine learning model outputs a confidence value of 85% for the confidence that the retention mask correctly incorporates the subject and does not include pixels from another person or object.

[0077] The diffusion module 206 receives as input a request to change the illumination in the initial image and the initial image. The diffusion module 206 outputs an output image that satisfies the request to change the illumination. In some embodiments, the diffusion module 206 performs text conditioning to generate an output image that is adjusted according to the text request. For example, if the text request is for a sunset, the diffusion module 206 performs text conditioning by generating a version of the initial image that is modified to look like it was captured during sunset. The diffusion model can perform diffusion until a balance is reached between the efficiency of the process and the quality of the output image.

[0078] In some embodiments, the diffusion module 206 trains the diffusion model using two types of training data. The first type of training data includes image pairs, where the pairs may include synthetic pairs generated by a prompt-to-prompt generative machine learning model. The prompt-to-prompt generative machine learning model is a diffusion model that receives a text prompt and uses cross-attention to extract keys and values ​​from the text prompt, and switches parts of an attention map previously generated for a first image based on the input text prompt to output a second image that matches the text prompt.

[0079] The second type of training data includes pairs with real images and synthetic images. The real images are received by a diffusion model, such as a denoising diffusion implicit model (DDIM). The diffusion model uses a reversal method to output a synthetic image based on the real image and instructions on how to edit the input image. The diffusion module 206 uses a forward process (where the diffusion model adds noise to the data) and a reverse process (where the diffusion model learns to recover the data from the noise) to train the diffusion model to generate an output image according to the request.

[0080] The diffusion module 206 trains the diffusion model to maintain photorealism and preserve the identity of the person shown in the image. During training, the diffusion model receives the editing instructions and modifies the editing instructions based on a language model (such as a large language model) to create corresponding prompts. For example, the diffusion module 206 uses the language model to convert the editing instruction "make it sunsettime" into prompts describing various outdoor scenes during the day and corresponding prompts for sunset. The diffusion model creates a set of input and output image pairs from the generated prompt pairs, where each prompt can generate N images (using different seeds). The diffusion module 206 filters certain images from the image pairs, such as image transformations that do not match a given editing instruction, image transformations that do not produce well-aligned images, and unmatched pairs. In some embodiments, the diffusion module 206 also filters images based on an edit alignment score that reflects the alignment between the image-to-image transformation and the original editing commentary, and an image text alignment score that reflects the alignment between the input / output image and the corresponding input / output prompt. In some embodiments, the diffusion module 206 trains the diffusion model by generating one or more loss functions based on the images filtered from the image pairs.

[0081] Once the diffusion model is trained, the diffusion model receives requests to change the lighting in the initial image.In some embodiments, the requests are not directly from the user, but are instead pre-populated by the media application 103, such as by the user selecting a request from a library of pre-made text requests.

[0082] In some embodiments, the resolution module 208 generates a super-resolution version of at least a portion of the output image. For example, the resolution module 208 can generate a super-resolution version from a portion of the output image corresponding to the sky segment. In some embodiments, the diffusion model works best for a low-resolution output image. Therefore, the resolution module 208 advantageously improves the quality of the sky segment by generating a super-resolution version of the output image.

[0083] In some embodiments, the resolution module 208 generates a super-resolution version of at least the portion of the output image using one or more of the following techniques. The resolution module 208 may perform pre-upsampling by upsampling the low-resolution output image to a coarse high-resolution image of a desired size using bicubic interpolation, and provide the coarse high-resolution image as input to a deep convolutional neural network (CNN) that outputs a super-resolution version. The resolution module 208 may perform post-upsampling by providing the low-resolution image to the CNN without increasing the resolution, and applying an upsampling layer at the end of the CNN. In some embodiments, as an alternative or in addition to the CNN, the resolution module 208 uses a diffusion model to output a super-resolution version.

[0084] The colorization module 210 modifies the colorization of the initial image to match the colorization of the output image. In some embodiments, the colorization module 210 modifies a portion of the initial image, such as the subject, but not the remainder of the initial image, because the remainder of the initial image is replaced with a super-resolution version of at least that portion of the output image during blending. Thus, the colorization module 210 can modify the colorization of a portion of the initial image based on the content of the initial image (i.e., whether the initial image includes more than a sky segment and a subject segment).

[0085] In some embodiments, the colorization module 210 determines a bilateral grid approximation, which is a three-dimensional array that combines a two-dimensional spatial domain corresponding to (x, y) positions in the image plane with a one-dimensional range dimension that is typically the image intensity. In some embodiments, the colorization module 210 performs a bilateral grid upsampling (BGU) that determines the local color transform between the initial image and the output image (except for the sky portion) by fitting a low-resolution version of the input image / output image pair and applying an affine model to the high-resolution input. The colorization module 210 applies the local color transform to the initial image while using a sky mask to prevent the color of the sky.

[0086] The blending module 212 blends the modified initial image with the output image to form a blended image, while during blending, using a subject mask to prevent modification of the subject from the modified initial image, and using a sky mask to prevent modification of the sky from the output image. The subject mask advantageously prevents the subject in the output image from blending with the modified initial image. Because the diffusion module 206 can generate an output image with distortion that is discernible in the subject, using the subject mask ensures that the distorted version of the subject is not blended with the initial image.

[0087] The pixels of the sky mask correspond to the pixels of the sky in the original image, and vice versa. When blending the output image, the pixels of the sky in the modified original image are not blended or modified respectively. The pixels of the subject mask correspond to the pixels of the subject in the original image and the pixels of the modified original image, and vice versa. When blending the modified original image, the pixels of the subject in the modified original image are not blended or modified respectively.

[0088] In some embodiments where a super-resolution version of the output image is generated, the blending module 212 blends the super-resolution version of at least a portion of the output image with the modified initial image, while during blending, using a sky mask to prevent modification of the sky from the modified image. For example, the blending module 212 may blend a portion of the super-resolution version of the output image corresponding to a sky segment with the modified image. The sky mask advantageously prevents the super-resolution version of the sky from being altered by the modified initial image. The blending module 212 may blend the super-resolution version of a portion of the output image after the output image is blended with the modified image, or may blend all three images during the same blending step.

[0089] In some embodiments, the diffusion module 206 generates an output image with one or more shadows of the corresponding object consistent with the lighting. For example, the shadow corresponds to the direction of sunlight projected from the sun. In some embodiments, the segmenter 204 determines to output a shadow mask for protecting shadows attached to people and / or objects. The blending module 212 can prevent modification of the shadows when blending the output image with the modified image.

[0090] Although the diffusion module 206, the resolution module 208, the shading module 210 and the blending module 212 Figure 2 206 are shown as separate components, but one or more components may be combined. For example, the diffusion module 206 may also perform the mixing functions described with reference to the mixing module 212.

[0091] Example Architecture

[0092] Figure 5 5 is a block diagram of an example architecture 500 for generating a blended image incorporating a request. An input image 505 is received by a media application, and segmentation 510 is performed on the input image to generate a segmentation mask 515. The segmentation mask includes a sky segment and a subject segment. The input image 505 and a request to change the lighting are received by a diffusion model 520, which generates an output image 525 that satisfies the text request. In this example, the request to change the lighting is a request to change the input image 505 to a night scene with a moon.

[0093] The media application performs a super-resolution 527 process on the sky portion of the output image 525 to increase the quality of the output image 525. As a result of the super-resolution 527 process, the media application outputs a super-resolution sky image 535.

[0094] The diffusion module applies BGU 530 to the initial image so that the initial image matches the coloring of the output image. For example, the modified image 540 has a darker color than the initial image 505 because the modified image 540 looks like it was captured at night.

[0095] The media application blends the modified image 540 with the super-resolution sky image 535, while using the segmentation mask 515 to prevent the sky in the super-resolution sky image 535 from being modified by the modified image 540, and to prevent the subjects in the modified image 540 from being combined with potentially distorted versions of the subjects from the super-resolution sky image 535. The media application blends the images to generate a final image 550.

[0096] Example Flowchart

[0097] Figure 6 An example flow chart of a method 600 for generating a blended image with modified lighting is shown. The method 600 may be performed by Figure 2 In some embodiments, the method 600 is performed by the user device 115, the media server 101, or partially performed on the user device 115 and partially performed on the media server 101.

[0098] Figure 6 The method 600 may begin at block 602. At block 602, an initial image and a request to change lighting in the initial image are provided as inputs to a diffusion model, wherein the initial image includes a subject and a sky.

[0099] The request to change the lighting may include a user providing a text request including properties selected from the group consisting of light level (“change this to a moonlit night”), cloudiness in the sky (“change this to a clear sky”), and / or color of the sky (“change this to a red and orange sky”). The request to change the lighting may include region suggestions associated with one or more regions of the initial image, a global preset, an options menu, and / or a library of pre-made text requests.

[0100] In some embodiments, it is determined that the initial image includes an outdoor scene, and a suggestion is provided to the user to modify the lighting, wherein a request to change the lighting is received in response to providing the suggestion. Block 602 may be followed by block 604 .

[0101] At block 604, the diffusion model outputs an output image that satisfies the request. Block 604 may be followed by block 606.

[0102] At block 606 , a sky segment and a body segment are determined from the initial image. Block 606 may be followed by block 608 .

[0103] At block 608 , a sky mask corresponding to the sky segment and a body mask corresponding to the body segment are generated. Block 608 may be followed by block 610 .

[0104] At block 610, the shading of the initial image is modified to match the shading of the output image. In some embodiments, modifying the shading of the initial image includes performing a BGU that identifies a local color transformation between the initial image and the output image, and applying the local color transformation to the initial image. Block 610 may be followed by block 612.

[0105] At block 612, an optional step includes generating a super-resolution version of at least a portion of the output image from the output image. Block 612 may be followed by block 614.

[0106] At block 614, the modified initial image is blended with the output image to form a blended image, while during blending, using the subject mask to prevent modification of the subject from the modified initial image and using the sky mask to prevent modification of the sky from the output image. If a super-resolution version of at least the portion of the output image is generated, blending includes blending the super-resolution version of at least the portion of the output image, while during blending, using the subject mask to prevent modification of the subject from the modified initial image and using the sky mask to prevent modification of the sky from the super-resolution version of at least the portion of the output image.

[0107] In some embodiments, the output image includes one or more shadows corresponding to one or more objects in the output image, and the method further includes: determining a shadow segment corresponding to the one or more shadows in the output image from the initial image, wherein blending the output image with the modified initial image includes: preventing modification of the one or more shadows from the output image during blending.

[0108] In addition to the above description, controls may be provided to allow the user to make a choice about whether and when the systems, programs, or features described herein may be able to collect user information (e.g., information about the user's social network, social behavior, or activities, occupation, user preferences, or the user's current location) and whether to send content or communications from a server to the user. In addition, before certain data is stored or used, it may be processed in one or more ways to remove personally identifiable information. For example, the user identification may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location for which location information is obtained may be generalized (such as to a city, zip code, or state level) so that the user's specific location cannot be determined. Thus, the user can control what information is collected about the user, how the information is used, and what information is provided to the user.

[0109] Therefore, according to the above, the media application provides an initial image and a request to change the lighting in the initial image as input to the diffusion model, wherein the initial image includes a subject and a sky. The media application outputs an output image that satisfies the request using the diffusion model. The media application determines a sky segment and a subject segment from the initial image. The media application generates a sky mask corresponding to the sky segment and a subject mask corresponding to the subject segment. The media application modifies the shading of the initial image to match the shading of the output image. The media application blends the modified initial image with the output image to form a blended image, while during blending, using the subject mask to prevent modification of the subject from the modified initial image, and using the sky mask to prevent modification of the sky from the output image.

[0110] In the above description, many specific details are set forth for the purpose of explanation in order to provide a thorough understanding of the specification. However, it will be apparent to those skilled in the art that the present disclosure can be practiced without these specific details. In some instances, structures and devices are shown in block diagram form to avoid obscuring the present description. For example, the above may mainly refer to user interfaces and specific hardware to describe embodiments. However, embodiments may be applied to any type of computing device that can receive data and commands, and any peripheral device that provides services.

[0111] References in the specification to "some embodiments" or "some examples" mean that a particular feature, structure, or characteristic described in conjunction with the embodiments or examples may be included in at least one implementation of the specification. The appearance of the phrase "in some embodiments" in various places in the specification does not necessarily all refer to the same embodiment.

[0112] Some portions of the above detailed description are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means by which those skilled in the art of data processing most effectively communicate the substance of their work to other persons skilled in the art. An algorithm is here and generally considered to be a self-consistent sequence of steps that produces a desired result. The steps are steps that require physical manipulation of physical quantities. Typically, although not necessarily, these quantities are typically in the form of electrical or magnetic data that can be stored, transferred, combined, compared, and otherwise manipulated. Mainly for general reasons, it has sometimes proven convenient to refer to these data as bits, values, elements, symbols, characters, terms, numbers, etc.

[0113] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically noted otherwise, as will be apparent from the following discussion, it should be understood that throughout this specification, discussions utilizing terms including "processing" or "computing" or "calculating" or "determining" or "displaying" and the like refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities within the computer system's registers and memories and transforms that data into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission or display devices.

[0114] Embodiments of the specification may also relate to a processor for performing one or more steps of the above method. The processor may be a dedicated processor selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory computer-readable storage medium, including but not limited to any type of disk, including an optical disk, ROM, CD-ROM, magnetic disk, RAM, EPROM, EEPROM, magnetic card or optical card, flash memory (including USB disk with non-volatile memory), or any type of medium suitable for storing electronic instructions, each of which is coupled to a computer system bus.

[0115] The specification can take the form of some complete hardware embodiments, some complete software embodiments, or some embodiments containing both hardware elements and software elements. In some embodiments, the specification is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0116] Furthermore, the present description may take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of the present description, a computer-usable or computer-readable medium may be a device that can contain, store, communicate, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or device.

[0117] A data processing system suitable for storing or executing program code will include at least one processor coupled directly or indirectly to a memory element via a system bus. The memory element may include local memory, bulk storage, and cache memory used during the actual execution of the program code, which provides temporary storage of at least some program code to reduce the number of times code must be retrieved from bulk storage during execution.

Claims

1. A computer-implemented method comprising: providing an initial image and a request to change lighting in the initial image as input to a diffusion model, wherein the initial image includes a subject and a sky; Outputting an output image satisfying the request using the diffusion model; determining a sky segment and a body segment from the initial image; generating a sky mask corresponding to the sky segment and a body mask corresponding to the body segment; modifying the coloring of the initial image to match the coloring of the output image; as well as The modified initial image is blended with the output image to form a blended image, while during the blending, the subject mask is used to prevent modification of the subject from the modified initial image, and the sky mask is used to prevent modification of the sky from the output image.

2. The method of claim 1 , wherein modifying the coloring of the initial image comprises: performing a bilateral grid upsampling (BGU) that identifies local color transformations between the initial image and the output image; as well as The local color transform is applied to the initial image.

3. The method according to claim 1, further comprising: generating from the output image a super-resolution version of at least a portion of the output image; Wherein blending the modified initial image with the output image comprises blending the super-resolution version of at least the portion of the output image, while during the blending, using the subject mask to prevent modification of the subject from the modified initial image, and using the sky mask to prevent modification of the sky from the super-resolution version of at least the portion of the output image.

4. The method of claim 1 , wherein the output image includes one or more shadows corresponding to one or more objects in the output image, and the method further comprises: determining from the output image a shadow segment corresponding to the one or more shadows in the output image; as well as generating a shadow mask corresponding to the shadow segment; Wherein blending the output image with the modified initial image comprises: during the blending, using the shadow mask to prevent modification of the one or more shadows from the output image.

5. The method of claim 1, wherein the request to change the lighting comprises a user providing a text request including an attribute selected from the group consisting of: light level, amount of clouds in the sky, color of the sky, and combinations thereof.

6. The method of claim 1, wherein the request to change the lighting is selected from the group consisting of: region suggestions associated with one or more regions of the initial image, global presets, an options menu, a library of pre-made text requests, and combinations thereof.

7. The method according to claim 1, further comprising: prior to receiving the request to change the lighting in the initial image, determining that the initial image includes an outdoor scene; as well as Suggestions are provided to the user for modifying the lighting.

8. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform operations, the operations comprising: providing an initial image and a request to change lighting in the initial image as input to a diffusion model, wherein the initial image includes a subject and a sky; Outputting an output image satisfying the request using the diffusion model; determining a sky segment and a body segment from the initial image; generating a sky mask corresponding to the sky segment and a body mask corresponding to the body segment; modifying the shading of the initial image to match the shading of the output image; as well as The modified initial image is blended with the output image to form a blended image, while during the blending, the subject mask is used to prevent modification of the subject from the modified initial image, and the sky mask is used to prevent modification of the sky from the output image.

9. The non-transitory computer-readable medium of claim 8, wherein modifying the coloring of the initial image comprises: performing a bilateral grid upsampling (BGU) that identifies local color transformations between the initial image and the output image; as well as The local color transform is applied to the initial image.

10. The non-transitory computer readable medium of claim 8, wherein the operations further comprise: generating from the output image a super-resolution version of at least a portion of the output image; Wherein blending the modified initial image with the output image comprises blending the super-resolution version of at least the portion of the output image, while during the blending, using the subject mask to prevent modification of the subject from the modified initial image, and using the sky mask to prevent modification of the sky from the super-resolution version of at least the portion of the output image.

11. The non-transitory computer-readable medium of claim 8, wherein the output image includes one or more shadows corresponding to one or more objects in the output image, and the operations further comprise: determining from the output image a shadow segment corresponding to the one or more shadows in the output image; as well as generating a shadow mask corresponding to the shadow segment; Wherein blending the output image with the modified initial image comprises: during the blending, using the shadow mask to prevent modification of the one or more shadows from the output image.

12. The non-transitory computer-readable medium of claim 8, wherein the request to change the lighting comprises a user providing a text request comprising an attribute selected from the group consisting of: light level, amount of clouds in the sky, color of the sky, and combinations thereof.

13. The non-transitory computer-readable medium of claim 8, wherein the request to change the lighting is selected from the group consisting of: region suggestions associated with one or more regions of the initial image, global presets, an options menu, a library of pre-made text requests, and combinations thereof.

14. The non-transitory computer readable medium of claim 8, wherein the operations further comprise: prior to receiving the request to change the lighting in the initial image, determining that the initial image includes an outdoor scene; as well as Suggestions are provided to the user for modifying the lighting.

15. A system comprising: processor; as well as A memory coupled to the processor, wherein the memory stores instructions, and when the instructions are executed by the processor, the processor performs operations, the operations comprising: providing an initial image and a request to change lighting in the initial image as input to a diffusion model, wherein the initial image includes a subject and a sky; Outputting an output image satisfying the request using the diffusion model; determining a sky segment and a body segment from the initial image; generating a sky mask corresponding to the sky segment and a body mask corresponding to the body segment; modifying the shading of the initial image to match the shading of the output image; and The modified initial image is blended with the output image to form a blended image, while during the blending, the subject mask is used to prevent modification of the subject from the modified initial image, and the sky mask is used to prevent modification of the sky from the output image.

16. The system of claim 15, wherein modifying the coloring of the initial image comprises: performing a bilateral grid upsampling (BGU) that identifies local color transformations between the initial image and the output image; as well as The local color transform is applied to the initial image.

17. The system of claim 15, wherein the operations further comprise: generating from the output image a super-resolution version of at least a portion of the output image; Wherein blending the modified initial image with the output image comprises blending the super-resolution version of at least the portion of the output image, while during the blending, using the subject mask to prevent modification of the subject from the modified initial image, and using the sky mask to prevent modification of the sky from the super-resolution version of at least the portion of the output image.

18. The system of claim 15, wherein the output image includes one or more shadows corresponding to one or more objects in the output image, and the operations further comprise: determining from the output image a shadow segment corresponding to the one or more shadows in the output image; as well as generating a shadow mask corresponding to the shadow segment; Wherein blending the output image with the modified initial image comprises: during the blending, using the shadow mask to prevent modification of the one or more shadows from the output image.

19. The system of claim 15, wherein the request to change the lighting comprises a user providing a text request comprising an attribute selected from the group consisting of: light level, amount of clouds in the sky, color of the sky, and combinations thereof.

20. The system of claim 15, wherein the request to change the lighting is selected from the group consisting of: region suggestions associated with one or more regions of the initial image, global presets, an options menu, a library of pre-made text requests, and combinations thereof.

Images