Segmentation of objects in images

Abstract

The media application performs object recognition on the initial image to identify a set of objects in the initial image. The media application determines whether the initial image is an outdoor scene. In response to the initial image being an outdoor scene, the media application determines a sky section from the initial image. The media application determines whether the initial image includes a subject that is a human or animal. In response to the initial image including the subject, the media application determines a subject segment from the initial image. The media application receives, at a user interface that includes an initial image, a user input corresponding to a selection of a selected object from a set of objects. The media application updates the user interface to include an indication that the selected object has been selected.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims U.S. Provisional Patent Application No. 63 / 465,232, filed on May 9, 2023, and entitled “Selecting a Region of an Image,” U.S. Provisional Patent Application No. 63 / 465,224, filed on May 9, 2023, and entitled “Relighting of Outdoor Images Using Machine Learning,” U.S. Provisional Patent Application No. 63 / 465,226, filed on May 9, 2023, and entitled “Prompt-Drive Image Editing Using Machine-Learning,” U.S. Provisional Patent Application No. 63 / 465,230, filed on May 9, 2023, and entitled “Repositioning Objects in an Image,” and U.S. Provisional Patent Application No. 63 / 465,231, filed on March 7, 2024, and entitled “Performing Scene Impact Editing Tasks Using Diffusion Neural Networks (Using Diffusion Neural Networks to Perform Scene-Affected Editing Tasks)" each of which is incorporated herein in its entirety. Background Art

[0002] As technology for editing images improves, it becomes increasingly important to intuitively translate user intent into object selection in user interfaces.

[0003] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the presently named inventors (to the extent that it is described in this background section) and aspects of the specification that might not have been considered prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art to the present disclosure. Summary of the invention

[0004] A computer-implemented method includes: performing object recognition on an initial image to identify a set of objects in the initial image. The method further includes: determining whether the initial image is an outdoor scene. In response to the initial image being an outdoor scene, the method determines a sky segment from the initial image. The method further includes: determining whether the initial image includes a subject that is a human or an animal. In response to the initial image including the subject, the method determines a subject segment from the initial image. The method determines whether the initial image includes one or more distracting objects. In response to the initial image including one or more distracting objects, the method determines one or more distracting segments from the initial image. The method further includes: receiving user input corresponding to a selection of a selected object from a set of objects at a user interface including the initial image. The method further includes: updating the user interface to include an indication that the selected object has been selected.

[0005] In some embodiments, the user input includes multiple taps on the selected object, and further includes: determining the number of taps from the user input; and determining the selected object based on the number of taps, wherein the first tap is associated with a different area than the second tap. In some embodiments, the method further includes: in response to the initial image including a subject, generating a background segment, wherein the subject segment is associated with a foreground area, the background segment is associated with a background area, and the pixels in the initial image are associated with the foreground area or the background area; and determining that the user input corresponds to the foreground area based on the user input contacting the pixels associated with the foreground area. In some embodiments, performing object recognition to identify the objects in the initial image includes determining an object bounding box for each of the objects, and the method further includes: determining that the user input corresponds to the selected object based on the proximity of the user input to the closest object bounding box.

[0006] In some embodiments, a convolutional neural network (CNN) performs segmentation, and the method further includes: providing the initial image and the heat map of the key points as input to the CNN; and using the convolutional neural network to output a segmentation mask corresponding to a sky segment, a subject segment, and one or more distractor segments. In some embodiments, the user input includes a selection of the sky, and the method further includes: receiving a request from the user to change the lighting in the initial image; providing the initial image and the request to change the lighting in the initial image as input to a diffusion model; and using the diffusion model to output an output image that satisfies the request. In some embodiments, the user input includes a selection of one or more background objects to be removed, and the method further includes: removing one or more distractor objects from the initial image based on object recognition; and generating a modified image including a repair of pixels associated with the one or more distractor segments.

[0007] In some embodiments, the selected object is an incomplete object, and the omitted portion of the incomplete object is truncated by the boundary of the initial image or occluded by another object, the method comprising: generating a segmentation mask including the incomplete object; removing the incomplete object from the initial image; generating a repaired image, the repaired image replacing the incomplete object pixels corresponding to the incomplete object with background pixels matching the background in the initial image; providing the segmentation mask, the incomplete object, and the repaired image as inputs to a diffusion model; using the diffusion model to output the complete object; and generating a modified image by blending one or more versions of the complete object with one or more versions of the repaired image using a retained mask. In some embodiments, the method further comprises: receiving a text request to change the selected object in the initial image; determining a facial segment of the face of the subject from the initial image based on the subject segment; generating a retained mask corresponding to the facial segment; providing the text request, the initial image, and the retained mask as inputs to the diffusion model; and using the diffusion model to output an output image that satisfies the text request.

[0008] A non-transitory computer-readable medium having instructions stored thereon, the instructions causing the one or more processors to perform operations when executed by one or more processors. The operations include: performing object recognition on an initial image to identify a set of objects in the initial image; determining whether the initial image is an outdoor scene; in response to the initial image being an outdoor scene, determining a sky segment from the initial image; determining whether the initial image includes a subject that is a human or an animal; in response to the initial image including the subject, determining a subject segment from the initial image; determining whether the initial image includes one or more distracting objects; in response to the initial image including the one or more distracting objects, determining one or more distracting segments from the initial image; receiving user input corresponding to a selection of a selected object from the set of objects at a user interface including the initial image; and updating the user interface to include an indication that the selected object has been selected.

[0009] In some embodiments, the user input includes a plurality of taps on the selected object, and the operation further includes: determining the number of taps from the user input; and determining the selected object based on the number of taps, wherein the first tap is associated with a different area than the second tap. In some embodiments, the operation further includes: in response to the initial image including a subject, generating a background segment, wherein the subject segment is associated with a foreground area, the background segment is associated with a background area, and the pixels in the initial image are associated with the foreground area or the background area; and determining that the user input corresponds to the foreground area based on the user input contacting the pixels associated with the foreground area. In some embodiments, performing object recognition to identify the objects in the initial image includes determining an object bounding box for each of the objects, and the operation further includes: determining that the user input corresponds to the selected object based on the proximity of the user input to the closest object bounding box. In some embodiments, where a CNN performs segmentation, the operation further includes: providing the initial image and a heat map of key points as input to the CNN; and using a convolutional neural network to output a segmentation mask corresponding to a sky segment, a subject segment, and one or more distractor segments. In some embodiments, the user input includes a selection of a sky, and the operation further includes: receiving a request from the user to change the lighting in the initial image; providing the initial image and the request to change the lighting in the initial image as input to the diffusion model; and using the diffusion model to output an output image that satisfies the request.

[0010] A system includes: a processor; and a memory coupled to the processor, the memory storing instructions, which when executed by the processor, cause the processor to perform operations. The operations include: performing object recognition on an initial image to identify a set of objects in the initial image; determining whether the initial image is an outdoor scene; in response to the initial image being an outdoor scene, determining a sky segment from the initial image; determining whether the initial image includes a subject that is a human or an animal; in response to the initial image including the subject, determining a subject segment from the initial image; determining whether the initial image includes one or more distracting objects; in response to the initial image including the one or more distracting objects, determining one or more distracting segments from the initial image; receiving user input corresponding to a selection of a selected object from the set of objects at a user interface including the initial image; and updating the user interface to include an indication that the selected object is selected.

[0011] In some embodiments, the user input includes a plurality of taps on the selected object, and the operation further includes: determining the number of taps from the user input; and determining the selected object based on the number of taps, wherein the first tap is associated with a different area than the second tap. In some embodiments, the operation further includes: in response to the initial image including a subject, generating a background segment, wherein the subject segment is associated with a foreground area, the background segment is associated with a background area, and the pixels in the initial image are associated with the foreground area or the background area; and determining that the user input corresponds to the foreground area based on the user input contacting the pixels associated with the foreground area. In some embodiments, performing object recognition to identify the objects in the initial image includes determining an object bounding box for each of the objects, and the operation further includes: determining that the user input corresponds to the selected object based on the proximity of the user input to the closest object bounding box. In some embodiments, where a CNN performs segmentation, the operation further includes: providing the initial image and a heat map of key points as input to the CNN; and using a convolutional neural network to output a segmentation mask corresponding to a sky segment, a subject segment, and one or more distractor segments. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram of an example network environment according to some embodiments described herein.

[0013] Figure 2 is a block diagram of an example computing device according to some embodiments described herein.

[0014] Figure 3 is a block diagram of an example architecture of a trained tap segmentation machine learning model according to some embodiments described herein.

[0015] FIG. 4A to FIG. 4C Illustrate an example user interface for selecting a region of an image according to some embodiments described herein.

[0016] Figure 5A An example initial image illustrating a child sitting on a bench and holding a balloon that is partially truncated by a border of the initial image in accordance with some embodiments described herein.

[0017] Figure 5B An example modified image is illustrated in which a child, a bench, and a balloon are moved to a second position in accordance with some embodiments described herein.

[0018] Figure 6 An example user interface including options for selecting different areas of an image to change, global presets to apply, fields for providing text, and an example output image is illustrated in accordance with some embodiments described herein.

[0019] Figure 7 An example flow chart illustrating a method of making modifications to an initial image according to some embodiments described herein.

[0020] FIG. 8A to FIG. 8B An example flow chart illustrating a method to segment an initial image according to some embodiments described herein. DETAILED DESCRIPTION

[0021] As technology for editing images improves, it becomes increasingly important to intuitively translate user intent into object selection in a user interface. Previous software applications for editing images perform segmentation of an object in an image after a user selects the object. This can result in unnecessary delays while the software application performs the segmentation. Additionally, the software application may attempt to correct the delays caused by the segmentation by performing the segmentation using a less precise but faster process. Less precise segmentation can result in inappropriate identification of the boundaries of an object, where pixels associated with the background or other objects are incorrectly classified as belonging to the object. If the editing includes moving an object, the object may appear out of place in the new location when the object includes pixels associated with the background from a previous location.

[0022] The media application performs pre-processing on the initial image prior to user interaction to identify a set of objects in the initial image. For example, the media application performs object recognition to identify subjects (e.g., people, dogs, children, etc.), trees, bystanders, the sky, etc. The media application performs segmentation of different objects based on the likelihood of the user selecting the object. For example, if the initial image is an outdoor scene, the user can select the sky and change the color of the sky, remove clouds, etc.

[0023] The media application determines whether the initial image is an outdoor scene based on object recognition. In response to the initial image being an outdoor scene, the media application determines a sky segment from the initial image, wherein pixels corresponding to the sky are identified as sky pixels. The media application determines whether the initial image includes a subject that is a human or an animal based on object recognition. In response to the initial image including a subject, the media application determines a subject segment from the initial image, wherein pixels corresponding to the subject are identified as subject pixels. The media application determines whether the initial image includes one or more interfering objects. In response to the initial image including one or more interfering objects, the media application determines one or more interfering segments from the initial image, wherein pixels corresponding to the sky are identified as interfering object pixels. In some embodiments, interfering objects are identified based on the type of objects that are frequently removed from the initial image.

[0024] The media application receives user input corresponding to a selection of a selected object from a set of objects identified based on performing object recognition at a user interface including the initial image. For example, the user can select a subject and provide a text request to add a hat to the subject, select a bystander and request that the bystander be removed from the image, select an incomplete object truncated by the boundary of the initial image and move the incomplete object to a new location, causing the media application to generate a complete object for the new location. The media application updates the user interface to include an indication that the selected object was selected. For example, the indication can include a highlighted object, an outline around the selected object, etc.

[0025] By performing segmentation prior to receiving user input, the media application advantageously reduces processing time the user spends waiting for segmentation to occur and improves the quality of the segmentation, resulting in an output image in which background pixels are not inappropriately associated with selected objects.

[0026] Sample Environment 100

[0027] Figure 1 1 is a block diagram illustrating an example environment 100. In some embodiments, the environment 100 includes a media server 101 coupled to a network 105, a user device 115a, and a user device 115n. Users 125a, 125n may be associated with respective user devices 115a, 115n. In some embodiments, the environment 100 may include Figure 1 Other servers or devices not shown in FIG. Figure 1 In the illustrative drawings and the remaining drawings, a letter following a reference number (e.g., "115a") indicates a reference to the element having that specific reference number. A reference number in the text without a following letter (e.g., "115") indicates a general reference to an embodiment of the element with that reference number.

[0028] The media server 101 may include a processor, a memory, and network communication hardware. In some embodiments, the media server 101 is a hardware server. The media server 101 is communicatively coupled to the network 105 via a signal line 102. The signal line 102 may be a wired connection (such as Ethernet, coaxial cable, fiber optic cable, etc.) or a wireless connection (such as Wi-Fi®, Bluetooth® or other wireless technologies). In some embodiments, the media server 101 sends data to and receives data from one or more of the user devices 115a, 115n via the network 105. The media server 101 may include a media application 103a and a database 199.

[0029] The database 199 may store machine learning models, training data sets, images, etc. The database 199 may also store social network data associated with the user 125, user preferences of the user 125, etc.

[0030] User device 115 may be a computing device including a memory coupled to a hardware processor. For example, user device 115 may include a mobile device, a tablet computer, a mobile phone, a wearable device, a head mounted display, a mobile email device, a portable game player, a portable music player, a reader device, or another electronic device capable of accessing network 105.

[0031] In the illustrated implementation, user device 115a is coupled to network 105 via signal line 108, and user device 115n is coupled to network 105 via signal line 110. Media application 103 may be stored on user device 115a as media application 103b and / or stored on user device 115n as media application 103c. Signal lines 108 and 110 may be wired connections (such as Ethernet, coaxial cable, fiber optic cable, etc.) or wireless connections (such as Wi-Fi®, Bluetooth® or other wireless technologies). User devices 115a, 115n are accessed by users 125a, 125n, respectively. Figure 1 The user devices 115a, 115n in FIG. 1 are used by way of example. Figure 1 Two user devices 115 a and 115 n are illustrated, but the present disclosure is applicable to a system architecture having one or more user devices 115 .

[0032] The media application 103 may be stored on the media server 101 or the user device 115. In some embodiments, the operations described herein are performed on the media server 101 or the user device 115. In some embodiments, some operations may be performed on the media server 101 and some operations may be performed on the user device 115. The execution of the operations is performed according to user settings. For example, the user 125a may specify a setting that an operation is performed on the user's corresponding user device 115a but not on the media server 101. With such settings, the operations described herein are performed entirely on the user device 115a, and no operations are performed on the media server 101. Further, the user 125a may specify that the user's images and / or other data will only be stored locally on the user device 115a and not on the media server 101. With such settings, no user data is sent to the media server 101 or stored on the media server. The transmission of user data to the media server 101, any temporary or permanent storage of such data by the media server 101, and the performance of operations on such data by the media server 101 are only performed if the user has consented to the transmission, storage, and performance of operations by the media server 101. The user is provided with the option to change the settings at any time, for example so that they can enable or disable the use of the media server 101.

[0033] If a machine learning model (e.g., a neural network or other type of model) is used for one or more operations, the machine learning model is stored and utilized locally on the user device 115, with specific user permissions. The server-side model is used only with user permission. Further, the trained model can be provided for use on the user device 115. During such use, if the user 125 permits, on-device training of the model can be performed. If the user 125 permits, for example, to enable federated learning, updated model parameters can be sent to the media server 101. The model parameters do not include any user data.

[0034] The media application 103 performs object recognition on the initial image to identify a set of objects in the initial image. The media application 103 determines whether the initial image is an outdoor scene. In response to the initial image being an outdoor scene, the media application 103 determines a sky segment from the initial image. The media application 103 determines whether the initial image includes a subject that is a human or an animal. In response to the initial image including a subject, the media application 103 determines a subject segment from the initial image. The media application 103 determines whether the initial image includes one or more interfering objects. In response to the initial image including one or more interfering objects, the media application 103 determines one or more interfering segments from the initial image.

[0035] The media application 103 receives user input corresponding to a selection of a selected object from the set of objects at a user interface including the initial image. The media application 103 updates the user interface to include an indication that the selected object is selected.

[0036] In some embodiments, media application 103 can be implemented using hardware, including a central processing unit (CPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a machine learning processor / coprocessor, any other type of processor, or a combination thereof. In some embodiments, media application 103a can be implemented using a combination of hardware and software.

[0037] Example computing device 200

[0038] Figure 2 is a block diagram of an example computing device 200 that can be used to implement one or more features described herein. The computing device 200 can be any suitable computer system, server, or other electronic or hardware device. In one example, the computing device 200 is a media server 101 for implementing a media application 103a. In another example, the computing device 200 is a user device 115.

[0039] In some embodiments, computing device 200 includes processor 235, memory 237, input / output (I / O) interface 239, display 241, camera 243, and storage device 245, all coupled via bus 218. Processor 235 may be coupled to bus 218 via signal line 222, memory 237 may be coupled to bus 218 via signal line 224, I / O interface 239 may be coupled to bus 218 via signal line 226, display 241 may be coupled to bus 218 via signal line 228, camera 243 may be coupled to bus 218 via signal line 230, and storage device 245 may be coupled to bus 218 via signal line 232.

[0040] The processor 235 may be one or more processors and / or processing circuits for executing program code and controlling the basic operation of the computing device 200. "Processor" includes any suitable hardware system, mechanism or component for processing data, signals or other information. The processor may include a general-purpose central processing unit (CPU) with one or more cores (e.g., in a single-core, dual-core or multi-core configuration), multiple processing units (e.g., in a multi-processor configuration), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a dedicated circuit system for implementing functionality, a dedicated processor for implementing processing based on a neural network model, a neural circuit, a processor optimized for matrix calculation (e.g., matrix multiplication) or other systems. In some embodiments, the processor 235 may include one or more coprocessors for implementing neural network processing. In some embodiments, the processor 235 may be a processor that processes data to generate a probabilistic output, for example, the output generated by the processor 235 may be inaccurate, or may be accurate within a range from the expected output. The processing does not have to be limited to a specific geographic location or have a time limit. For example, the processor may perform the functions of the processor in real time, offline, in batch mode, etc. Portions of the processing may be performed by different (or the same) processing systems at different times and different locations.A computer may be any processor in communication with a memory.

[0041] The memory 237 is generally provided in the computing device 200 so as to be accessed by the processor 235, and may be any suitable processor-readable storage medium suitable for storing instructions for execution by a processor or a set of processors and located separately from and / or integrated with the processor 235, such as a random access memory (RAM), a read-only memory (ROM), an electrically erasable read-only memory (EEPROM), a flash memory, etc. The memory 237 may store software operated by the processor 235 on the computing device 200, including the media application 103.

[0042] The memory 237 may include an operating system 262, other applications 264, and application data 266. The other applications 264 may include, for example, an image library application, an image management application, an image gallery application, a communication application, a web hosting engine or application, a media sharing application, etc. One or more methods disclosed herein may operate in a number of environments and platforms, such as, for example, as a stand-alone computer program that can run on any type of computing device, as a web application with a web page, as a mobile application ("app") running on a mobile computing device, etc.

[0043] Application data 266 may be data generated by other applications 264 or hardware of computing device 200. For example, application data 266 may include images used by an image library application, user actions recognized by other applications 264 (e.g., a social networking application), and the like.

[0044] The I / O interface 239 may provide functionality to enable the computing device 200 to be interfaced with other systems and devices. The interfaced devices may be included as part of the computing device 200, or may be separate and in communication with the computing device 200. For example, network communication devices, storage devices (e.g., memory 237 and / or storage device 245), and input / output devices may communicate via the I / O interface 239. In some embodiments, the I / O interface 239 may be connected to interface devices such as input devices (keyboards, pointing devices, touch screens, microphones, scanners, sensors, etc.) and / or output devices (display devices, speaker devices, printers, monitors, etc.).

[0045] Some examples of devices that can be connected to the I / O interface 239 can include a display 241, which can be used to display content (e.g., images, videos, and / or user interfaces of output applications as described herein) and receive touch (or gesture) input from a user. For example, the display 241 can be used to display a user interface including a graphical guide on a viewfinder. The display 241 may include any suitable display device, such as a liquid crystal display (LCD), a light emitting diode (LED), or a plasma display screen, a cathode ray tube (CRT), a television, a monitor, a touch screen, a three-dimensional display screen, or other visual display device. For example, the display 241 can be a flat display screen set on a mobile device, multiple display screens embedded in a glasses shape or a head-mounted device, or a monitor screen of a computer device.

[0046] Camera 243 can be any type of image capture device that can capture images and / or video. In some embodiments, camera 243 captures images or videos, and I / O interface 239 sends the images or videos to media application 103.

[0047] The storage device 245 stores data related to the media application 103. For example, the storage device 245 may store a training data set including labeled images, a machine learning model, output from the machine learning model, and the like.

[0048] Figure 2 The example media application 103 stored in the memory 237 is illustrated and includes a segmenter 202 , a user interface module 204 , a repairer module 206 , and a diffusion module 208 .

[0049] Segmentation is the process of labeling pixels in the initial image as being associated with a particular class. Segmentation can be used for a variety of reasons. For example, segmentation can be used to identify objects in the image that the user wants to remove, such as bystanders, power lines, scooters, etc. Segmentation can also be used to select objects for enhancement. For example, the user may want to change the background of an image or replace the clothing of a subject in the image. Segmentation can also be used to identify areas of the initial image to be retained by generating a retention mask that includes pixels associated with the object that are prevented from being modified and mixed with the synthetically generated image.

[0050] Once the pixels are marked, the output of the segmentation is one or more segmentation masks including pixels associated with the segmented objects or regions in the initial image. The segmentation mask can be used as a grouping of pixels associated with the object or region so that when the user interface receives user input, the user interface module 204 determines whether the user input corresponds to a specific segmentation mask based on the location of the user input. For example, the user interface module 204 can identify that the user input touches a number of pixels associated with the background segmentation mask. The segmentation mask can be used as a retention mask to prevent modification of pixels associated with the retention mask when modifying pixels not associated with the retention mask. For example, during the process of generating an output image using a diffusion model, a retention mask is used on the face of the subject to prevent the face from becoming distorted during the generation of the output image.

[0051] The segmenter 202 receives an initial image. The initial image may be captured by a camera 243 associated with the computing device 200, received from other applications 264, etc. The segmenter 202 performs object recognition on the initial image to identify a set of objects in the initial image. Object recognition may be performed by a machine learning model or another algorithm. In some embodiments, the segmenter 202 determines an object bounding box for each of the objects in the set of objects. The object bounding box may include pixels associated with a particular object and is associated with metadata describing the object bounding box, such as (x, y) coordinates describing the edges of the object bounding box.

[0052] The segmenter 202 performs segmentation of the initial image. For example, the segmenter 202 identifies pixels associated with a subset of the set of objects in the initial image based on object recognition and the likelihood that a user will select the subset of the set of objects in the initial image. The likelihood that a user will select the subset of objects can be based on anonymous information about what people select in an image. For example, a user may be most likely to select the subject of the image, likely to select distracting objects in the image, and least likely to select aesthetically pleasing background objects, such as trees, buildings in a city landscape, ships on the ocean, etc.

[0053] In some embodiments, segmenter 202 determines whether the initial image has a specific type of object, and performs segmentation in response to the initial image including the specific type of object. For example, segmenter 202 determines whether the initial image is an outdoor scene based on object recognition to identify the presence of the sky. An outdoor scene is characterized by an image including the sky. In some embodiments, segmenter 202 determines that the initial image is an outdoor scene based on the initial image including certain colors associated with an outdoor scene and / or certain colors being located in an area where the sky is expected to be located. An outdoor scene may include additional objects, such as buildings, trees, beaches, etc. If the initial image is an outdoor scene, segmenter 202 determines the sky segment of the initial image.

[0054] The segmenter 202 determines whether the initial image includes a subject that is a human or an animal based on object recognition to identify objects associated with the human and / or animal categories. For example, the subject may be a cat, a chicken, a person, etc. If the initial image includes a subject that is a human or an animal, the segmenter 202 determines a subject segment from the initial image.

[0055] The segmenter 202 determines whether the initial image includes one or more interfering objects. Interfering objects may be based on the types of objects that are frequently removed from the initial image, such as people, cars, power lines, etc. that are not the subject of the initial image. Conversely, the segmenter 202 may not segment objects such as trees because trees are not frequently removed from the initial image. In some embodiments, classifying an object as an interfering object is based on ranking the types of objects removed from the initial image with a cutoff value (e.g., the likelihood of the top 20 most frequently removed objects being classified as types of interfering objects, exceeding a threshold likelihood value that a type of object will be removed from the initial image, etc.). If the initial image includes one or more interfering objects, the segmenter determines one or more interfering segments from the initial image.

[0056] In some embodiments, segmentation also includes foreground / background segmentation, sky segmentation, and / or panorama segmentation (e.g., segmenting an image into semantically meaningful parts or regions). Foreground / background segmentation can be used by a media application 103 that performs selective tone mapping. Tone mapping is used to modify the tone values ​​of pixels. Tone mapping can be used to adjust the tone values ​​of an initial image with a high dynamic range for applications such as viewing on a digital display.

[0057] The segmenter 202 may use different methods to segment subsets of objects in the image. In some embodiments, the segmenter 202 segments the object into regions. In some embodiments, the segmenter 202 divides the image into foreground and background, and segments the object based on whether the object is located in the foreground or the background.

[0058] In some embodiments, segmenter 202 generates different kinds of segmentation masks for segmentation performed on an image. For example, segmenter 202 can generate a subject mask that retains the face of a subject or includes more parts of a subject (such as an entire head, hands, body of a subject, etc.). In some embodiments, the segmentation mask is generated based on generating superpixels of an image and matching superpixel centroids with depth map values ​​(e.g., obtained by camera 243 using a depth sensor or by deriving depth from pixel values) to cluster detections based on depth. More specifically, the depth values ​​in the masked area can be used to determine a depth range, and superpixels that fall within the depth range can be identified.

[0059] Another technique for generating a segmentation mask includes weighting the depth values ​​based on how close they are to the mask, where the weights are represented by a distance transform map.

[0060] In some embodiments, the segmenter 202 may specify a circuit configuration (e.g., for a programmable processor, for a field programmable gate array (FPGA), etc.) that enables the processor 235 to apply the machine learning model. In some embodiments, the segmenter 202 may include software instructions, hardware instructions, or a combination. In some embodiments, the segmenter 202 may provide an application programming interface (API) that may be used by the operating system 262 and / or other applications 264 to call the segmenter 202, for example, to apply the machine learning model to the application data 266 to output a segmentation mask.

[0061] The segmenter 202 uses the training data to generate a trained machine learning model. In some embodiments, the training data includes an image (e.g., a red, green, blue (RGB) image) and a heat map of key points in the image. Key points are unique or salient points in the initial image that are used to identify, describe, or match objects or features in a scene. For example, a scale-invariant feature transform (SIFT) can be used to determine key points. In some embodiments, the training data also includes a corresponding segmentation mask.

[0062] The training data may be obtained from any source (e.g., a data repository specifically labeled for training, data provided with permission to be used as training data for machine learning, etc.) In some embodiments, the training may be performed on a media server 101 that provides the training data directly to the user device 115, the training may be performed locally on the user device 115, or a combination of both.

[0063] In some embodiments, the segmenter 202 uses weights that were taken from another application and not edited / transferred. For example, in these embodiments, the trained model can be generated, for example, on a different device, and provided as part of the segmenter 202. In various embodiments, the trained model can be provided as a data file that includes a model structure or form (e.g., defining the number and type of neural network nodes, connectivity between nodes, and organization of nodes into multiple layers) and associated weights. The segmenter 202 can read the data file of the trained model and implement the neural network using node connectivity, layers, and weights based on the model structure or form specified in the trained model.

[0064] The trained machine learning model may include one or more model forms or structures. For example, the model form or structure may include any type of neural network, such as a linear network, a deep learning neural network that implements multiple layers (e.g., a "hidden layer" between an input layer and an output layer, each of which is a linear network), a convolutional neural network (CNN) (e.g., a network that splits or divides input data into multiple parts or tiles, processes each tile separately using one or more neural network layers, and aggregates the results from the processing of each tile), a sequence-to-sequence neural network (e.g., a network that receives sequential data (such as words in a sentence, frames in a video, etc.) as input and produces a sequence of results as output), etc.

[0065] The model form or structure can specify the connectivity between various nodes and the organization of nodes into layers. For example, the nodes of the first layer (e.g., the input layer) can receive data as input data or application data. For example, when the trained model is used for analysis (e.g., of an initial image), such data may include, for example, one or more pixels per node. Subsequent intermediate layers may receive the outputs of the nodes of the previous layers as inputs according to the connectivity specified in the model form or structure. These layers may also be referred to as hidden layers. For example, the first layer may output the segmentation between the foreground and the background. The final layer (e.g., the output layer) generates the output of the machine learning model. For example, the output layer may receive the segmentation of the initial image into the foreground and the background, and output whether the pixel is part of the segmentation mask. In some embodiments, the model form or structure also specifies the number and / or type of nodes in each layer.

[0066] Figure 3 3 is a block diagram of an example architecture 300 of a trained tap segmentation machine learning model according to some embodiments described herein. The example architecture includes a CNN that receives input and generates output. The CNN includes convolutional layers that apply filters to input data to extract features. The convolutional layers can be followed by pooling layers to reduce spatial dimensions and increase computational efficiency.

[0067] CNN includes CNN encoder 315 and CNN decoder 320. Encoder receives an image and encodes the image into a vector or matrix representation of the image. CNN encoder 315 receives RGB image 305 and corresponding heat map of key points 310. RGB image 305 is an image including pixels containing one of three color channels red, green and blue. Key points 310 include the location of user contact within the initial image. Key points 310 can be defined as the location where the user input exceeds a threshold user input value.

[0068] CNN encodes RGB image 305 into increasingly abstract information, where each convolutional layer represents a different level of abstraction. CNN decoder 320 decodes the abstracted information and outputs a segmentation mask 325 that identifies pixels associated with one or more objects in RGB image 305. For example, RGB image 305 may be an image of a coffee cup on a table, and the heat map of key points 310 has a key point in the center of the coffee cup to indicate that users typically select the coffee cup rather than anything else in the image. CNN decoder 320 outputs a segmentation mask that segments the coffee cup from the rest of the image because it is likely that the user taps on the coffee cup rather than other objects in the image.

[0069] In various embodiments, the trained model may include one or more models. One or more of the models may include multiple nodes arranged in layers according to the model structure or form. In some embodiments, the node may be a memoryless computing node, which is configured to process a unit of input to produce a unit of output, for example. The calculation performed by the node may include, for example, multiplying each of the multiple node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum using a bias or intercept value to produce a node output. In some embodiments, the calculation performed by the node may also include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a nonlinear function. In various embodiments, such calculations may include operations such as matrix multiplication. In some embodiments, the calculations performed by multiple nodes may be performed in parallel using, for example, multiple processor cores of a multi-core processor, a separate processing unit using a graphics processing unit (GPU), or a dedicated neural circuit system. In some embodiments, the node may include memory, for example, it may be possible to store one or more earlier inputs and use the one or more earlier inputs when processing subsequent inputs. For example, a node with memory may include a long short-term memory (LSTM) node. LSTM nodes may use memory to maintain a "state" that allows the node to behave like a finite state machine (FSM).

[0070] In some embodiments, the trained model may include embeddings or weights of individual nodes. For example, the model may be initiated as a plurality of nodes organized into layers as specified by the model form or structure. Upon initialization, corresponding weights may be applied to the connections between each pair of nodes connected in the model form (e.g., nodes in successive layers of a neural network). For example, corresponding weights may be randomly assigned, or initialized to default values. The model may then be trained, for example, using training data, to produce results.

[0071] Training can include applying supervised learning techniques. In supervised learning, the training data can include multiple inputs (e.g., images, segmentation masks, etc.) and corresponding true value outputs for each input (e.g., a true value mask that correctly identifies a portion of a subject (such as the subject's face) in each image). Based on a comparison of the output of the model with the true value output, the value of the weight is automatically adjusted, for example, in a manner that increases the probability that the model produces the true output for the image.

[0072] In various embodiments, the trained model includes a set of weights or embeddings corresponding to the model structure. In some embodiments, the trained model may include a fixed set of weights, such as downloaded from a server that provides weights. In various embodiments, the trained model includes a set of weights or embeddings corresponding to the model structure. In embodiments where data is omitted, the segmenter 202 may generate a trained model based on previous training performed, such as by a developer of the segmenter 202, by a third party, etc. In some embodiments, the trained model may include a fixed set of weights, such as downloaded from a server that provides weights.

[0073] In some embodiments, the trained machine learning model receives an initial image with objects identified by object recognition. In some embodiments, the trained machine learning model outputs one or more segmentation masks corresponding to one or more of the objects. For example, the trained machine learning model outputs segmentation masks for the sky, the subject, and one or more interfering objects. In another example, the trained machine learning model outputs segmentation masks for the background and foreground.

[0074] The user interface module 204 generates graphical data for displaying a user interface including an image. The user interface displays different options for associating user input with corresponding areas in the image. FIG. 4A to FIG. 4C Illustrate an example user interface for selecting a region of an image according to some embodiments described herein.

[0075] Figure 4A A first user interface 400 is included in which a user is instructed to circle any object the user wants to select according to some embodiments described herein. This may be referred to as a stroke selection. In this example, the user has circled 402 a subject in an image. In a second user interface 405, the user is instructed to tap one of the circles to select an object. For example, the user may select circle 406 to select the sky, circle 407 to select a tree, circle 408 to select a user, etc. In a third user interface 410, the user is instructed to select one of the areas / objects from a list 412 of sky, person, car, sign, background, and clothing. "Person" in list 412 is highlighted as an indication showing that the user has selected a person.

[0076] Figure 4BA fourth user interface 415 is included in which a single circle is associated with multiple regions / objects in accordance with some embodiments described herein. In this example, circle 416 may be selected a first time to select the sky, and circle 416 may be selected a second time to select the background. In response to the user selecting circle 416 once, user interface module 204 may update the user interface to display a segmentation mask to indicate pixels associated with the sky segment. In response to the user selecting circle 416 twice, user interface module 204 may update the user interface to display a segmentation mask to indicate pixels associated with the background segment.

[0077] In the fifth user interface 420, the segmenter 202 has segmented the image into a foreground segment and a background segment. The person is in the foreground, and everything else is in the background. Therefore, selecting any area within the foreground area, as illustrated by the example foreground arrow 422, results in the selection of the person. Selecting any area within the background area, as illustrated by the background arrow 424, results in the selection of the background.

[0078] In the sixth user interface 423 , the user selects a pixel corresponding to the foreground segment in the fifth user interface 420 , and the user interface module 204 updates the user interface to include an indicator, which is the segmentation mask 425 associated with the foreground segment.

[0079] Figure 4C A seventh user interface 430 is included in which a user can tap on an object to select the corresponding object. An object is associated with an object bounding box. If the user taps within the bounding box, the object is selected. For example, tapping within bounding box 426 results in the selection of a car. Tapping within bounding box 427 results in the selection of a stop sign. This scenario can lead to some confusion, for example when a user taps on a segment located within two bounding boxes (such as area 429 where bounding box 427 and bounding box 428 overlap). In some embodiments, if the selection is ambiguous, the user interface can display text asking the user to confirm which object the user intends to select, or the user interface can update the display to provide an indicator of which object the user is more likely to intend to select, and if the user disagrees, the user can change the indicator.

[0080] Once the user interface module 204 determines what object / area the user input corresponds to, the user interface module 204 generates graphical data for displaying an indicator that the object is selected. For example, the user interface may add an outline around the selected object, highlight the selected object, etc.

[0081] Figure 5AAn example initial image 500 is illustrated of a child 505 sitting on a bench 510 and holding a balloon 515 that is partially truncated by the border of the initial image 500 in accordance with some embodiments described herein. In this example, the user interface module 204 provides the user interface with options for the user to select objects segmented by the segmenter 202. The user selects the child 505, the bench 510, and the balloon 515 at a first position, where the balloon 515 represents an incomplete image.

[0082] The user interface module 204 includes an option for moving the selected object to a different location. The user selects a second location. The segmenter 202 removes the selected object from the initial image. The repairer module 206 generates a repaired image that replaces object pixels corresponding to the removed object with background pixels that match the background in the initial image.

[0083] The diffusion module 208 receives as input the coordinates of the selected object and the second location, and outputs a balloon of the complete object and a longer bench. Figure 5B An example modified image 550 is illustrated in which a child 555, a bench 560, and a balloon 565 are moved to a second position in accordance with some embodiments described herein. In this example, the diffusion module 208 outputs a modified image that uses a segmentation mask to blend one or more versions of the child 555, the bench 560, and the balloon 565 with one or more versions of the inpainted image.

[0084] Figure 6 Example user interfaces 600, 625, 650 including options for selecting different areas of an image to be changed, global presets to be applied, fields for providing text, and example output images are illustrated according to some embodiments described herein. Specifically, the first user interface 600 automatically provides a global preset 605 for the user to select to change the input image 601 to look like a painting, a surreal world, or a nostalgic scene.

[0085] The first user interface 600 also includes circles 610, 611, 612 representing identification of different areas in the initial image 601. The user can specify changes to the sky by tapping the first circle 610, to the bridge by tapping the second circle 611, and to the person by tapping the third circle 612.

[0086] In response to the user selecting one of the circles 610, 611, 612, the user interface may update the display to provide a menu of options (not shown). For example, selecting the first circle 610 may cause the user interface to display suggestions, such as changing a cloudy sky to a clear sky. Selecting the second circle 611 may cause the user interface to display suggestions, such as an option to remove the bridge associated with the second circle 611, an option to replace the bridge with a different type of bridge or boat, etc. Selecting the third circle 612 may cause the user interface to display a suggestion to remove the character.

[0087] The second user interface 625 includes an input image 626 and a text input field 630 in which the user can specify the changes they want to make. The user can include a description that is specific enough to cover the object the user wants to change (e.g., change the boots to colorful glitter boots), or the user can select the object the user wants to change in the second user interface 625 and then describe the specific changes to make. For example, the user can select the object by tapping on the object, circling the object, drawing on the object, etc. In this case, the user selects the boots 627 on the subject.

[0088] The third user interface 650 includes an output image 651 in which a text request 652 of "colorful glitter boots" is satisfied. The boots 653 are changed into glittering colorful stars.

[0089] For the situation of removing the object from the initial image, the inpainting module 206 generates an image through repairing that replaces the object pixels corresponding to one or more objects with background pixels. The background pixels can be based on the pixels of the reference image without the object from the same position. Alternatively, the inpainting module 206 can identify the background pixels to replace the removed object based on the proximity of the background pixels and other pixels around the object. The inpainting module 206 can use the gradient of the neighborhood pixels to determine the attribute of the background pixels. For example, when the bystander is standing on the ground, the inpainting module 206 replaces the background pixels with the pixels on the ground. Other inpainting techniques are possible, including the inpainting technique based on machine learning that outputs the background pixels based on the training data of the image that includes similar structures.

[0090] In embodiments where the user chooses to erase the selected object, the user interface module 204 may display an inpainted image in which the selected object is removed and the selected object pixels are replaced with background pixels.

[0091] The diffusion model includes a forward process in which the diffusion model adds noise to the data and a reverse process in which the diffusion model learns to recover the data from the noise. For example, in the case of moving a selected object from a first location to a second location, the diffusion module 208 applies the diffusion model by mixing the selected object with a progressively noisy version of the inpainted image and then a progressively denoised version. In some embodiments, an object splicing diffusion model is used to move the object from a first location to a second location. In some embodiments, a generative diffusion model is used when the object is an incomplete object and a portion of the object is generated and / or for a new object generated from a text prompt.

[0092] Object splicing diffusion model

[0093] In some embodiments, an object splicing diffusion model is used when moving an object from a first position to a second position. In some embodiments, the diffusion module 208 includes an object image encoder that extracts semantic features from a selected object, a diffusion model that mixes the object with the image, and a content adapter that transforms a sequence of visual words into a sequence of text words to overcome the domain gap between the image and the text. In some embodiments, the diffusion module 208 uses self-supervision to train the diffusion model based on training data, wherein the training data includes image and text pairs. In some embodiments, the diffusion model is trained on synthetic data that simulates real-world scenarios. The diffusion model can also be trained using data augmentation generated by introducing random shifts and cropping augmentation during training while ensuring that the foreground object is contained within the cropping window.

[0094] During the first stage, a content adapter is trained using image and text pairs to maintain the high-level semantics of the object, and during the second stage, the content adapter is trained in the context of a diffusion model to encode key identity features of the object by facilitating the visual reconstruction of the object in the original image. The diffusion model can be trained on the embeddings produced by the content adapter through the cross-attention block.

[0095] The diffusion model uses a preserved mask to blend the inpainted image with the object. The diffusion model can denoise the masked area. The content adapter can transform the visual features from the object image encoder into text features (words) used as conditioning for the diffusion model.

[0096] Generative Diffusion Model

[0097] In some embodiments, the generative diffusion model is used to output a complete object based on an incomplete object, or to generate a new object. The diffusion module 208 trains the generative diffusion model based on training data. The training data may include image and text pairs for creating an embedding space of images and text. The image and text pairs may include images associated with corresponding texts, such as an image of a dog and text including "pitbull". The diffusion module 208 may be trained using a loss that reflects the cosine distance between the embedding of the text hint and the estimated clean image (i.e., the object generated without text).

[0098] The diffusion module 208 can use the training data to perform text conditioning, where text conditioning describes the process of outputting objects conditioned on text prompts. The diffusion module 208 can train the neural network to output objects based on text prompts provided by the user or by the media application. For example, the text prompt can be a suggestion generated by the media application based on the context of the initial image (e.g., where the initial image is a beach, the text prompt can be for a beach ball, a turtle, etc.).

[0099] In some embodiments, the diffusion module 208 may output at least a portion of the missing portion of the object based on receiving the incomplete object as input data, the location within the image to which the incomplete object is moved, and the output size (e.g., the original size or the modified size if the object is resized). For example, if a user selects an object that is partially truncated by a boundary in a user interface and moves the object from a first location to a second location, where the second location also truncates a portion of the object, the diffusion module 208 may output a modified object that includes more of the object that is visible based on moving the object in the image. In some embodiments, the diffusion module 208 may output a complete object based on the incomplete object selected by the user. For example, where the user selects a beach ball that is partially occluded by another object, the diffusion model may be trained to output a complete beach ball.

[0100] In some embodiments, the diffusion module 208 generates a progressively noisier version of the complete object compared to the previous version and a progressively noisier version of the inpainted image compared to the previous version. For example, a forward Markov noise process generates a series of noisy inpainted images by gradually adding Gaussian noise until a nearly isotropic Gaussian noise sample is obtained. The forward noise process defines a series of image manifolds, each of which consists of noisy images.

[0101] The diffusion module 208 may use the retention mask to spatially blend the noisy version of the complete object with the corresponding noisy version of the inpainted image. For example, the diffusion module 208 may use the retention mask to blend each noisy version of the complete object with each corresponding noisy version of the inpainted image, wherein the retention mask delineates the boundaries of the complete object, so that the retention mask delineates the area that is modified during the blending process. In some embodiments, the diffusion process may include a local complete object guided diffusion, wherein the image generation loss determined during the training process is used under the retention mask during the position object generation diffusion.

[0102] The diffusion module 208 can perform a diffusion step to denoise the latent space in the direction according to the text prompt. The diffusion module 208 generates a progressively denoised version of the complete object compared to the previous version and a progressively denoised version of the inpainted image compared to the previous version. For example, an inverse Markov process transforms the Gaussian noise sample by repeatedly denoising the inpainted image using a learned posterior. Each step of the denoising diffusion process projects the noisy image onto the next less noisy manifold.

[0103] The diffusion module 208 performs a denoising diffusion step after each blending to restore coherence by projecting onto the next manifold. Once the spatial blending is complete, the diffusion module 208 preserves the background by replacing the areas outside the preservation mask with corresponding areas from the inpainted image.

[0104] In some embodiments, the diffusion module 208 uses cross-domain synthesis to apply an iterative refinement scheme to infuse contextual information to the object so that the object matches the style of the inpainted image. For example, if the object is generated for an indoor occasion and is added to an outdoor inpainted image, the object can be modified to be brighter to match the inpainted image. In another example, if the object is located at a first location that is in the shadow, and a second location is in full sunlight, the object can be modified to match the brightness of the second location.

[0105] Object Removal Model

[0106] In some embodiments, instead of using the segmenter 202 to remove objects and using the inpaintor module 206 to add pixels to the removed areas in the initial image, the diffusion model is trained to include an object removal model.

[0107] The diffusion module 208 generates counterfactual training data to train the diffusion model to include the object removal model. For each counterfactual image pair, the diffusion module 208: captures a factual image containing the object in the scene; physically removes the object while avoiding camera movement, lighting changes, or motion of other objects; captures a counterfactual image of the scene without the object, and segments the factual image to create a retained mask. Segmenting the factual image includes creating a segmentation map (M) for the object O removed from the factual image X. o ).

[0108] The diffusion module 208 creates a combined image and a counterfactual image for each image pair, including a factual image and a retention mask. The retention mask may be a binary retention mask (M o (X)), and the counterfactual image pair can be described as an input pair of the factual image and the binary preservation mask (X, M o (X)), and output the counterfactual image (X cf ).

[0109] The diffusion module 208 estimates the distribution of counterfactual images given a factual image x and a binary preservation mask by training a diffusion model based on counterfactual image pairs. The diffusion module 208 minimizes the loss function by using the following equation To determine the estimate:

[0110]

[0111] in is a denoiser network with the following input: the noisy latent representation of the counterfactual image , the latent representation x of the image containing the object to be removed cond , a mask m indicating the location of the object, a timestamp t, and an encoding p of the empty string (text hint). t Use the following forward process equation to calculate:

[0112]

[0113] where x represents the image without the object (counterfactual), α t and σ t is determined by the noise addition plan, and .

[0114] Once the diffusion model including the object removal model is trained, the user interface module 204 can receive a request to remove the selected object from the first modified image. The initial image and the request are provided as input to the object removal model, which outputs a modified image that does not include the selected object.

[0115] Object Insertion Model

[0116] In some embodiments, instead of using segmenter 202 to remove objects, using inpaintor module 206 to add pixels to the removed areas in the initial image, and using a diffusion model to blend the objects with the pixels at the new locations, a diffusion model is trained to include an object insertion model.

[0117] In some embodiments, the object insertion model is trained on a number of image pairs that exceeds the number of counterfactual image pairs available. Thus, diffusion module 208 generates synthetic training data. For each synthetic image pair, diffusion module 208 selects an original image that includes an object, uses the object removal model to output a modified image without the object from the original image, generates an input image by inserting the object into the modified image, and segments the original image to create a retained mask. The modified image lacking the object is referred to as z i , which uses the following equation:

[0118]

[0119] where the original images are x1, x2, …, x n , and the corresponding retention mask is M o (x1), M o (x2),…,M o (x n The diffusion module 208 inserts the object into the object-free scene z i The input image is generated from to produce an image without shadows and reflections using the following equation:

[0120]

[0121] The synthetic image pair is (y i , M o (x i )), and the corresponding target is the original image x i Although both the input image and the output image contain object o, the input image does not contain the effect of the object on the scene, while the output image does. In some embodiments, the diffusion module 208 utilizes the diffusion objective presented in Equation 1 to train the object insertion model.

[0122] For each synthetic image pair, the diffusion module 208 creates a second combined image including the original image and the retained mask and the input image. The diffusion module 208 pre-trains the diffusion model to include an object insertion model based on using the synthetic image pairs, and fine-tunes the diffusion model to include an object insertion model based on using the counterfactual image pairs used to train the object removal model.

[0123] In some embodiments, the user interface module 204 generates graphical data for displaying a user interface that provides the user with options to specify the location of the object and options to adjust the size of the object. The diffusion module 208 adds the selected object removed from the initial image to the new location. In some embodiments, the diffusion module 208 provides the selected object and the location where the selected object will be located in the modified image as input to a diffusion model, and outputs a modified image that mixes the selected object with the inpainted image. For example, the diffusion module 208 can spatially mix a noisy version of the inpainted image with a noisy version of the selected object.

[0124] In some embodiments, the diffusion module 208 can add a shadow to the selected object in the new position. The shadow can match the direction of the light in the image. For example, if the sun casts light from the upper left corner of the image, the shadow can appear to the right of the person and / or object. In some embodiments, the diffusion module 208 uses a machine learning model to output a shadow mask for generating a shadow attached to the object.

[0125] Diffusion model for text requests

[0126] In some embodiments, a user may select an object or region and provide a request to change the selected object or region. For example, a user may select a subject to change the subject's clothing, or select the sky to change the lighting of the sky. The diffusion model receives as input a request (e.g., a text request provided directly by the user, a selection of a pre-made prompt, a selection of a global preset, a selection of an option from a menu, etc.), an initial image, and a retention mask. The diffusion model encodes the image in latent space, performs diffusion, and decodes back to pixel space.

[0127] The diffusion module 208 performs text conditioning of the request. Text conditioning describes the process of generating an image that is conditioned on (e.g., aligned with) a text prompt. For example, if the text request is for replacing a red shirt worn by a subject in an initial image with a blue shirt, the diffusion module 208 performs text conditioning by generating an output image of a blue shirt.

[0128] In some embodiments, the diffusion module 208 uses two types of training data to train the diffusion model. The first type of training data includes image pairs, where the pair may include a synthetic pair generated by a prompt-to-prompt generative machine learning model. The prompt-to-prompt generative machine learning model is a diffusion model that receives a text prompt and uses self-attention to extract keys and values ​​from the text prompt, and switches parts of an attention map previously generated for an input image based on the input text prompt to output an output image to match the text prompt.

[0129] The prompt-to-prompt generative machine learning model generates a self-attention map. Self-attention computes the interactions between different elements of the input sequence (e.g., different words in a text request). This is in contrast to cross-attention, where the interactions are between two different input sequences (e.g., how the text request relates to the original prompt).

[0130] The self-attention map describes the structure and different semantic regions in the image. For example, in the self-attention map, an image described as "pepperoni pizza next to orange juice" includes how the pixels on the pizza crust pay attention to other pixels on the crust. Conversely, in the cross-attention map, the pixels on the pizza crust pay attention to the orange juice.

[0131] The self-attention map is used in a text-conditional diffusion model to use the structure and different semantic regions in the input image to change one or more word-meta values ​​while fixing the self-attention map to preserve scene composition. In some embodiments, the diffusion model adds new words to the prompt and freezes attention on previous word-meta while allowing new attention to flow to the new word-meta. This results in a global edit or modification of specific objects in the input image to match the text request.

[0132] Each diffusion step predicts noise and text embeddings from the noisy image. At the final step, the process produces a generated image. The interaction between the textual cues and the image is performed during the noise prediction, where the embeddings of visual and textual features are fused using a self-attention layer that produces a spatial attention map for each text token.

[0133] The second type of training data includes pairs with real images and synthetic images. The real images are received by a diffusion model such as a denoising diffusion implicit model (DDIM). The diffusion model uses an inversion method to output a synthetic image based on the real image and instructions on how to edit the input image. The diffusion module 208 trains the diffusion model to generate an output image from a request using a forward process in which the diffusion model adds noise to the data and a reverse process in which the diffusion model learns to recover the data from the noise.

[0134] The diffusion module 208 trains the diffusion model to maintain photo-realism and preserve the identity of the objects shown in the image. During training, the diffusion model receives editing instructions and modifies the editing instructions based on a language model, such as a large language model, to create corresponding prompts. For example, the diffusion module 208 uses a language model to convert the editing instruction "make person look like an astronaut" into prompts that describe various aspects of how the clothes of a space suit will look.

[0135] The diffusion model creates a set of input and output image pairs from the generated prompt pairs, where each prompt can generate N images (using different seeds). The diffusion module 208 filters out certain images from the image pairs, such as image transformations that do not match a given edit instruction, image transformations that do not produce well-aligned images, and unmatched pairs. In some embodiments, the diffusion module 208 also filters images based on an edit alignment score that reflects the alignment between the image-to-image transformation and the original edit instruction text, and an image-text alignment score that reflects the alignment between the input / output image and the corresponding input / output prompt. In some embodiments, the diffusion module 208 trains the diffusion model by generating one or more loss functions based on the images filtered out from the image pairs.

[0136] The diffusion model is trained to generate images by gradually adding noise to the image, and then learns how to gradually remove this noise. The diffusion model applies the denoising process to a random seed to generate a realistic image. By simulating diffusion, the diffusion model generates one or more noisy images.

[0137] Once the diffusion model is trained, the diffusion model receives an input image and performs an inverse diffusion process on the initial image to generate a noisy image based on the initial image. In some embodiments, the diffusion module 208 performs inverse diffusion using DDIM inversion.

[0138] The diffusion model provides a noisy image to a first CNN with features and self-attention mechanism. The first CNN samples the input image and extracts features from the input image. The first CNN injects the extracted features and self-attention map directly into the second CNN. The first CNN performs forward diffusion of the noisy initial image, which is a process of progressively denoising the noisy image using sampling to output a denoised initial image.

[0139] The text request and the noisy image are provided as input to the second CNN. The second CNN uses a self-attention map to align the semantic features of the text request with the structure of the noisy image to generate a noisy transformed image. The second CNN performs forward diffusion of the noisy transformed image to output a denoised transformed image.

[0140] The denoised original image is combined with the denoised transformed image and the retained mask. This advantageously prevents modification of the face that might otherwise be modified in a way that produces unrealistic features. In some embodiments, the diffusion module 208 performs blending by using a mask smoothing algorithm and Poisson blending.

[0141] In some embodiments, the preservation mask includes other parts of the subject, such as: the subject's hair, if the user wants their hair to remain unchanged; the subject's fingers, because fingers are often modified by machine learning models in unrealistic ways; the subject's entire body (where the subject is a pet) to prevent the pet from being overly modified, etc. In some embodiments where the output image modifies the subject's clothes, the preservation mask can include everything except the subject's clothes, so that the body (minus the clothes) and the background of the original image are preserved.

[0142] The combined denoised image and retained mask are blended with the denoised transformed image to form an output image that satisfies the text request.

[0143] Example Flowchart

[0144] The media application 103 may include different ways to edit the original image. Figure 7 An example flow chart illustrating a method 700 of making modifications to an initial image according to some embodiments described herein. The method 700 may be performed by Figure 2 In some embodiments, the method 700 is performed by the user device 115, the media server 101, or partially on the user device 115 and partially on the media server 101.

[0145] Figure 7 The method 700 of can begin at block 705. At block 705, it is determined whether the user grants permission to access the initial image. If the user does not grant permission, the method 700 ends. If the user grants permission, block 705 can be followed by block 710.

[0146] At box 710, a request to modify the entirety of the initial image, a portion of the initial image, or a text request is received. Modifications to the entirety of the initial image may include, for example, a request to change the style of the initial image to look like an impressionist painting. Modifications to a portion of the initial image may include, for example, a request to move an object from one place to another, a request to remove power lines, etc. Modifications including text requests may involve specific objects in the image (e.g., a request to replace the subject's shirt with a jacket), involve creating new objects (e.g., a request to add a turtle to the initial image at the beach), or changes to the entire image (e.g., a request to change an outdoor scene from a daylight image to a moonlight image). Box 710 may be followed by box 715 for modifying the entire image, box 720 for modifying a portion of the image, or box 725 for text requests.

[0147] At block 715, in response to the request being to modify the entire image, a selection of a preset is received. The presets may include: changing an outdoor scene to a sunset, night, or cloudy scene, etc.; changing an initial image to a painting, surreal, nostalgic, etc.; changing a theme to a sea adventurer, an ancient warrior, a space crusader, a wise wizard, a noble, a space mission, etc. Block 715 may be followed by block 730.

[0148] At block 720, in response to modifying a portion of the image, a selection of a region is received. The region may include a group of objects (such as a sky with clouds) or a single object. The region may be selected by clicking a circle in a user interface, circling a region, tapping a region until the desired region is highlighted with an indicator, etc. Block 720 may be followed by block 730.

[0149] At block 725 , in response to the request to modify using the text request, an open text prompt is used. Block 725 may be followed by block 730 .

[0150] At block 730 , a modified image is generated. Block 730 may be followed by block 735 .

[0151] At block 735 , it is determined whether the user is satisfied with the modified image. If the user is not satisfied with the modified image, block 735 may be followed by block 740 .

[0152] At block 740 , in response to the user providing additional user input, the modified image is modified or refreshed. The loop from block 735 to block 740 is repeated until the user is satisfied with the modified image, at which point block 735 may be followed by block 745 .

[0153] At block 745, the modified image is saved.

[0154] FIG. 8A to FIG. 8B An example flow chart illustrating a method 800 for segmenting an initial image according to some embodiments described herein. The method 800 may be performed by Figure 2 In some embodiments, the method 800 is performed by the user device 115, the media server 101, or partially on the user device 115 and partially on the media server 101.

[0155] The method 800 of FIG. 8 may begin at block 802. At block 802, it is determined whether the user grants permission to access the initial image. If the user does not grant permission, the method 800 ends. If the user grants permission, block 802 may be followed by block 804.

[0156] At 810, object recognition is performed on the initial image to identify objects in the input image. In some embodiments, performing object recognition to identify objects in the initial image includes determining an object bounding box for each of the objects, and the method further includes determining that the user input corresponds to the selected object based on a proximity of the user input to the closest object bounding box. Block 810 is followed by block 815.

[0157] At block 815, it is determined whether the initial image is an indoor scene. If the initial image is an indoor scene, block 815 may be followed by block 820. If the initial image is not an indoor scene, block 815 may be followed by block 825.

[0158] At block 820 , a sky segment is determined from the initial image. Block 820 may be followed by block 825 .

[0159] At block 825, it is determined whether the initial image has a subject that is a human or an animal. If the initial image has a subject that is a human or an animal, block 825 may be followed by block 830. In some embodiments, the method further includes: in response to the initial image including the subject, generating background segments, wherein the subject segments are associated with the foreground area, the background segments are associated with the background area, and pixels in the initial image are associated with the foreground area or the background area; and determining that the user input corresponds to the foreground area based on the user input contacting the pixels associated with the foreground area. If the initial image does not have a subject that is a human or an animal, block 825 may be followed by block 835.

[0160] At block 830 , a subject segment is determined from the initial image. Block 830 may be followed by block 835 .

[0161] At block 835 , it is determined whether the initial image has one or more distracting objects. If the image does not have one or more distracting objects, block 835 may be followed by block 840 .

[0162] At block 840 , the selected object is segmented in response to receiving the user input.

[0163] If the initial image has one or more distracting objects, block 835 may be followed by Figure 8B Frame 845 in.

[0164] At block 845, in response to the initial image including one or more distracting objects, one or more distracting segments are determined from the initial image. In some embodiments, a convolutional neural network (CNN) performs segmentation, and method 800 further includes: providing the initial image and the heat map of the key points as input to the CNN; and using the convolutional neural network to output a segmentation mask corresponding to the sky segment, the subject segment, and the one or more distracting segments. Block 845 may be followed by block 850.

[0165] At block 850, a user interface including an initial image receives a user input corresponding to a selected object from a set of objects. The user input may include multiple taps on the selected object. In this case, method 800 may further include: determining a number of taps from the user input; and determining the selected object based on the number of taps, wherein a first tap is associated with a different area than a second tap.

[0166] In some embodiments, the user input includes a selection of a sky, and the method further includes: receiving a request from the user to change the lighting in the initial image; providing the initial image and the request to change the lighting in the initial image as input to the diffusion model; and using the diffusion model to output an output image that satisfies the request.

[0167] In some embodiments, the user input includes a selection of one or more background objects to be removed, and the method further includes: removing one or more distracting objects from the initial image based on object recognition; and generating a modified image including inpainted pixels associated with the one or more distracting portions.

[0168] In some embodiments, the selected object is an incomplete object, and the omitted portion of the incomplete object is truncated by a boundary of the initial image or occluded by another object, the method further includes: generating a segmentation mask including the incomplete object; removing the incomplete object from the initial image; generating an inpainted image that replaces incomplete object pixels corresponding to the incomplete object with background pixels that match the background in the initial image; providing the segmentation mask, the incomplete object, and the inpainted image as inputs to a diffusion model; using the diffusion model to output the complete object; and generating a modified image by blending one or more versions of the complete object with one or more versions of the inpainted image using the retained mask. Block 845 may be followed by block 855.

[0169] At block 855 , the user interface is updated to include an indication that the selected object is selected.

[0170] In some embodiments, the method further includes: receiving a text request to change a selected object in an initial image; determining a facial segment of a subject's face from the initial image based on the subject segment; generating a retention mask corresponding to the facial segment; providing the text request, the initial image, and the retention mask as input to a diffusion model; and using the diffusion model to output an output image that satisfies the text request.

[0171] In addition to the above description, controls may be provided to the user, which allow the user to make a choice about whether and when the system, program, or feature described herein can implement the collection of user information (e.g., information about the user's social network, social actions, or activities, occupation, user preferences, or the user's current location) and whether to send content or communications from the server to the user. In addition, before storing or using certain data, it can be processed in one or more ways to remove personally identifiable information. For example, the user's identity can be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location for which location information is obtained can be generalized (such as to a city, zip code, or state level) so that the user's specific location cannot be determined. Therefore, the user can control what information is collected about the user, how the information is used, and what information is provided to the user.

[0172] According to the above description, the media application performs object recognition on the initial image to identify a set of objects in the initial image. The media application determines whether the initial image is an outdoor scene. In response to the initial image being an outdoor scene, the media application determines a sky segment from the initial image. The media application determines whether the initial image includes a subject that is a human or an animal. In response to the initial image including a subject that is a human or an animal, the media application determines a subject segment from the initial image. The media application receives user input corresponding to a selection of a selected object from the set of objects at a user interface including the initial image. The media application updates the user interface to include an indication that the selected object is selected.

[0173] In the above description, for the purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding of this specification. However, it will be apparent to those skilled in the art that the present disclosure can be practiced without these specific details. In some cases, structures and devices are shown in block diagram form to avoid ambiguous descriptions. For example, the above may mainly describe embodiments with reference to user interfaces and specific hardware. However, embodiments may be applicable to any type of computing device that can receive data and commands and any peripheral device that provides services.

[0174] References in this specification to "some embodiments" or "some examples" mean that a particular feature, structure, or characteristic described in conjunction with the embodiments or examples may be included in at least one implementation described. The appearances of the phrase "in some embodiments" in various places in this specification are not necessarily all referring to the same embodiment.

[0175] Some portions of the above detailed description are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means by which those skilled in the art of data processing most effectively communicate the substance of their work to other skilled in the art. An algorithm is here and generally considered to be a self-consistent sequence of steps that produces a desired result. Steps are those steps that require physical manipulation of physical quantities. Although not necessary, these quantities are typically in the form of electrical or magnetic data that can be stored, transferred, combined, compared, and otherwise manipulated. It has been demonstrated that, primarily for reasons of common use, these data can sometimes be conveniently referred to as bits, values, elements, symbols, characters, terms, numbers, etc.

[0176] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specifically noted, as will become apparent from the following discussion, it should be understood that throughout the description, use of terms including "processing" or "computing" or "calculating" or "determining" or "displaying" refers to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities within the computer system's registers and memories and transforms the data into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission or display devices.

[0177] Embodiments of the present specification may also relate to a processor for performing one or more steps of the method described above. The processor may be a dedicated processor selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory computer-readable storage medium, which includes but is not limited to any type of disk, including an optical disk, ROM, CD-ROM, magnetic disk, RAM, EPROM, EEPROM, magnetic card or optical card, flash memory including USB key with non-volatile memory, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0178] This specification can take the form of some complete hardware embodiments, some complete software embodiments, or some embodiments containing both hardware and software elements. In some embodiments, this specification is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0179] Furthermore, the description may take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable medium may be any device that can contain, store, communicate, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or device.

[0180] A data processing system suitable for storing or executing program code will include at least one processor coupled directly or indirectly to a memory element via a system bus. The memory element may include local memory, mass storage devices, and cache memories employed during actual execution of the program code, which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from mass storage devices during execution.

Claims

1. A computer-implemented method comprising: performing object recognition on an initial image to identify a set of objects in the initial image; Determining whether the initial image is an outdoor scene; In response to the initial image being an outdoor scene, determining a sky segment from the initial image; determining whether the initial image includes a subject that is a human or an animal; In response to the initial image including the subject, determining a subject segment from the initial image; determining whether the initial image includes one or more interfering objects; In response to the initial image including one or more interfering objects, determining one or more interfering segments from the initial image; receiving, at a user interface including the initial image, a user input corresponding to a selection of a selected object from the set of objects; and The user interface is updated to include an indication that the selected object is selected.

2. The method of claim 1, wherein the user input comprises a plurality of taps on the selected object, and the method further comprises: determining a number of taps from the user input; as well as The selected object is determined based on a number of the taps, wherein a first tap is associated with a different area than a second tap.

3. The method of claim 1, further comprising: In response to the initial image including the subject, generating a background segment, wherein the subject segment is associated with a foreground region, the background segment is associated with a background region, and pixels in the initial image are associated with the foreground region or the background region; as well as The user input is determined to correspond to the foreground region based on the user input contacting a pixel associated with the foreground region.

4. The method of claim 1 , wherein performing object recognition to identify objects in the initial image comprises determining an object bounding box for each of the objects, and the method further comprises: The user input is determined to correspond to the selected object based on a proximity of the user input to a closest object bounding box.

5. The method of claim 1, wherein a convolutional neural network (CNN) performs segmentation, and the method further comprises: Providing the initial image and the heat map of key points as input to the CNN; as well as The convolutional neural network is utilized to output a segmentation mask corresponding to the sky segment, the subject segment, and the one or more interference segments.

6. The method of claim 1, wherein the user input comprises a selection of the sky, and the method further comprises: receiving a request from a user to change lighting in the initial image; providing as input to a diffusion model an initial image and a request to change illumination in the initial image; as well as The diffusion model is used to output an output image that satisfies the request.

7. The method of claim 1, wherein the user input comprises a selection of one or more background objects to be removed, and the method further comprises: Based on object recognition, removing the one or more interfering objects from the initial image; as well as A modified image is generated that includes an inpainting of pixels associated with the one or more interfering objects.

8. The method of claim 1, wherein the selected object is an incomplete object, and the omitted portion of the incomplete object is truncated by a boundary of the initial image or occluded by another object, the method further comprising: generating a retention mask including the incomplete object; removing the incomplete object from the initial image; generating an inpainted image, the inpainted image replacing incomplete object pixels corresponding to the incomplete object with background pixels matching the background in the initial image; providing the retained mask, the incomplete object, and the inpainted image as input to a diffusion model; outputting a complete object using the diffusion model; as well as A modified image is generated by blending one or more versions of the intact object with one or more versions of the inpainted image using the preservation mask.

9. The method of claim 1, further comprising: receiving a text request to change the selected object in the initial image; determining a facial segment of the subject's face from the initial image based on the subject segment; generating a preservation mask corresponding to the facial segment; providing the text request, the initial image, and the retention mask as input to a diffusion model; as well as The diffusion model is used to output an output image that satisfies the text request.

10. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: performing object recognition on an initial image to identify a set of objects in the initial image; Determining whether the initial image is an outdoor scene; In response to the initial image being an outdoor scene, determining a sky segment from the initial image; determining whether the initial image includes a subject that is a human or an animal; In response to the initial image including the subject, determining a subject segment from the initial image; determining whether the initial image includes one or more interfering objects; In response to the initial image including one or more interfering objects, determining one or more interfering segments from the initial image; receiving, at a user interface including the initial image, a user input corresponding to a selection of a selected object from the set of objects; and The user interface is updated to include an indication that the selected object is selected.

11. The non-transitory computer readable medium of claim 10, wherein the user input comprises a plurality of taps on the selected object, and the operations further comprise: determining a number of taps from the user input; as well as The selected object is determined based on a number of the taps, wherein a first tap is associated with a different area than a second tap.

12. The non-transitory computer readable medium of claim 10, wherein the operations further comprise: In response to the initial image including the subject, generating a background segment, wherein the subject segment is associated with a foreground region, the background segment is associated with a background region, and pixels in the initial image are associated with the foreground region or the background region; as well as The user input is determined to correspond to the foreground region based on the user input contacting a pixel associated with the foreground region.

13. The non-transitory computer readable medium of claim 10, wherein performing object recognition to identify objects in the initial image comprises determining an object bounding box for each of the objects, and the operations further comprise: The user input is determined to correspond to the selected object based on a proximity of the user input to a closest object bounding box.

14. The non-transitory computer readable medium of claim 10, wherein a convolutional neural network (CNN) performs segmentation, and the operations further comprise: Providing the initial image and the heat map of key points as input to the CNN; as well as The convolutional neural network is utilized to output a segmentation mask corresponding to the sky segment, the subject segment, and the one or more interference segments.

15. The non-transitory computer readable medium of claim 10, wherein the user input comprises a selection of the sky, and the operations further comprise: receiving a request from a user to change lighting in the initial image; providing as input to a diffusion model an initial image and a request to change illumination in the initial image; as well as The diffusion model is used to output an output image that satisfies the request.

16. A system comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored thereon, the instructions, when executed by the processor, causing the processor to perform operations, the operations comprising: performing object recognition on an initial image to identify a set of objects in the initial image; Determining whether the initial image is an outdoor scene; In response to the initial image being an outdoor scene, determining a sky segment from the initial image; determining whether the initial image includes a subject that is a human or an animal; In response to the initial image including the subject, determining a subject segment from the initial image; determining whether the initial image includes one or more interfering objects; In response to the initial image including one or more interfering objects, determining one or more interfering segments from the initial image; receiving, at a user interface including the initial image, a user input corresponding to a selection of a selected object from the set of objects; and The user interface is updated to include an indication that the selected object is selected.

17. The system of claim 16, wherein the user input comprises a plurality of taps on the selected object, and the operations further comprise: determining a number of taps from the user input; as well as The selected object is determined based on a number of the taps, wherein a first tap is associated with a different area than a second tap.

18. The system of claim 16, wherein the operations further comprise: In response to the initial image including the subject, generating a background segment, wherein the subject segment is associated with a foreground region, the background segment is associated with a background region, and pixels in the initial image are associated with the foreground region or the background region; as well as The user input is determined to correspond to the foreground region based on the user input contacting a pixel associated with the foreground region.

19. The system of claim 16, wherein performing object recognition to identify objects in the initial image comprises determining an object bounding box for each of the objects, and the operations further comprise: The user input is determined to correspond to the selected object based on a proximity of the user input to a closest object bounding box.

20. The system of claim 16, wherein a convolutional neural network (CNN) performs segmentation, and the operations further comprise: Providing the initial image and the heat map of key points as input to the CNN; as well as The convolutional neural network is utilized to output a segmentation mask corresponding to the sky segment, the subject segment, and the one or more interference segments.

Images