Relocating, replacing and generating objects in image

Abstract

A media application receives a selection of an incomplete object in an initial image. The media application generates an object mask that includes incomplete object pixels associated with the incomplete object. The media application removes incomplete object pixels associated with the incomplete object from the initial image. The media application generates a repaired image that replaces incomplete object pixels corresponding to the incomplete object with the repaired pixels. The media application outputs the complete object. The media application outputs a modified image by fusing the one or more versions of the complete object with the one or more versions of the repaired image using the object mask.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 465,230, entitled “Repositioning Objects in an Image,” filed on May 9, 2023, and U.S. Provisional Patent Application No. 63 / 562,634, entitled “Performing Scene Impact Editing Tasks Using Diffusion Neural Networks,” filed on March 7, 2024, each of which is incorporated herein in its entirety. Background Art

[0002] A user may capture an image in which an object is in an unexpected location. For example, an object may be cut off by the border of an image, cut off by another object, etc. Technology exists for moving objects within an image; however, attempting to move an object to a different location in an image may have disastrous results. For example, pixels associated with an object may be incorrectly identified, causing a portion of the object to remain in the original location while the remainder of the object is moved to a different location (e.g., a chicken's body is moved while the chicken's feet remain). In another example, the empty space caused by the removal of pixels associated with the moved object may be filled with pixels that appear inconsistent. In yet another example, the pixels around the moved object may appear different from the background and result in an image that appears to be poorly edited.

[0003] The purpose of the background description provided herein is to generally present the context of the present disclosure. The work of the presently named inventors (to the extent that it is described in this background section) and aspects of the specification that might not have been considered prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art to the present disclosure. Summary of the invention

[0004] A computer-implemented method includes receiving a selection of an incomplete object in an initial image, wherein the incomplete object is associated with a first position within the initial image, and a missing portion of the incomplete object is truncated by a boundary of the initial image or occluded by another object. The method further includes generating an object mask including incomplete object pixels associated with the incomplete object. The method further includes removing the incomplete object pixels associated with the incomplete object from the initial image. The method further includes generating an inpainted image, the inpainted image replacing the incomplete object pixels corresponding to the incomplete object with the inpainted pixels. The method further includes providing the object mask, the incomplete object, and the inpainted image as inputs to a diffusion model. The method further includes outputting a complete object using the diffusion model. The method further includes generating a modified image by fusing one or more versions of the complete object with one or more versions of the inpainted image using the object mask, wherein the complete object is located at a second position in the modified image that is different from the first position in the initial image.

[0005] In some embodiments, the modified image is a first modified image, and the method further includes receiving a request to remove the selected object from the first modified image; providing the first modified image and the request as input to an object removal model associated with the diffusion model; and outputting, using the object removal model, a second modified image that does not include the selected object. In some embodiments, the modified image is a first modified image, and the method further includes receiving a request to move the selected object from a third position to a fourth position; providing the first modified image and the request as input to an object insertion model associated with the diffusion model; and outputting, using the object insertion model, a second modified image that includes the selected object at the fourth position based on the request.

[0006] In some embodiments, the modified image is a first modified image, and the method further includes receiving a request to add an additional object to the initial image; outputting the additional object using a diffusion model; and outputting a second modified image using the diffusion model by fusing one or more versions of the additional object with one or more versions of the inpainted image. In some embodiments, the request to add the additional object includes a textual prompt describing the additional object.

[0007] In some embodiments, the complete object is resized based on a change from a first position in the initial image to a second position in the modified image. In some embodiments, the method further includes receiving a command to uncrop the modified image to extend the uncropped boundary of the modified image to the extended boundary; and outputting an uncropped image, the uncropped image including the repaired pixels between the uncropped boundary of the modified image and the extended boundary based on the command. In some embodiments, the command to uncrop the repaired image includes: selection of an uncrop button; and a command to directly extend the uncropped boundary of the modified image to the extended boundary, or a movement of the complete object to extend the uncropped boundary of the modified image to the extended boundary. In some embodiments, the method further includes modifying the lighting of the modified image; and adding a shadow to the complete object based on the direction of the lighting of the modified image.

[0008] In some embodiments, a non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations. The operations include receiving a selection of an incomplete object in an initial image, wherein the incomplete object is associated with a first position within the initial image, and a missing portion of the incomplete object is truncated by a boundary of the initial image or occluded by another object; generating an object mask including incomplete object pixels associated with the incomplete object; removing the incomplete object pixels associated with the incomplete object from the initial image; generating an inpainted image that replaces the incomplete object pixels corresponding to the incomplete object with the inpainted pixels; providing the object mask, the incomplete object, and the inpainted image as inputs to a diffusion model; and outputting a modified image by fusing one or more versions of the complete object with one or more versions of the inpainted image using the diffusion model using the object mask, wherein the complete object is positioned at a second position in the modified image that is different from the first position in the initial image.

[0009] In some embodiments, the modified image is a first modified image, and the operation further includes receiving a request to remove the selected object from the first modified image; providing the first modified image and the request as input to an object removal model associated with the diffusion model; and outputting, using the object removal model, a second modified image that does not include the selected object. In some embodiments, the modified image is a first modified image, and the operation further includes receiving a request to move the selected object from a third position to a fourth position; providing the first modified image and the request as input to an object insertion model associated with the diffusion model; and outputting, using the object insertion model, a second modified image that includes the selected object at the fourth position based on the request.

[0010] In some embodiments, the modified image is a first modified image, and the operation further includes receiving a request to add an additional object to the initial image, the request including a textual prompt describing the additional object; outputting the additional object using a diffusion model; and outputting a second modified image using the diffusion model by fusing one or more versions of the additional object with one or more versions of the inpainted image. In some embodiments, the complete object is resized based on a change from a first position in the initial image to a second position in the modified image. In some embodiments, the operation further includes receiving a command to inverse crop the modified image to extend an uncropped boundary of the modified image to an extended boundary; and outputting an inverse cropped image including inpainted pixels between an uncropped boundary of the modified image and an extended boundary based on the command.

[0011] In some embodiments, a system includes a processor and a memory coupled to a processor, the memory having instructions stored thereon, which when executed by the processor causes the processor to perform operations. The operations include receiving a selection of an incomplete object in an initial image, wherein the incomplete object is associated with a first location within the initial image, and a missing portion of the incomplete object is truncated by a boundary of the initial image or occluded by another object; generating an object mask including incomplete object pixels associated with the incomplete object; removing the incomplete object pixels associated with the incomplete object from the initial image; generating an inpainted image, the inpainted image replacing the incomplete object pixels corresponding to the incomplete object with the inpainted pixels; providing the object mask, the incomplete object, and the inpainted image as inputs to a diffusion model; and outputting a modified image by fusing one or more versions of the complete object with one or more versions of the inpainted image using the diffusion model using the object mask, wherein the complete object is located at a second location in the modified image that is different from the first location in the initial image.

[0012] In some embodiments, the modified image is a first modified image, and the operation further includes receiving a request to remove the selected object from the first modified image; providing the first modified image and the request as input to an object removal model associated with the diffusion model; and outputting, using the object removal model, a second modified image that does not include the selected object. In some embodiments, the modified image is a first modified image, and the operation further includes receiving a request to move the selected object from a third position to a fourth position; providing the first modified image and the request as input to an object insertion model associated with the diffusion model; and outputting, using the object insertion model, a second modified image that includes the selected object at the fourth position based on the request.

[0013] In some embodiments, the modified image is a first modified image, and the operation further includes receiving a request to add an additional object to the initial image, the request including a textual prompt describing the additional object; outputting the additional object using a diffusion model; and outputting a second modified image using the diffusion model by fusing one or more versions of the additional object with one or more versions of the inpainted image. In some embodiments, the complete object is resized based on a change from a first position in the initial image to a second position in the modified image. In some embodiments, the operation further includes receiving a command to inverse crop the modified image to extend an uncropped boundary of the modified image to an extended boundary; and outputting an inverse cropped image including inpainted pixels between an uncropped boundary of the modified image and an extended boundary based on the command. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a block diagram of an example network environment according to some embodiments described herein.

[0015] Figure 2 is a block diagram of an example computing device according to some embodiments described herein.

[0016] Figure 3A An example initial image is illustrated according to some embodiments described herein.

[0017] Figure 3B Examples of some embodiments described herein wherein Figure 3A Two objects are selected for modification of the example initial image.

[0018] Figure 3C An example initial image is illustrated in which a mask surrounds an object according to some embodiments described herein.

[0019] Figure 3D An example inpainted image is illustrated in which bystander objects are removed and subject objects are moved and resized according to some embodiments described herein.

[0020] Figure 3E An example inpainted image is illustrated in which a road is replaced with grass, a first type of trees is replaced with a second type of trees, and a cloudy sky is replaced with a clear sky in accordance with some embodiments described herein.

[0021] Figure 4A An example initial image illustrating a child sitting on a bench and holding a balloon, which is partially cut off by a boundary of the initial image, is illustrated in accordance with some embodiments described herein.

[0022] Figure 4BAn example modified image is illustrated in which a child, a bench, and a balloon are moved to a second position in accordance with some embodiments described herein.

[0023] Figure 5A An example user interface illustrating an initial image including buttons for changing the boundaries of the initial image according to some embodiments described herein.

[0024] Figure 5B An example user interface with an indicator for expanding the boundaries of an initial image is illustrated according to some embodiments described herein.

[0025] Figure 5C An example user interface illustrating an inversely cropped image that is output based on an initial image according to some embodiments described herein.

[0026] Figure 5D An alternative example user interface is illustrated in which a selected object is used to extend the boundaries of an initial image in accordance with some embodiments described herein.

[0027] Figure 6 An example flow chart illustrating a method for generating a modified image of a complete object from an incomplete object according to some embodiments described herein.

[0028] Figure 7 An example flow chart illustrating a method for outputting an inverse cropped image from an initial image according to some embodiments described herein.

[0029] Figure 8 An example flow chart illustrating a method for training an object removal model according to some embodiments described herein.

[0030] Fig. 9 An example flow chart illustrating a method for training an object insertion model according to some embodiments described herein. DETAILED DESCRIPTION

[0031] A user may capture an image in which an object is in an unexpected location. For example, an object may be cut off by the border of an image, cut off by another object, etc. Technology exists for moving objects within an image; however, attempting to move an object to a different location in an image may have disastrous results. For example, pixels associated with an object may be incorrectly identified, causing a portion of the object to remain in the original location while the remainder of the object is moved to a different location (e.g., a chicken's body is moved while the chicken's feet remain). In another example, the empty space caused by the removal of pixels associated with the moved object may be filled with pixels that appear inconsistent. In yet another example, the pixels around the moved object may appear different from the background and result in an image that appears to be poorly edited.

[0032] The techniques described below advantageously address these issues by providing an incomplete object as input to a diffusion machine learning model (referred to herein as a diffusion model) and outputting a complete object. An incomplete object is a partial representation of an object in an image. An object is partially (but not completely) present in the image. The portion of an object that is not present in the image is referred to herein as the "missing portion" of the incomplete object. A user can select an incomplete object in the initial image and move the position of the incomplete object.

[0033] The space left by the incomplete object is inpainted with inpainted pixels to form an inpainted image. A complete object is a complete representation of the object including the incomplete object and the missing portions of the incomplete object. The inpainted image is an image that is different from the initial image in that the incomplete object pixels associated with the incomplete object are removed from the inpainted image and the incomplete object is replaced with inpainted pixels, which may be selected based on proximity to surrounding pixels, selected from a reference image including background pixels, etc.

[0034] The diffusion model ensures that the complete object fits the new position. For example, if an object is moved from the background to the foreground, the diffusion model increases the size of the moved object. The diffusion model outputs a modified image in which the complete object is seamlessly merged with the restored image by fusing one or more versions of the complete object with one or more versions of the restored image using an object mask. For example, the diffusion model can fuse a version of the complete object with gradually increasing noise with a corresponding noisy version of the restored image, while also generating a denoised version of the complete object and a corresponding denoised version of the restored image. A noisy version of the complete object is created by increasing the entropy of the image, where more noise makes the details of the complete object less discernible in the image. Similarly, a noisy version of the restored image is created by increasing the entropy of the restored image, where more noise makes the details of the restored image less discernible.

[0035] By employing a diffusion model rather than other machine learning models, media applications maintain a realistic appearance of modified images in a wide variety of situations. The techniques described below enable correction of imperfect images, i.e., images that include incomplete objects, in an efficient manner. The complete objects created by utilizing the diffusion model are of high quality and free of the errors described above. The image processing described herein effectively and efficiently corrects images with respect to incomplete objects present in the images.

[0036] Sample Environment 100

[0037] Figure 11 is a block diagram illustrating an example environment 100. In some embodiments, the environment 100 includes a media server 101 coupled to a network 105, a user device 115a, and a user device 115n. Users 125a, 125n may be associated with respective user devices 115a, 115n. In some embodiments, the environment 100 may include Figure 1 Other servers or devices not shown in FIG. Figure 1 In the illustrative drawings and the remaining drawings, a letter following a reference number (e.g., "115a") indicates a reference to the element having that specific reference number. A reference number in the text without a following letter (e.g., "115") indicates a general reference to an embodiment of the element with that reference number.

[0038] The media server 101 may include a processor, a memory, and network communication hardware. In some embodiments, the media server 101 is a hardware server. The media server 101 is communicatively coupled to the network 105 via a signal line 102. The signal line 102 may be a wired connection (such as Ethernet, coaxial cable, fiber optic cable, etc.) or a wireless connection (such as Wi-Fi®, Bluetooth® or other wireless technologies). In some embodiments, the media server 101 sends data to one or more user devices of the user devices 115a, 115n via the network 105 and receives data from the one or more user devices. The media server 101 may include a media application 103a and a database 199.

[0039] The database 199 may store machine learning models, training data sets, images, etc. The database 199 may also store social network data associated with the user 125, user preferences of the user 125, etc.

[0040] User device 115 may be a computing device including a memory coupled to a hardware processor. For example, user device 115 may include a mobile device, a tablet computer, a mobile phone, a wearable device, a head mounted display, a mobile email device, a portable game player, a portable music player, a reader device, or another electronic device capable of accessing network 105.

[0041] In the illustrated embodiment, user device 115a is coupled to network 105 via signal line 108, and user device 115n is coupled to network 105 via signal line 110. Media application 103 can be stored on user device 115a as media application 103b and / or stored on user device 115n as media application 103c. Signal lines 108 and 110 can be wired connections (such as Ethernet, coaxial cable, fiber optic cable, etc.) or wireless connections (such as Wi-Fi®, Bluetooth® or other wireless technologies). User devices 115a, 115n are accessed by users 125a, 125n, respectively. By way of example, Figure 1 Although Figure 1 Two user devices, 115 a and 115 n , are illustrated, but the present disclosure is applicable to system architectures having one or more user devices 115 .

[0042] The media application 103 may be stored on the media server 101 or the user device 115. In some embodiments, the operations described herein are performed on the media server 101 or the user device 115. In some embodiments, some operations may be performed on the media server 101 and some operations may be performed on the user device 115. The execution of the operations is based on user settings. For example, the user 125a may specify a setting that an operation is to be performed on its corresponding device 115a instead of on the media server 101. With such settings, the operations described herein are performed entirely on the user device 115a, and no operations are performed on the media server 101. In addition, the user 125a may specify that the user's images and / or other data will only be stored locally on the user device 115a and not on the media server 101. With such settings, no user data is transmitted to or stored on the media server 101. The transmission of user data to the media server 101, any temporary or permanent storage of such data by the media server 101, and the performance of operations on such data by the media server 101 are only performed if the user has consented to the transmission, storage and performance of operations by the media server 101. The user is provided with the option to change the settings at any time, for example so that they can enable or disable the use of the media server 101.

[0043] Machine learning models (e.g., diffusion models, neural networks, or other types of models), if used for one or more operations, are stored and utilized locally on the user device 115 with specific user permission. Server-side models are used only with user permission. In addition, trained models can be provided for use on the user device 115. During such use, if the user 125 permits, on-device training of the model can be performed. If the user 125 permits, updated model parameters can be transmitted to the media server 101, for example, to implement federated learning. The model parameters do not include any user data.

[0044] The media application 103 receives an initial image. For example, the media application 103 receives the initial image from a camera that is part of the user device 115, or the media application 103 receives the initial image over the network 105. The media application 103 receives a selection of an incomplete object in the initial image. The incomplete object is associated with a first position within the initial image, and a missing portion of the incomplete object is truncated by a boundary of the initial image or is obscured by another object. The incomplete object can be selected when the user 125 clicks on the object, draws a shape (e.g., a circle) around the object, confirms a suggestion made by the media application 103 to modify the object, etc.

[0045] The media application 103 generates an object mask including incomplete object pixels associated with the incomplete object and removes the incomplete object pixels associated with the incomplete object from the initial image. The media application 103 generates an inpainted image that replaces the incomplete object pixels corresponding to the incomplete object with the inpainted pixels.

[0046] The media application 103 outputs a complete object using a diffusion model. For example, where an incomplete object is truncated by an edge of an initial image and the incomplete object is moved to the center of the initial image, the diffusion model outputs a complete object that fills in the missing portion of the incomplete object. The media application 103 outputs a modified image using a diffusion model by fusing one or more versions of the complete object with one or more versions of the repaired image using an object mask, wherein the complete object is positioned at a second location in the modified image that is different from the first location in the initial image. In some embodiments, the modified image may include a watermark or other indicator to identify that the modified image was generated using a machine learning model.

[0047] In some embodiments, media application 103 can be implemented using hardware, including a central processing unit (CPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a machine learning processor / coprocessor, any other type of processor, or a combination thereof. In some embodiments, media application 103a can be implemented using a combination of hardware and software.

[0048] Example computing device 200

[0049] Figure 2 is a block diagram of an example computing device 200 that can be used to implement one or more features described herein. The computing device 200 can be any suitable computer system, server, or other electronic or hardware device. In one example, the computing device 200 is a media server 101 for implementing a media application 103a. In another example, the computing device 200 is a user device 115.

[0050] In some embodiments, computing device 200 includes processor 235, memory 237, input / output (I / O) interface 239, display 241, camera 243, and storage device 245, all coupled via bus 218. Processor 235 may be coupled to bus 218 via signal line 222, memory 237 may be coupled to bus 218 via signal line 224, I / O interface 239 may be coupled to bus 218 via signal line 226, display 241 may be coupled to bus 218 via signal line 228, camera 243 may be coupled to bus 218 via signal line 230, and storage device 245 may be coupled to bus 218 via signal line 232.

[0051] The processor 235 may be one or more processors and / or processing circuits for executing program code and controlling the basic operation of the computing device 200. A "processor" includes any suitable hardware system, mechanism or component for processing data, signals or other information. The processor may include a system having: a general-purpose central processing unit (CPU) with one or more cores (e.g., in a single-core, dual-core or multi-core configuration), multiple processing units (e.g., in a multi-processor configuration), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), a dedicated circuit system for implementing functionality, a dedicated processor for implementing processing based on a neural network model, a neural circuit, a processor optimized for matrix calculations (e.g., matrix multiplication), or other systems. In some embodiments, the processor 235 may include one or more coprocessors for implementing neural network processing. In some embodiments, the processor 235 may be a processor that processes data to generate a probabilistic output, for example, the output generated by the processor 235 may be imprecise, or may be accurate within a range from the expected output. Processing does not have to be limited to a specific geographic location or have time constraints. For example, the processor may perform its functions in real time, offline, in batch mode, etc. Portions of the processing may be performed at different times and in different locations by different (or the same) processing systems.A computer may be any processor in communication with a memory.

[0052] The memory 237 is provided in the computing device 200 for access by the processor 235 and may be any suitable processor-readable storage medium suitable for storing instructions for execution by the processor or a set of processors and located separately from and / or integrated with the processor 235, such as a random access memory (RAM), a read-only memory (ROM), an electrically erasable read-only memory (EEPROM), flash memory, etc. The memory 237 may store software operated by the processor 235 on the computing device 200, including the media application 103.

[0053] The memory 237 may include an operating system 262, other applications 264, and application data 266. The other applications 264 may include, for example, an image library application, an image management application, an image gallery application, a communication application, a web hosting engine or application, a media sharing application, etc. One or more methods disclosed herein may operate in a number of environments and platforms, such as as a stand-alone computer program that can run on any type of computing device, as a web application with a web page, as a mobile application ("app") running on a mobile computing device, etc.

[0054] Application data 266 may be data generated by other applications 264 or hardware of computing device 200. For example, application data 266 may include images used by an image library application, user actions recognized by other applications 264 (eg, a social networking application), and the like.

[0055] The I / O interface 239 may provide functionality to enable the computing device 200 to interface with other systems and devices. The interfaced device may be included as part of the computing device 200, or may be separate and in communication with the computing device 200. For example, a network communication device, a storage device (e.g., memory 237 and / or storage device 245), and an input / output device may communicate through the I / O interface 239. In some embodiments, the I / O interface 239 may be connected to an interface device, such as an input device (keyboard, pointing device, touch screen, microphone, scanner, sensor, etc.) and / or an output device (display device, speaker device, printer, monitor, etc.).

[0056] Some examples of docked devices that can be connected to I / O interface 239 can include a display 241, which can be used to display content (e.g., images, videos, and / or user interfaces of output applications as described herein), and to receive touch (or gesture) input from a user. For example, display 241 can be used to display a user interface including a graphical guide on a viewfinder. Display 241 can include any suitable display device, such as a liquid crystal display (LCD), a light emitting diode (LED), or a plasma display screen, a cathode ray tube (CRT), a television, a monitor, a touch screen, a three-dimensional display screen, or other visual display device. For example, display 241 can be a flat display screen provided on a mobile device, multiple display screens embedded in a glasses form factor or a head-mounted device, or a monitor screen for a computer device.

[0057] Camera 243 can be any type of image capture device capable of capturing images and / or videos. In some embodiments, camera 243 captures images or videos, and I / O interface 239 transmits the images or videos to media application 103.

[0058] The storage device 245 stores data related to the media application 103. For example, the storage device 245 may store a training data set including labeled images, a machine learning model, outputs from the machine learning model, and the like.

[0059] Figure 2 The example media application 103 stored in the memory 237 is illustrated and includes a user interface module 202 , a segmenter 204 , a repairer module 206 , and a diffusion module 208 .

[0060] The user interface module 202 generates graphical data for displaying a user interface including an image. In some embodiments, the user interface module 202 receives an initial image. The initial image can be received from the camera 243 of the computing device 200 or from the media server 101 via the I / O interface 239. The initial image includes a subject, such as a person. In some embodiments, the user interface includes options for selecting various people and other objects in the initial image. For example, a user can select a person by tapping on the person, circling the object, brushing the object, etc. In some embodiments, the user interface generates a recommendation for modifying the image, such as displaying text asking the user whether he or she wishes to move a bystander away from the object.

[0061] In some embodiments, once the user selects an object, the user interface module 202 updates the graphical data to include a highlighted version of the selected object. The user can modify the selected object. For example, the user can drag and drop the selected object from a first location to a second location, the user can resize the image, the user can select a button to erase the image, etc.

[0062] In some embodiments, the segmenter 204 generates a segmentation score that reflects the recognition quality of the pixels associated with the selected object in the initial image. The user interface may include different options for modifying the selected object based on the segmentation score. For example, if the segmentation score exceeds a threshold, the user interface module 202 provides options for moving the selected object, replacing the selected object with a different object, or erasing the selected object. In another example, if the segmentation score does not exceed the threshold, the user interface module 202 does not provide an option for moving the selected object, but provides an option for replacing the selected object or erasing the selected object.

[0063] In some embodiments, the user interface module 202 generates graphical data for displaying a restored image in which a selected object is moved from a first position within the image to a second position, the selected image is resized, additional objects are added, etc. The user interface may also include options for editing the restored image, sharing the restored image, adding the restored image to an album, etc.

[0064] The segmenter 204 segments the selected object from the initial image by identifying pixels corresponding to the selected object. In some embodiments, the segmenter 204 uses an alpha map as part of a technique for distinguishing the foreground and background of the initial image during segmentation. The segmenter 204 may also identify the texture of the selected object in the foreground of the initial image. In some embodiments, the segmenter 204 generates a segmentation map that identifies pixels associated with one or more objects in the initial image. For example, the segmentation map may include an identification of pixels associated with the selected object.

[0065] The segmenter 204 can perform segmentation by detecting objects in the initial image. The object can be a person, an animal, a car, a building, etc. The person can be the subject of the initial image, or not the subject of the initial image (i.e., a bystander). The bystander can include a person walking, running, riding a bicycle, standing behind the subject, or otherwise within the initial image. In different examples, the bystander can be in the foreground (e.g., a person passing in front of the camera), at the same depth as the subject (e.g., a person standing to the side of the subject), or in the background. In some examples, there may be more than one bystander in the initial image. The bystander can be a human being in any posture, for example, standing, sitting, squatting, lying, jumping, etc. The bystander can face the camera, can be at a certain angle to the camera, or can have his back to the camera.

[0066] The segmenter 204 may detect the type of object by performing object recognition, comparing the object to object priors of people, vehicles, buildings, etc. to identify the expected shape of the object, so as to determine whether the pixel is associated with the selected object or the background. The segmenter 204 may generate a region of interest, such as a bounding box with x, y coordinates and a scale, for the selected object.

[0067] The segmenter 204 generates one or more object masks for one or more selected objects in the initial image. The object mask represents the region of interest. The object mask will be described in more detail below with reference to the diffusion model.

[0068] In some embodiments, one or more object masks are generated based on generating superpixels of an image and matching superpixel centroids to depth map values ​​(e.g., obtained by camera 243 using a depth sensor or by deriving depth from pixel values) to cluster detections based on depth. More specifically, depth values ​​in masked areas can be used to determine a depth range, and superpixels that fall within the depth range can be identified. Another technique for generating a mask includes weighting depth values ​​based on their proximity to an object mask, where the weights are represented by a distance transform map.

[0069] In some embodiments, segmenter 204 uses a machine learning algorithm (such as a neural network or more specifically, a convolutional neural network) to segment the initial image and generate an object mask. Segmenter 204 can specify a circuit configuration (e.g., for a programmable processor, for a field programmable gate array (FPGA), etc.) that enables processor 235 to apply a machine learning model. In some embodiments, segmenter 204 can include software instructions, hardware instructions, or a combination. In some embodiments, segmenter 204 can provide an application programming interface (API) that can be used by operating system 262 and / or other applications 264 to call segmenter 204, for example, to apply a machine learning model to application data 266 to output an object mask.

[0070] The segmenter 204 uses the training data to generate a trained machine learning model. For example, the training data may include pairs of an initial image with one or more objects and an output image with one or more corresponding object masks.

[0071] The training data may be obtained from any source, such as a data repository specifically labeled for training, data for which permission is provided for use as training data for machine learning, etc. In some embodiments, training may occur on a media server 101 that provides the training data directly to the user device 115, training may occur locally on the user device 115, or a combination of both.

[0072] In some embodiments, the segmenter 204 uses weights that are obtained from another application and not edited / transferred. For example, in these embodiments, the trained model can be generated, for example, on a different device, and provided as part of the segmenter 204. In various embodiments, the trained model can be provided as a data file that includes a model structure or form (e.g., the model structure or form defines the number and type of neural network nodes, the connectivity between nodes, and the organization of nodes into multiple layers) and associated weights. The segmenter 204 can read the data file of the trained model and implement a neural network with node connectivity, layers, and weights based on the model structure or form specified in the trained model.

[0073] The trained machine learning model may include one or more model forms or structures. For example, the model form or structure may include any type of neural network, such as a linear network, a deep learning neural network that implements multiple layers (e.g., a "hidden layer" between an input layer and an output layer, where each layer is a linear network), a convolutional neural network (e.g., a network that splits or partitions input data into multiple parts or tiles, uses one or more neural network layers to process each tile separately, and aggregates the results from the processing of each tile), a sequence-to-sequence neural network (e.g., a network that receives sequential data such as words in a sentence, frames in a video, etc. as input and produces a sequence of results as output), etc.

[0074] The model form or structure can specify the connectivity between various nodes and the organization of nodes into layers. For example, the nodes of the first layer (e.g., the input layer) can receive data as input data or application data. For example, when the trained model is used for analysis of, for example, an initial image, such data can include, for example, one or more pixels per node. Subsequent intermediate layers can receive the output of the nodes of the previous layer as input according to the connectivity specified in the model form or structure. These layers may also be referred to as hidden layers. For example, the first layer can output the segmentation between the foreground and the background. The final layer (e.g., the output layer) produces the output of the machine learning model. For example, the output layer can receive the segmentation that divides the initial image into the foreground and the background, and the output pixel is part of the object mask or the rest of the initial image. In some embodiments, the model form or structure also specifies the number and / or type of nodes in each layer.

[0075] In various embodiments, the trained model may include one or more models. One or more models in the model may include multiple nodes, which are arranged in layers according to the model structure or form. In some embodiments, the node may be a computing node without memory, for example, configured to process an input unit to generate an output unit. The calculation performed by the node may include, for example, multiplying each node input in multiple node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum using a bias or intercept value to generate a node output. In some embodiments, the calculation performed by the node may also include applying a step / activation function to the adjusted weighted sum. In some embodiments, the step / activation function may be a nonlinear function. In various embodiments, such calculations may include operations such as matrix multiplication. In some embodiments, the calculations performed by multiple nodes may be performed in parallel, for example, using multiple processor cores of a multi-core processor, using individual processing units of a graphics processing unit (GPU), or a dedicated neural circuit system. In some embodiments, the node may include memory, for example, it may be possible to store one or more earlier inputs and use the one or more earlier inputs to process subsequent inputs. For example, a node with memory may include a long short-term memory (LSTM) node. LSTM nodes can use memory to maintain the “state” that allows the node to act like a finite state machine (FSM).

[0076] In some embodiments, a trained model may include embeddings or weights for individual nodes. For example, a model may start as a plurality of nodes organized into layers as specified by the model form or structure. Upon initialization, corresponding weights may be applied to connections between each pair of nodes connected in the model form (e.g., nodes in successive layers of a neural network). For example, corresponding weights may be randomly assigned, or initialized to default values. The model may then be trained, for example using training data, to produce results.

[0077] Training can include applying supervised learning techniques. In supervised learning, the training data can include multiple inputs (e.g., initial images, object masks, object masks, etc.) and corresponding true value outputs for each input (e.g., true value masks that correctly identify objects in each image). Based on a comparison of the output of the model with the true value output, the value of the weights is automatically adjusted, for example, in a manner that increases the probability that the model produces a true value output for an image.

[0078] In various embodiments, the trained model includes a set or embedding of weights corresponding to the model structure. In some embodiments, the trained model may include a fixed set of weights, such as downloaded from a server that provides weights. In various embodiments, the trained model includes a set or embedding of weights corresponding to the model structure. In embodiments where data is omitted, the segmenter 204 may generate a trained model based on a priori training performed, for example, by a developer of the segmenter 204, by a third party, etc. In some embodiments, the trained model may include a fixed set of weights, such as downloaded from a server that provides weights.

[0079] In some embodiments, the trained machine learning model receives an initial image having one or more selected objects. In some embodiments, the trained machine learning model outputs one or more object masks including the one or more objects.

[0080] After one or more object masks are output by the segmenter 204 (eg, from a machine learning model), the segmenter 204 removes the one or more selected objects from the initial image.

[0081] The repairer module 206 generates an inpainted image, and the inpainted image replaces the object pixels corresponding to one or more objects with the inpainted pixels. The inpainted pixels can be based on the pixels of the reference image without objects from the same position. Alternatively, the inpainter module 206 can identify the inpainted pixels to replace the removed objects based on the proximity of the inpainted pixels to other pixels around the object. The inpainter module 206 can use the gradient of the neighborhood pixels to determine the attributes of the inpainted pixels. For example, in the case where a bystander is standing on the ground, the inpainter module 206 replaces the inpainted pixels with the pixels of the ground. Other inpainting techniques are possible, including inpainter techniques based on machine learning that output inpainted pixels based on training data of images including similar structures.

[0082] In embodiments where the user chooses to erase the selected object, the user interface module 202 may display an inpainted image in which the selected object is removed and the selected object pixels are replaced with inpainted pixels.

[0083] In embodiments where the user chooses to move the selected object, replace the selected object, or add the object to the image, the diffusion module 208 performs fusion of the object with the object mask and the inpainted image using a diffusion model. The diffusion model may receive as input the object mask, the incomplete object, and the inpainted image. The diffusion model may receive additional inputs, such as a text request, the number of pixels to be filled to output a complete object based on the incomplete object, the size of the modified image, etc.

[0084] The diffusion model includes a forward process in which the diffusion model adds noise to the data and a reverse process in which the diffusion model learns to recover the data from the noise. For example, in the case of moving a selected object from a first location to a second location, the diffusion module 208 applies the diffusion model by fusing the selected object with a progressively noisier version of the repaired image and then a progressively denoised version. In some embodiments, an object splicing diffusion model is used to move the object from a first location to a second location. In some embodiments, a generative diffusion model is used when the object is an incomplete object and a portion of the object is generated, and / or for a new object generated from a text prompt.

[0085] Object splicing diffusion model

[0086] In some embodiments, an object splicing diffusion model is used when moving an object from a first position to a second position. In some embodiments, the diffusion module 208 includes an object image encoder that extracts semantic features from a selected object, a diffusion model that fuses the object with the image, and a content adapter that transforms a sequence of visual words into a sequence of text words to overcome the domain gap between the image and the text. In some embodiments, the diffusion module 208 uses self-supervision to train the diffusion model based on training data, wherein the training data includes image and text pairs. In some embodiments, the diffusion model is trained on synthetic data that simulates real-world scenarios. The diffusion model can also be trained using data augmentation generated by introducing random shifts and cropping augmentation during training while ensuring that the foreground object is contained within the cropping window.

[0087] During the first stage, a content adapter is trained using image and text pairs to maintain the high-level semantics of the object, and during the second stage, the content adapter is trained in the context of a diffusion model to encode key identity features of the object by facilitating the visual reconstruction of the object in the original image. The diffusion model can be trained on the embeddings produced by the content adapter through the cross-attention block.

[0088] The diffusion model uses the object mask to fuse the inpainted image with the object. The diffusion model can denoise the masked area. The content adapter can transform the visual features from the object image encoder into text features (words) used as conditioning for the diffusion model.

[0089] Generative Diffusion Model

[0090] In some embodiments, the generative diffusion model is used to output a complete object based on an incomplete object, or to generate a new object. The diffusion module 208 trains the generative diffusion model based on training data. The training data may include image and text pairs for creating an embedding space of images and text. The image and text pairs may include images associated with corresponding texts, such as an image of a dog and text including "pitbull". The diffusion module 208 may be trained using a loss that reflects the cosine distance between the embedding of the text hint and the estimated clean image (i.e., the object generated without text).

[0091] The diffusion module 208 can use the training data to perform text conditioning, where text conditioning describes the process of outputting objects conditioned on text prompts. The diffusion module 208 can train the neural network to output objects based on text prompts provided by the user or by the media application. For example, the text prompt can be a suggestion generated by the media application based on the context of the initial image (e.g., where the initial image is a beach, the text prompt can be for a beach ball, a turtle, etc.).

[0092] In some embodiments, the diffusion module 208 can output at least a portion of the missing portion of the object based on receiving an incomplete object as input data, the location within the image where the incomplete object is being moved, and the size of the output (e.g., the original size or the modified size if the object is resized). For example, if a user selects an object that is partially truncated by a demarcation in a user interface and moves the object from a first position to a second position, where the second position also truncates a portion of the object, the diffusion module 208 can output a modified object that includes more of the object that is visible based on moving the object in the image. In some embodiments, the diffusion module 208 can output a complete object based on the incomplete object selected by the user. For example, in the case where the user selects a beach ball that is partially occluded by another object, the diffusion model can be trained to output a complete beach ball.

[0093] In some embodiments, the diffusion module 208 generates a version of the complete object with progressively more noise than the previous version and a version of the inpainted image with progressively more noise than the previous version. For example, a forward Markov noise addition process generates a series of noisy inpainted images by gradually adding Gaussian noise until a nearly isotropic Gaussian noise sample is obtained. The forward noise addition process defines a series of image manifolds, where each manifold consists of a noisy image.

[0094] The diffusion module 208 may use the object mask to spatially fuse the noisy version of the complete object with the corresponding noisy version of the inpainted image. For example, the diffusion module 208 may use the object mask to fuse each noisy version of the complete object with each corresponding noisy version of the inpainted image, wherein the object mask delineates the boundaries of the complete object, so that the object mask delineates the area that is modified during the fusion process. In some embodiments, the diffusion process may include local complete object guided diffusion, wherein the image generation loss determined during the training process is used under the object mask during the positional object generation diffusion.

[0095] The diffusion module 208 can perform diffusion steps to denoise the latent space in a direction that depends on the textual prompt. The diffusion module 208 generates a progressively denoised version of the complete object compared to the previous version and a progressively denoised version of the inpainted image compared to the previous version. For example, an inverse Markov process transforms the Gaussian noise samples by repeatedly denoising the inpainted image using a learned posterior. Each step of the denoising diffusion process projects the noisy image onto the next less noisy manifold.

[0096] The diffusion module 208 performs a denoising diffusion step after each fusion to restore coherence by projecting onto the next manifold.Once the spatial fusion is complete, the diffusion module 208 preserves the background by replacing the regions outside the object mask with corresponding regions from the inpainted image.

[0097] In some embodiments, the diffusion module 208 uses cross-domain synthesis to apply an iterative refinement scheme to infuse contextual information to the object so that the object matches the style of the inpainted image. For example, if the object is generated for an indoor occasion and is added to an outdoor inpainted image, the object can be modified to be brighter to match the inpainted image. In another example, if the object is located at a first location that is in the shadow, and a second location is in full sunlight, the object can be modified to match the brightness of the second location.

[0098] Object Removal Model

[0099] In some embodiments, instead of using the segmenter 204 to remove objects and using the inpainting module 206 to add pixels to the removed areas in the initial image, the diffusion model 208 is trained to include an object removal model.

[0100] The diffusion module 208 generates counterfactual training data to train the diffusion model to include the object removal model. For each counterfactual image pair, the diffusion module 208 captures a factual image containing the object in the scene; physically removes the object while avoiding camera movement, lighting changes, or motion of other objects; captures a counterfactual image of the scene without the object; and segments the factual image to create an object mask. Segmenting the factual image includes creating a segmentation map (M) for the object O removed from the factual image X. o ).

[0101] The diffusion module 208 creates a combined image including a factual image and an object mask and a counterfactual image for each image pair. The object mask may be a binary object mask (M o (X)), and the counterfactual image pair can be described as an input pair of the factual image and the binary object mask (X, M o (X)), and output the counterfactual image (X cf ).

[0102] The diffusion module 208 estimates the distribution P(X) of counterfactual images given a factual image x and a binary object mask by training a diffusion model based on using counterfactual image pairs. cf |X=x, M o (X)). The diffusion module 208 minimizes the loss function ℒ( ) to determine this estimate:

[0103] Equation 1

[0104] in is a denoiser network with the following input: the noisy latent representation of the counterfactual image , the latent representation x of the image containing the object to be removed cond , a mask m indicating the location of the object, a timestamp t, and an encoding p of the empty string (text hint). t Use the following forward process equation to calculate:

[0105]

[0106] where x represents an image without an object (counterfactual), α t and σ t Determined by the noise schedule, and ϵ~ (O, I).

[0107] Once the diffusion model including the object removal model is trained, the user interface module 202 can receive a request to remove the selected object from the first modified image. The initial image and the request are provided as input to the object removal model, which outputs a modified image that does not include the selected object.

[0108] Object Insertion Model

[0109] In some embodiments, instead of using segmenter 204 to remove objects, using inpaintor module 206 to add pixels to the removed areas in the initial image, and using diffusion model 208 to blend the objects with the pixels at the new locations, diffusion model 208 is trained to include an object insertion model.

[0110] In some embodiments, the object insertion model is trained on a number of image pairs that exceeds the number of counterfactual image pairs available. Thus, diffusion module 208 generates synthetic training data. For each synthetic image pair, diffusion module 208 selects an original image that includes an object, uses the object removal model to output a modified image without the object from the original image, generates an input image by inserting the object into the modified image, and segments the original image to create an object mask. The modified image lacking the object is referred to as z i , which uses the following equation:

[0111]

[0112] The original image is x 1 , x 2 ,…, x n , and the corresponding object mask is M o (x 1 ), M o (x 2 ),…, M o (x n The diffusion module 208 inserts the object into the object-free scene z i The input image is generated from to produce an image without shadows and reflections using the following equation:

[0113]

[0114] The synthetic image pair is (y i ,M o (x i )), and the corresponding target is the original image x i Although both the input image and the output image contain object o, the input image does not contain the effect of the object on the scene, while the output image does. In some embodiments, the diffusion module 208 utilizes the diffusion objective presented in Equation 1 to train the object insertion model.

[0115] For each synthetic image pair, the diffusion module 208 creates a second combined image including the original image and the object mask and the input image. The diffusion module 208 pre-trains the diffusion model to include an object insertion model based on using the synthetic image pairs, and fine-tunes the diffusion model to include an object insertion model based on using the counterfactual image pairs used to train the object removal model.

[0116] In some embodiments, the user interface module 202 generates graphical data for displaying a user interface that provides the user with options to specify the location of the object and options to resize the object. The diffusion module 208 adds the selected object removed from the initial image to the new location. In some embodiments, the diffusion module 208 provides the selected object as input to a diffusion model, and provides the location where the selected object will be located in the modified image, and outputs a modified image that merges the selected object with the inpainted image. For example, the diffusion module 208 can spatially merge a noisy version of the inpainted image with a noisy version of the selected object.

[0117] In some embodiments, the diffusion module 208 can add a shadow to the selected object in the new position. The shadow can match the direction of the light in the image. For example, if the sun casts light from the upper left corner of the image, the shadow can appear to the right of the person and / or object. In some embodiments, the diffusion module 208 uses a machine learning model to output a shadow mask for generating a shadow attached to the object.

[0118] Once the selected object is added to the inpainted image, the user interface module 202 may include additional features for changing the inpainted image, such as an option to change the lighting of the inpainted image.

[0119] Figure 3A An example initial image 300 is illustrated. The initial image includes a person 301 as a subject of the initial image 300, a bystander 302 in the foreground of the initial image 300, grass 303, a road 304, trees 305, and a cloudy sky 306.

[0120] Figure 3B Examples include those from Figure 3A310. The two objects are shown with outlines 311, 312 of how the user selected the two objects. The user may have selected the two objects by circling, brushing, double-clicking, etc., the two objects using a finger, a mouse, or other object. The user interface module 202 may use an object selection tool, a lasso tool, an artificial intelligence segmentation tool, etc. to identify the objects in the initial image 310. In some embodiments, the user interface module 202 may have suggested selecting the two objects, and the two objects are highlighted in response to the user confirming the selection.

[0121] Once the two objects are selected, the segmenter 204 segments the objects from the initial image 310 and generates an object mask. Figure 3C An example initial image 320 is illustrated in which object masks 321, 322 surround two objects. The person is surrounded by the first object mask 321 and the bystander is surrounded by the second object mask 322. The segmenter 204 removes the person and the bystander from the initial image 320.

[0122] Once from Figure 3C By removing the person and the bystander from the initial image 320, the restoration module 206 generates a restored image, which replaces the object pixels corresponding to the removed objects with the restored pixels. Figure 3D An example inpainted image 330 is illustrated in which bystanders in the initial image 320 are removed and person 331 is moved and resized.

[0123] In some embodiments, diffusion module 208 resizes the object to be larger or smaller than the object in the initial image. For example, the object can be resized to be larger when moved to the front or resized to be smaller when moved to the back. In this example, the user provides input to resize person 331 to be smaller, and diffusion module 208 resizes person 331 and blends the person with the new position. In some embodiments, moving person 331 can be a separate action from resizing, or both moving and resizing can be part of the same action.

[0124] Additional changes may be made to the restored image. Figure 3E Examples include those from Figure 3D The restored image 340 also includes a first type of tree replaced with a second type of tree 342. The restored image 340 also includes a first type of tree replaced with a second type of tree 342. Figure 3DThe cloudy sky in the image is replaced with a clear sky with sunlight 343 originating from the upper right corner of the sky. The direction of the sun 343 causes the diffusion module 208 to output a shadow 344 that matches the person 345. In some embodiments, the road is replaced with grass 341 by utilizing the diffusion module 208 to output the grass and blending the grass with the inpainted image.

[0125] In some embodiments, the diffusion module 208 receives an incomplete object as input and outputs a complete object. The diffusion module 208 may also add the complete object to a second location within the inpainted image by blending complete object pixels corresponding to the complete object with the inpainted pixels.

[0126] In some embodiments, if an object is captured at the edge of the image and a portion of the complete object is missing, the diffusion module 208 can complete the missing portion of the object before performing fusion. For example, if a woman is wearing a long skirt and a portion of the skirt is missing, the diffusion module 208 can output the complete skirt based on the incomplete skirt.

[0127] Figure 4A An example initial image 400 is shown in which a child 405 is sitting on a bench 410 and holding a balloon 415, which is partially cut off by the boundary of the initial image 400. In this example, the user interface module 202 provides the user interface with an option for the user to select an object. The user selects the child 405, the bench 410, and the balloon 415 at a first position, wherein the balloon 415 represents an incomplete image. The segmenter 204 segments the child 405, the bench 410, and the balloon 415 to separate the objects from the initial image 400.

[0128] The user interface module 202 includes an option for moving the selected object to a different location. The user selects a second location. The segmenter 204 removes the selected object from the initial image. The inpainting module 206 generates an inpainted image that replaces the object pixels corresponding to the removed object with the inpainted pixels.

[0129] The diffusion module 208 receives as input the coordinates of the selected object and the second location, and outputs a balloon of the complete object and a longer bench. Figure 4B An example modified image 450 is illustrated in which a child 455, a bench 460, and a balloon 465 are moved to a second position. In this example, the diffusion module 208 outputs a modified image that uses an object mask to blend one or more versions of the child 455, the bench 460, and the balloon 465 with one or more versions of the inpainted image.

[0130] In some embodiments, the user interface module 202 receives a command from the user interface to inverse crop the image. The command to inverse crop the image can occur on a modified image, an initial image, etc. The command to inverse crop the image can be a button as part of the user interface that is a suggestion to help center an object such as a person in the middle of the image. In some embodiments, the command can be based on the user specifying a new boundary of the image by directly extending the boundary of the image, or the movement of a selected object that extends the boundary of the image.

[0131] The repairer module 206 receives as input the inverse cropped image and the size of the inverse cropped image, and outputs the inverse cropped image, which replaces the boundary between the image and the inverse cropped image with inpainted pixels that match the image. For example, where the boundary is water, the repairer module 206 may use the pixels of the water for the inverse cropped portion of the image.

[0132] Figure 5A An example user interface 500 illustrating an initial image 504 includes a button 412 for changing the border of the initial image 504. In this example, the user interface 500 includes a first button 510 for editing the image and a second button 512 for changing the border of the image. Other mechanisms for providing commands to inversely crop the image are possible. For example, the user can select an edge 506 of the initial image 504 and drag and drop the edge 506 to indicate where the user wants the new border to end.

[0133] The user may change the borders of the initial image because the person 508 in the image is not centered in the image and cropping the image to shrink the image on the right side would result in an image that is too narrow.

[0134] Figure 5B An example user interface 515 is illustrated with an indicator 521 for dilating the border of an initial image 517. In this example, the user clicks and drags the indicator 521 to dilate the left border of the initial image 517. The dilated area 519 is illustrated with pixelated content when the invertor module 206 generates an inverse cropped image.

[0135] In some embodiments, the repairer module 206 receives as input the inverse cropped image and the size of the inverse cropped image. For example, the size includes the length and width of the new border on the left side of the original image.

[0136] The repairer module 206 outputs an inversely cropped image that includes the repaired pixels between the uncropped border of the modified image and the expanded border based on the size. In this case, the repairer module 206 replicates the pixels of bushes, rocks, water, and flowers. Figure 5CAn example user interface 530 illustrating a reverse-cropped image 532 that is output based on an initial image.

[0137] Figure 5D An alternative example user interface 540 is illustrated in which the uncropped boundaries of the initial image 542 are extended using the selected person 544. In this alternative example, instead of using an indicator such as Figure 5B indicator 521 in the Figure 5A The edge 506 in the image 506 is used to extend the uncropped boundary of the initial image 542, and the user can select an object in the image and move the object to extend the boundary. Figure 5D , the user has moved person 544 to the edge of the initial image 542. The new position of person 544 defines a new edge to be generated for the inverse cropped image, with 555 corresponding to the expanded area.

[0138] Example Method

[0139] Figure 6 An example flow chart illustrating a method 600 for generating a modified image of a complete object from an incomplete object according to some embodiments described herein. The method 600 may be performed by Figure 2 In some embodiments, the method 600 is performed by the user device 115, the media server 101, or partially on the user device 115 and partially on the media server 101.

[0140] Figure 6 The method 600 of can start at block 602. At block 602, it is determined whether the user grants permission to access the initial image. If permission is not granted, the method 600 ends. If permission is granted, block 602 can be followed by block 604.

[0141] At block 604, a selection of an incomplete object in the initial image is received, wherein the incomplete object is associated with a first location within the initial image, and a missing portion of the incomplete object is truncated by a boundary of the initial image or is obscured by another object. For example, the incomplete object may be a car that is truncated by an edge of the initial image. The user may select the incomplete object by clicking on the object, circling the object, accepting a suggestion to modify the object generated through a user interface, etc. Block 604 may be followed by block 606.

[0142] At block 606, an object mask is generated that includes incomplete object pixels associated with the incomplete object. The incomplete object pixels may be determined by segmenting the initial image to identify pixels associated with the incomplete object in the initial object. Block 606 may be followed by block 608.

[0143] At block 608 , incomplete object pixels associated with the incomplete object are removed from the initial image. Block 608 may be followed by block 610 .

[0144] At block 610 , an inpainted image is generated that replaces the incomplete object pixels with inpainted pixels. Block 610 may be followed by block 612 .

[0145] At block 612 , the object mask, the incomplete object, and the inpainted image are provided as inputs to a diffusion model. Block 612 may be followed by block 614 .

[0146] At block 614, the diffusion model outputs the complete object. For example, the diffusion model may receive the incomplete object, a second location in the modified image where the complete object is to be placed, and a size of the complete object, including a resized size if the user resized the incomplete object or if a change from the first location to the second location resulted in a resize of the incomplete object. Block 614 may be followed by block 616.

[0147] At block 616, a modified image is generated by fusing one or more versions of the complete object with one or more versions of the inpainted image using the object mask, wherein the complete object is positioned at a second location in the modified image that is different from the first location in the initial image. Continuing with the example above, the complete image may include a complete version of the car. The car may be resized based on being moved from the foreground to the background, and reduced in size to account for the distance reflected by being positioned in the background.

[0148] In some embodiments, the modified image is a first modified image, and the method further includes receiving a request to remove the selected object from the first modified image; providing the first modified image and the request as input to an object removal model associated with the diffusion model; and outputting, using the object removal model, a second modified image that does not include the selected object. In some embodiments, the modified image is a first modified image, and the method further includes receiving a request to move the selected object from a third position to a fourth position; providing the first modified image and the request as input to an object insertion model associated with the diffusion model; and outputting, using the object insertion model, a second modified image that includes the selected object at the fourth position based on the request.

[0149] In some embodiments, the method further includes receiving a request to add an additional object to the initial image; outputting the additional object using the diffusion model; and outputting a modified image using the diffusion model by fusing one or more versions of the additional object with one or more versions of the inpainted image, wherein the additional object is positioned at a third location in the inpainted image that is different from a second location in the modified image. In some embodiments, the request to add the additional object includes a textual prompt describing the additional object, and the diffusion model outputs the additional object using generative artificial intelligence.

[0150] In some embodiments, the selected person in the inpainted image can be resized to account for being moved forward or backward from the first position. For example, the selected person can be made smaller or larger than the person in the initial image. In some embodiments, the diffusion model resizes the complete object based on the change from the first position in the initial image to the second position in the modified image.

[0151] Shadows corresponding to the selected person in different positions may also be generated, wherein the shadows match the direction of the light in the image. In some embodiments, a first image in the initial image is replaced with a second object from the initial image, wherein the modified image includes the first object replaced with the second object. In some embodiments, the lighting of the repaired image is also changed. For example, the sky may be made brighter, darker, have thicker clouds to reduce illumination, etc. In some embodiments, the method further includes modifying the lighting of the modified image, and adding shadows to the complete object based on the direction of the lighting of the modified image.

[0152] Figure 7 An example flow chart illustrating a method 700 for outputting an inversely cropped image from an initial image according to some embodiments described herein. The method 700 may be performed by Figure 2 In some embodiments, the method 700 is performed by the user device 115, the media server 101, or partially on the user device 115 and partially on the media server 101.

[0153] Figure 7 Method 700 may begin at block 702. At block 702, an initial image is displayed in a user interface. Block 702 may be followed by block 704.

[0154] At block 704, a command to inverse crop the initial image to extend the uncropped boundary of the initial image to the extended boundary is received, wherein the command is based on at least one action selected from the group consisting of: selecting an inverse crop button, moving an indicator to define an extended boundary, moving an edge of the initial image to define an extended boundary, moving a selected object to define an extended boundary, and combinations thereof. Block 704 may be followed by block 706.

[0155] At block 706 , an inversely cropped image is output, the inversely cropped image comprising inpainted pixels between the uncropped boundary of the original image and the command-based expanded boundary.

[0156] Figure 8 An example flow chart illustrating a method 800 for training an object removal model according to some embodiments described herein. The method 800 may be performed by Figure 2 In some embodiments, the method 800 is performed by the user device 115, the media server 101, or partially on the user device 115 and partially on the media server 101.

[0157] Method 800 begins at block 802. At block 802, counterfactual training data is generated for each counterfactual image pair by: capturing a factual image containing an object in a scene; physically removing the object while avoiding camera movement, lighting changes, or motion of other objects; capturing a counterfactual image of the scene without the object; and segmenting the factual image to create an object mask. Block 802 may be followed by block 804.

[0158] At block 804 , a combined image including the factual image and the object mask and the counterfactual image is created for each counterfactual image pair. Block 804 may be followed by block 806 .

[0159] At block 806 , a diffusion model is trained based on using the counterfactual image pairs to include an object removal model.

[0160] Fig. 9 An example flow chart illustrating a method 900 for training an object insertion model according to some embodiments described herein. The method 900 may be performed by Figure 2 In some embodiments, the method 900 is performed by the user device 115, the media server 101, or partially on the user device 115 and partially on the media server 101.

[0161] Method 900 begins at block 902. At block 902, counterfactual training data is generated for each counterfactual image pair by: capturing a factual image containing an object in a scene; physically removing the object while avoiding camera movement, lighting changes, or motion of other objects; capturing a counterfactual image of the scene without the object; and segmenting the factual image to create an object mask. Block 902 may be followed by block 904.

[0162] At block 904 , a combined image including the factual image and the object mask and the counterfactual image is created for each counterfactual image pair. Block 904 may be followed by block 906 .

[0163] At block 906 , a diffusion model is trained based on using the counterfactual image pairs to include an object removal model. Block 906 may be followed by block 908 .

[0164] At block 908, synthetic training data is generated for each synthetic image pair by: selecting an original image that includes an object; using an object removal model to output a modified image without the object from the original image, generating an input image by inserting the object into the modified image; and segmenting the original image to create an object mask. Block 908 may be followed by block 910.

[0165] At block 910 , a second combined image is created for each composite pair that includes the original image and the object mask and the input image. Block 910 may be followed by block 912 .

[0166] At block 912, a diffusion model is pre-trained based on using the synthetic image pairs to include an object insertion model. Block 912 may be followed by block 914.

[0167] At block 914, the diffusion model is fine-tuned based on using the counterfactual image pairs to include an object insertion model.

[0168] In addition to the above description, controls may be provided to allow the user to make a choice about whether and when the systems, programs, or features described herein may be able to collect user information (e.g., information about the user's social network, social behavior, or activities, occupation, user preferences, or the user's current location) and whether to send content or communications from a server to the user. In addition, before certain data is stored or used, it may be processed in one or more ways to remove personally identifiable information. For example, the user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location for which location information is obtained may be generalized (such as to a city, zip code, or state level) so that the user's specific location cannot be determined. Thus, the user can control what information is collected about the user, how the information is used, and what information is provided to the user.

[0169] In the above description, many specific details are set forth for the purpose of explanation in order to provide a thorough understanding of this specification. However, it will be apparent to those skilled in the art that the present disclosure can be practiced without these specific details. In some instances, structures and devices are shown in block diagram form to avoid obscuring this description. For example, the above may mainly refer to user interfaces and specific hardware to describe embodiments. However, embodiments may be applied to any type of computing device that can receive data and commands, as well as any peripheral device that provides services.

[0170] References in the specification to "some embodiments" or "some examples" mean that a particular feature, structure, or characteristic described in conjunction with the embodiment or example may be included in at least one embodiment of the specification. The appearance of the phrase "in some embodiments" in various places in the specification does not necessarily all refer to the same embodiment.

[0171] Some portions of the above detailed description are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the art of data processing to most effectively convey the substance of their work to other persons skilled in the art. An algorithm is here and generally considered to be a self-consistent sequence of steps that produces a desired result. The steps are those that require physical manipulation of physical quantities. Typically, although not necessarily, these quantities take the form of electrical or magnetic data that can be stored, transferred, combined, compared, and otherwise manipulated. Mainly for general reasons, it has sometimes proven convenient to refer to these data as bits, values, elements, symbols, characters, terms, numbers, etc.

[0172] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically noted otherwise, as will be apparent from the following discussion, it should be understood that throughout this specification, discussions utilizing terms including "processing" or "computing" or "calculating" or "determining" or "displaying" and the like refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities within the computer system's registers and memories and transforms that data into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission or display devices.

[0173] Embodiments of the specification may also relate to a processor for performing one or more steps of the above method. The processor may be a dedicated processor selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory computer-readable storage medium, including but not limited to any type of disk, including an optical disk, ROM, CD-ROM, magnetic disk, RAM, EPROM, EEPROM, magnetic card or optical card, flash memory (including USB disk with non-volatile memory), or any type of medium suitable for storing electronic instructions, each of which is coupled to a computer system bus.

[0174] The specification can take the form of some complete hardware embodiments, some complete software embodiments, or some embodiments containing both hardware elements and software elements. In some embodiments, the specification is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0175] Furthermore, the present description may take the form of a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of the present description, a computer-usable or computer-readable medium may be a device that can contain, store, communicate, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or device.

[0176] A data processing system suitable for storing or executing program code will include at least one processor coupled directly or indirectly to a memory element via a system bus. The memory element may include local memory, bulk storage, and cache memory used during the actual execution of the program code, which provides temporary storage of at least some program code to reduce the number of times code must be retrieved from bulk storage during execution.

Claims

1. A computer-implemented method comprising: receiving a selection of an incomplete object in an initial image, wherein the incomplete object is associated with a first location within the initial image, and a missing portion of the incomplete object is truncated by a boundary of the initial image or occluded by another object; generating an object mask including incomplete object pixels associated with the incomplete object; removing the incomplete object pixels associated with the incomplete object from the initial image; generating an inpainted image, wherein the inpainted image replaces pixels of the incomplete object corresponding to the incomplete object with inpainted pixels; providing the object mask, the incomplete object, and the inpainted image as input to a diffusion model; outputting a complete object using the diffusion model; as well as A modified image is generated by fusing one or more versions of the complete object with one or more versions of the inpainted image using the object mask, wherein the complete object is positioned at a second location in the modified image that is different from the first location in the initial image.

2. The method of claim 1, wherein the modified image is a first modified image, and the method further comprises: receiving a request to remove a selected object from the first modified image; providing the first modified image and the request as input to an object removal model associated with the diffusion model; as well as A second modified image excluding the selected object is outputted using the object removal model.

3. The method of claim 1, wherein the modified image is a first modified image, and the method further comprises: receiving a request to move the selected object from the third location to a fourth location; providing the first modified image and the request as input to an object insertion model associated with the diffusion model; as well as A second modified image including the selected object at the fourth position based on the request is outputted using the object insertion model.

4. The method of claim 1, wherein the modified image is a first modified image, and the method further comprises: receiving a request to add an additional object to the initial image; outputting the additional object using the diffusion model; as well as A second modified image is outputted by fusing one or more versions of the additional object with one or more versions of the inpainted image using the diffusion model.

5. The method of claim 4, wherein the request to add the additional object includes a textual hint describing the additional object.

6. The method of claim 1, wherein the complete object is resized based on a change from the first position in the initial image to the second position in the modified image.

7. The method of claim 1, further comprising: receiving a command to inversely crop the modified image to extend an uncropped boundary of the modified image to an extended boundary; as well as An inversely cropped image is output, the inversely cropped image comprising inpainted pixels between the uncropped boundary of the modified image and the expanded boundary based on the command.

8. The method of claim 7, wherein the command to inversely crop the repaired image comprises: Selection of the reverse crop button; as well as A command to directly extend the uncropped bounds of the modified image to the extended bounds, or a movement of the complete object to extend the uncropped bounds of the modified image to the extended bounds.

9. The method of claim 1, further comprising: modifying lighting of the modified image; as well as A shadow is added to the complete object based on the direction of the lighting of the modified image.

10. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform operations comprising: receiving a selection of an incomplete object in an initial image, wherein the incomplete object is associated with a first location within the initial image, and a missing portion of the incomplete object is truncated by a boundary of the initial image or occluded by another object; generating an object mask including incomplete object pixels associated with the incomplete object; removing the incomplete object pixels associated with the incomplete object from the initial image; generating an inpainted image, wherein the inpainted image replaces pixels of the incomplete object corresponding to the incomplete object with inpainted pixels; providing the object mask, the incomplete object, and the inpainted image as input to a diffusion model; outputting a complete object using the diffusion model; as well as A modified image is generated by fusing one or more versions of the complete object with one or more versions of the inpainted image using the object mask, wherein the complete object is positioned at a second location in the modified image that is different from the first location in the initial image.

11. The non-transitory computer readable medium of claim 10, wherein the modified image is a first modified image, and the operations further comprise: receiving a request to remove a selected object from the first modified image; providing the first modified image and the request as input to an object removal model associated with the diffusion model; as well as A second modified image excluding the selected object is outputted using the object removal model.

12. The non-transitory computer readable medium of claim 10, wherein the modified image is a first modified image, and the operations further comprise: receiving a request to move the selected object from the third location to a fourth location; providing the first modified image and the request as input to an object insertion model associated with the diffusion model; as well as A second modified image including the selected object at the fourth position based on the request is outputted using the object insertion model.

13. The non-transitory computer readable medium of claim 10, wherein the modified image is a first modified image, and the operations further comprise: receiving a request to add an additional object to the initial image, the request including a textual hint describing the additional object; outputting the additional object using the diffusion model; as well as A second modified image is outputted by fusing one or more versions of the additional object with one or more versions of the inpainted image using the diffusion model.

14. The non-transitory computer readable medium of claim 10, wherein the operations further comprise: receiving a command to inversely crop the modified image to extend an uncropped boundary of the modified image to an extended boundary; as well as An inversely cropped image is output, the inversely cropped image comprising inpainted pixels between the uncropped boundary of the modified image and the expanded boundary based on the command.

15. A system comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored thereon, the instructions, when executed by the processor, causing the processor to perform operations, the operations comprising: receiving a selection of an incomplete object in an initial image, wherein the incomplete object is associated with a first location within the initial image, and a missing portion of the incomplete object is truncated by a boundary of the initial image or occluded by another object; generating an object mask including incomplete object pixels associated with the incomplete object; removing the incomplete object pixels associated with the incomplete object from the initial image; generating an inpainted image, wherein the inpainted image replaces pixels of the incomplete object corresponding to the incomplete object with inpainted pixels; providing the object mask, the incomplete object, and the inpainted image as input to a diffusion model; outputting a complete object using the diffusion model; and A modified image is generated by fusing one or more versions of the complete object with one or more versions of the inpainted image using the object mask, wherein the complete object is positioned at a second location in the modified image that is different from the first location in the initial image.

16. The system of claim 15, wherein the modified image is a first modified image, and the operations further comprise: receiving a request to remove a selected object from the first modified image; providing the first modified image and the request as input to an object removal model associated with the diffusion model; as well as A second modified image excluding the selected object is outputted using the object removal model.

17. The system of claim 15, wherein the modified image is a first modified image, and the operations further comprise: receiving a request to move the selected object from the third location to a fourth location; providing the first modified image and the request as input to an object insertion model associated with the diffusion model; as well as A second modified image including the selected object at the fourth position based on the request is outputted using the object insertion model.

18. The system of claim 15, wherein the modified image is a first modified image, and the operations further comprise: receiving a request to add an additional object to the initial image, the request including a textual hint describing the additional object; outputting the additional object using the diffusion model; as well as A second modified image is outputted by fusing one or more versions of the additional object with one or more versions of the inpainted image using the diffusion model.

19. The system of claim 15, wherein the complete object is resized based on a change from the first position in the initial image to the second position in the modified image.

20. The system of claim 15, wherein the operations further comprise: receiving a command to inversely crop the modified image to extend an uncropped boundary of the modified image to an extended boundary; as well as An inversely cropped image is output, the inversely cropped image comprising inpainted pixels between the uncropped boundary of the modified image and the expanded boundary based on the command.

Images