REPOSITIONING, REPLACING AND CREATING OBJECTS IN AN IMAGE

Abstract

A media application receives a selection of an incomplete object in a source image. The media application creates an object mask containing pixels of the incomplete object that correspond to that object. The media application removes the pixels of the incomplete object that correspond to the incomplete object from the source image. The media application creates a reconstructed image that replaces the pixels of the incomplete object that correspond to the incomplete object with reconstructed pixels. The media application outputs a complete object. The media application outputs a modified image by blending one or more versions of the complete object with one or more versions of the reconstructed image using the object mask.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 465,230, filed May 9, 2023, entitled "Repositioning Objects in an Image," and U.S. Provisional Patent Application No. 63 / 562,634, filed March 7, 2024, entitled "Performing Scene Impact Editing Tasks Using Diffusion Neural Networks," each of which is incorporated herein in its entirety. GENERAL STATE OF THE ART

[0002] A user may capture an image containing objects in undesirable positions. For example, an object may be cut off by an edge of the image, cut off by another object, etc. Techniques exist for moving objects within images; however, attempts to move the objects to different positions in the image can have disastrous results. For example, pixels associated with an object may be misidentified, leaving one section of the object in its original position while a remaining section of the object is moved to another (e.g., a chicken's body is moved while the chicken's feet remain). In another example, the empty spaces caused by removing pixels associated with a moving object may be filled with pixels that look out of place.In yet another example, the pixels surrounding a moving object may look different than the background, resulting in an image that looks poorly edited.

[0003] The description of the general prior art provided herein is intended to provide a general context for the disclosure. Works of the presently named inventors within the scope described in this general prior art section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are not expressly or implicitly considered prior art with respect to the present disclosure. SUMMARY

[0004] A computer-implemented method includes receiving a selection of an incomplete object in an output image, the incomplete object associated with a first location within the output image, and an omitted portion of the incomplete object being clipped by a boundary of the output image or obscured by another object. The method further includes generating an object mask including pixels of an incomplete object associated with the incomplete object. The method further includes removing the pixels of the incomplete object associated with the incomplete object from the output image. The method further includes generating an inpainted image that replaces the pixels of the incomplete object corresponding to the incomplete object with reconstructed pixels.The method further includes providing the object mask, the incomplete object, and the reconstructed image as input to a diffusion model. The method further includes outputting, with the diffusion model, a complete object. The method further includes generating a modified image by blending one or more versions of the complete object with one or more versions of the reconstructed image using the object mask, wherein the complete object is positioned at a second position in the modified image that is different from the first position in the output image.

[0005] In some embodiments, the modified image is a first modified image, and the method further includes receiving a request to remove a selected object from the first modified image; providing the first modified image and the request as input to an object removal model associated with the diffusion model; and outputting, with the object removal model, a second modified image that does not include the selected object.In some embodiments, the modified image is a first modified image, and the method further includes receiving a request to move a selected object from a third position to a fourth position; providing the first modified image and the request as input to an object insertion model associated with the diffusion model; and outputting, with the object insertion model, a second modified image including the selected object at the fourth position based on the request.

[0006] In some embodiments, the modified image is a first modified image, and the method further includes receiving a request to add an additional object to the source image; outputting, with the diffusion model, the additional object; and outputting, with the diffusion model, a second modified image by blending one or more versions of the additional object with one or more versions of the reconstructed image. In some embodiments, the request to add the additional object includes a textual prompt describing the additional object.

[0007] In some embodiments, the size of the entire object is changed based on a change from the first position in the source image to the second position in the modified image. In some embodiments, the method further includes receiving a modified image cropping command to expand edges of an expanded portion of the modified image into expanded edges, and outputting a cropped image including reconstructed pixels between the edges of the expanded portion of the modified image and the expanded edges based on the command.In some embodiments, the command to expand the reconstructed image includes selecting a button to expand the image area and either a command to directly expand the edges of the expanded image area of ​​the modified image to the expanded edges or a movement of the full object that expands the edges of the expanded image area of ​​the modified image to the expanded edges. In some embodiments, the method further includes modifying an illumination of the modified image; and adding a shadow to the full object based on an illumination direction of the modified image.

[0008] In some embodiments, a non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations. The operations include receiving a selection of an incomplete object in an output image, the incomplete object associated with a first location within the output image and an omitted portion of the incomplete object truncated by a boundary of the output image or obscured by another object; generating an object mask including pixels of the incomplete object associated with the incomplete object; removing the pixels of the incomplete object associated with the incomplete object from the output image;Generating a reconstructed image that replaces the pixels of the incomplete object corresponding to the incomplete object with reconstructed pixels; providing, as input to a diffusion model, the object mask, the incomplete object, and the reconstructed image; outputting, with the diffusion model, a complete object; and generating a modified image by blending one or more versions of the complete object with one or more versions of the reconstructed image using the object mask, wherein the complete object is positioned at a second position in the modified image that is different from the first position in the output image.

[0009] In some embodiments, the modified image is a first modified image, and the operations further include receiving a request to remove a selected object from the first modified image; providing the first modified image and the request as input to an object removal model associated with the diffusion model; and outputting, with the object removal model, a second modified image that does not include the selected object.In some embodiments, the modified image is a first modified image, and the operations further include receiving a request to move a selected object from a third position to a fourth position; providing the first modified image and the request as input to an object insertion model associated with the diffusion model; and outputting, with the object insertion model, a second modified image including the selected object at the fourth position based on the request.

[0010] In some embodiments, the modified image is a first modified image, and the operations further comprise: receiving a request to add an additional object to the source image, the request including a textual prompt describing the additional object; outputting, with the diffusion model, the additional object; and outputting, with the diffusion model, a second modified image by blending one or more versions of the additional object with one or more versions of the reconstructed image. In some embodiments, the size of the full object is changed based on a change from the first position in the source image to the second position in the modified image.In some embodiments, the operations further include receiving a modified image cropping command to expand edges of the expanded modified image cropping into expanded edges, and outputting an expanded image including reconstructed pixels between the edges of the expanded modified image cropping and the expanded edges based on the command.

[0011] In some embodiments, a system includes a processor and a memory coupled to the processor, having instructions stored thereon that, when executed by the processor, cause the processor to perform operations. The operations include receiving a selection of an incomplete object in an output image, the incomplete object associated with a first location within the output image and an omitted portion of the incomplete object truncated by a boundary of the output image or obscured by another object; generating an object mask including pixels of the incomplete object associated with the incomplete object; removing the pixels of the incomplete object associated with the incomplete object from the output image;Generating a reconstructed image that replaces the pixels of the incomplete object corresponding to the incomplete object with reconstructed pixels; providing, as input to a diffusion model, the object mask, the incomplete object, and the reconstructed image; outputting, with the diffusion model, a complete object; and generating a modified image by blending one or more versions of the complete object with one or more versions of the reconstructed image using the object mask, wherein the complete object is positioned at a second position in the modified image that is different from the first position in the output image.

[0012] In some embodiments, the modified image is a first modified image, and the operations further include receiving a request to remove a selected object from the first modified image; providing the first modified image and the request as input to an object removal model associated with the diffusion model; and outputting, with the object removal model, a second modified image that does not include the selected object.In some embodiments, the modified image is a first modified image, and the operations further include receiving a request to move a selected object from a third position to a fourth position; providing the first modified image and the request as input to an object insertion model associated with the diffusion model; and outputting, with the object insertion model, a second modified image including the selected object at the fourth position based on the request.

[0013] In some embodiments, the modified image is a first modified image, and the operations further comprise: receiving a request to add an additional object to the source image, the request including a textual prompt describing the additional object; outputting, with the diffusion model, the additional object; and outputting, with the diffusion model, a second modified image by blending one or more versions of the additional object with one or more versions of the reconstructed image. In some embodiments, the size of the full object is changed based on a change from the first position in the source image to the second position in the modified image.In some embodiments, the operations further include receiving a modified image cropping command to expand edges of the expanded modified image cropping into expanded edges, and outputting an expanded image including reconstructed pixels between the edges of the expanded modified image cropping and the expanded edges based on the command. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a block diagram of an example network environment according to some embodiments described herein. Fig. 2 is a block diagram of an example computing device according to some embodiments described herein. Fig. 3A illustrates an example output image according to some embodiments described herein. Fig. Figure 3B illustrates an example initial image where the two objects from Fig. 3A were selected for modification according to some embodiments described herein. Fig. 3C illustrates an example output image with masks surrounding the objects, according to some embodiments described herein. Fig. 3D shows an example reconstructed image in which a bystander object has been removed and the subject object has been moved and resized in accordance with some embodiments described herein. Fig. 3E illustrates an example reconstructed image in which the road has been replaced with grass, a first tree species has been replaced with a second tree species, and a cloudy sky has been replaced with a clear sky, in accordance with some embodiments described herein. Fig. 4A illustrates an example source image of a child sitting on a bench and holding balloons that are partially cut off by a boundary of the source image, according to some embodiments described herein. Fig. 4B illustrates an example modified image in which the child, bench, and balloons have been moved to a second position, in accordance with some embodiments described herein. Fig. 5A illustrates an example user interface of a source image including a button for changing a border of a source image, in accordance with some embodiments described herein. Fig. 5B illustrates an example user interface with an indicator used to extend an edge of the source image according to some embodiments described herein. Fig. 5C illustrates an example user interface of an enhanced cropped image output based on the source image, according to some embodiments described herein. Fig. 5D illustrates an alternative example user interface in which a selected object is used to extend an edge of the source image, in accordance with some embodiments described herein. Fig. 6 illustrates an exemplary flowchart of a method for generating a modified image of a complete object from an incomplete object, in accordance with some embodiments described herein. Fig. 7 illustrates an exemplary flowchart of a method for outputting a cropped image from an output image in accordance with some embodiments described herein. Fig. 8 illustrates an example flowchart of a method for training an object distance model in accordance with some embodiments described herein. Fig. 9 illustrates an example flowchart of a method for training an object insertion model in accordance with some embodiments described herein. DETAILED DESCRIPTION

[0014] A user may capture an image containing objects in undesirable positions. For example, an object may be cut off by an edge of the image, cut off by another object, etc. Techniques exist for moving objects within images; however, attempts to move the objects to different positions in the image can have disastrous results. For example, pixels associated with an object may be misidentified, leaving one section of the object in its original position while a remaining section of the object is moved to another (e.g., a chicken's body is moved while the chicken's feet remain). In another example, the empty spaces caused by removing pixels associated with a moving object may be filled with pixels that look out of place.In yet another example, the pixels surrounding a moving object may look different than the background, resulting in an image that looks poorly edited.

[0015] The technology described below advantageously solves these problems by providing an incomplete object as input to a diffusion machine learning model, referred to herein as a diffusion model, and outputting a complete object. An incomplete object is a partial representation of an object in an image. The object is partially (and not entirely) present in the image. A portion of the object that is not present in the image is referred to herein as the "dropped portion" of the incomplete object. A user can select the incomplete object in an output image and move the position of the incomplete object.

[0016] The space left by the incomplete object is reconstructed using reconstructed pixels to create a reconstructed image. A complete object is a complete representation of the object that includes the incomplete object and the omitted portion of the incomplete object. A reconstructed image is an image that differs from the original image in that pixels of the incomplete object associated with the incomplete object are removed from the reconstructed image and the pixels of the incomplete object are replaced with reconstructed pixels, which can be selected based on proximity to surrounding pixels, selected from a reference image that includes background pixels, etc.

[0017] The diffusion model ensures that the complete object fits into the new position. For example, when an object is moved from the background to the foreground, the diffusion model increases the size of the moving object. The diffusion model outputs a modified image that blends the complete object seamlessly with the reconstructed image by blending one or more versions of the complete object with one or more versions of the reconstructed image using the object mask. For example, the diffusion model can blend increasingly noisy versions of the complete object with corresponding noisy versions of the reconstructed image, while also generating denoised versions of the complete object and corresponding denoised versions of the reconstructed image.A noisy version of the full object is created by increasing the entropy of the image. More noise makes the details of the full object less visible in the image. Similarly, a noisy version of the reconstructed image is created by increasing the entropy of the reconstructed image. More noise makes the details of the reconstructed image less visible.

[0018] By using the diffusion model instead of other machine learning models, the media application maintains a realistic appearance of the modified image under a wide variety of situations. The technology described below enables the efficient correction of defective images, i.e., images containing incomplete objects. Complete objects created using the diffusion model are of high quality and do not contain the defects described above. The image processing described herein effectively and efficiently corrects images for incomplete objects present in the images. Exemplary environment 100

[0019] Fig. 1 illustrates an exemplary block diagram of an exemplary environment 100. In some embodiments, the environment 100 includes a media server 101, a user device 115a, and a user device 115n coupled to a network 105. The users 125a, 125n may be associated with the respective user devices 115a, 115n. In some embodiments, the environment 100 may include other servers or devices that are Fig. 1 are not shown. In Fig. In Figure 1 and the remaining figures, a letter following a reference number, e.g., "115a," represents a reference to the element having that particular reference number. A reference number in the text without a following letter, e.g., "115," represents a general reference to embodiments of the element bearing that reference number.

[0020] The media server 101 may include a processor, memory, and network communication hardware. In some embodiments, the media server 101 is a hardware server. The media server 101 is communicatively coupled to the network 105 via a signal line 102. The signal line 102 may be a wired connection, such as Ethernet, coaxial cable, fiber optic cable, etc., or a wireless connection, such as Wi-Fi®, Bluetooth®, or other wireless technology. In some embodiments, the media server 101 sends and receives data to and from one or more of the user devices 115a, 115n via the network 105. The media server 101 may include a media application 103a and a database 199.

[0021] The database 199 may store machine learning models, training datasets, images, etc. The database 199 may also store social network data associated with users 125, user preferences for the users 125, etc.

[0022] User device 115 may be a computing device including memory coupled to a hardware processor. For example, user device 115 may include a mobile device, a tablet computer, a mobile phone, a wearable device, a head-mounted display, a mobile email device, a portable game player, a portable music player, a reading device, or any other electronic device capable of accessing network 105.

[0023] In the illustrated embodiment, user device 115a is coupled to network 105 via signal line 108, and user device 115n is coupled to network 105 via signal line 110. Media application 103 may be stored as media application 103b on user device 115a and / or media application 103c on user device 115n. Signal lines 108 and 110 may be wired connections, such as Ethernet, coaxial cable, fiber optic cable, etc., or wireless connections, such as Wi-Fi®, Bluetooth®, or other wireless technology. User devices 115a, 115n are accessed by users 125a and 125n, respectively. User devices 115a, 115n in Fig. 1 are used as examples. While Fig. 1 illustrates two user devices 115a and 115n, the disclosure applies to a system architecture having one or more user devices 115.

[0024] The media application 103 may be stored on the media server 101 or the user device 115. In some embodiments, the operations described herein are performed on the media server 101 or the user device 115. In some embodiments, some operations may be performed on the media server 101 and some may be performed on the user device 115. The performance of operations occurs according to user preferences. For example, the user 125a may specify settings to perform operations on their respective device 115a and not on the media server 101. With such settings, the operations described herein are performed entirely on the user device 115a, and no operations are performed on the media server 101.Furthermore, a user 125a may specify that images and / or other user data should only be stored locally on a user device 115a and not on the media server 101. With such settings, no user data is transmitted to or stored on the media server 101. Transmission of user data to the media server 101, any temporary or permanent storage of such data by the media server 101, and the performance of operations on such data by the media server 101 will only occur if the user has consented to the transmission, storage, and performance of operations by the media server 101. Users are provided with options to change the settings at any time, e.g., to enable or disable the use of the media server 101.

[0025] Machine learning models (e.g., diffusion models, neural networks, or other types of models), when used for one or more operations, are stored and used locally on a user device 115 with specific user consent. Server-side models are only used if the user consents. Furthermore, a trained model may be provided for use on a user device 115. During such use, if the user 125 consents, training of the model may be performed on the device. Updated model parameters may be transmitted to the media server 101 if the user 125 consents, for example, to enable federated learning. Model parameters do not include any user data.

[0026] The media application 103 receives an output image. For example, the media application 103 receives an output image from a camera that is part of the user device 115, or the media application 103 receives the output image via the network 105. The media application 103 receives a selection of an incomplete object in the output image. The incomplete object is associated with a first position within the output image, and a omitted portion of the incomplete object is cropped by a boundary of the output image or obscured by another object. The incomplete object may be selected when a user 125 taps the object, draws a shape (e.g., a circle) around the object, confirms a suggestion from the media application 103 to modify the object, etc.

[0027] The media application 103 generates an object mask that includes pixels of the incomplete object associated with the incomplete object and removes the pixels of the incomplete object associated with the incomplete object from the source image. The media application 103 generates a reconstructed image that replaces the pixels of the incomplete object corresponding to the incomplete object with reconstructed pixels.

[0028] The media application 103 outputs a complete object using a diffusion model. For example, if an incomplete object is clipped by the edges of a source image and the incomplete object is moved to the center of the source image, the diffusion model outputs a complete object that fills the missing portion of the incomplete object. The media application 103 outputs a modified image using the diffusion model by blending one or more versions of the complete object with one or more versions of the reconstructed image using the object mask, with the complete object positioned at a second position in the modified image that is different from the first position in the source image.In some embodiments, the modified image may include a watermark or other indicator to indicate that the modified image was generated using a machine learning model.

[0029] In some embodiments, media application 103 may be implemented using hardware including a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a machine learning processor / coprocessor, any other type of processor, or a combination thereof. In some embodiments, media application 103a may be implemented using a combination of hardware and software. Example computing device 200

[0030] Fig. 2 is a block diagram of an example computing device 200 that may be used to implement one or more of the features described herein. Computing device 200 may be any suitable computing system, server, or hardware device. In one example, computing device 200 is a media server 101 used to implement media application 103a. In another example, computing device 200 is a user device 115.

[0031] In some embodiments, computing device 200 includes a processor 235, a memory 237, an input / output (I / O) interface 239, a display 241, a camera 243, and a storage device 245, all coupled via a bus 218. The processor 235 may be coupled to the bus 218 via the signal line 222, the memory 237 may be coupled to the bus 218 via the signal line 224, the I / O interface 239 may be coupled to the bus 218 via the signal line 226, the display 241 may be coupled to the bus 218 via the signal line 228, the camera 243 may be coupled to the bus 218 via the signal line 230, and the storage device 245 may be coupled to the bus 218 via the signal line 232.

[0032] Processor 235 may be one or more processors and / or processing circuits to execute program code and control basic operations of computing device 200. A "processor" includes any suitable hardware system, device, or mechanism that processes data, signals, or other information. A processor may include a system having a general-purpose central processing unit (CPU) with one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), multiple processing units (e.g.,in a multiprocessor configuration), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), associated circuitry for achieving functionality, a special-purpose processor for implementing processing based on neural network models, neural circuits, processors optimized for matrix computations (e.g., matrix multiplication), or other systems. In some embodiments, processor 235 may include one or more co-processors that implement neural network processing. In some embodiments, processor 235 may be a processor that processes data to produce a probabilistic output, e.g.,The output produced by processor 235 may be inaccurate or accurate within a range of an expected output. Processing need not be restricted to a particular geographic location or have time constraints. For example, a processor may perform its functions in real time, offline, in a batch mode, etc. Portions of processing may be performed at different times and locations by different (or the same) processing systems. A computer may be any processor in communication with a memory.

[0033] Memory 237 is provided in computing device 200 for access by processor 235 and may be any suitable storage medium readable by the processor, such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, etc., suitable for storing instructions for execution by the processor or sets of processors and separate from and / or integrated with processor 235. Memory 237 may store software run on computing device 200 by processor 235, including a media application 103.

[0034] The memory 237 may include an operating system 262, other applications 264, and application data 266. Other applications 264 may include, for example, an image library application, an image management application, an image gallery application, communication applications, web hosting engines or applications, media sharing applications, etc. One or more of the methods disclosed herein may operate in multiple environments and on multiple platforms, e.g., as a standalone computer program capable of running on any type of computing device, as a web application having web pages, as a mobile application ("app") capable of running on a mobile computing device, etc.

[0035] The application data 266 may be data generated by the other applications 264 or hardware of the computing device 200. For example, the application data 266 may include images used by the image library application and user actions identified by the other applications 264 (e.g., a social networking application), etc.

[0036] The I / O interface 239 may provide functionality to enable the computing device 200 to be linked to other systems and devices. Linked devices may be included as part of the computing device 200 or may be separate and communicate with the computing device 200. For example, network communication devices, storage devices (e.g., memory 237 and / or storage device 245), and input / output devices may communicate via the I / O interface 239. In some embodiments, the I / O interface 239 may be connected to interface devices such as input devices (keyboard, pointing device, touchscreen, microphone, scanner, sensors, etc.) and / or output devices (display devices, speaker devices, printers, monitors, etc.).

[0037] Some examples of interface devices that may be connected to the I / O interface 239 may include a display 241, which may be used to present content, such as images, videos, and / or a user interface of an output application, as described herein, and to receive touch (or gesture) input from a user. For example, the display 241 may be used to present a user interface that includes a graphical guide on a viewfinder. The display 241 may include any suitable display device, such as a liquid crystal display (LCD), light-emitting diode (LED), or plasma display, a cathode ray tube (CRT), television, monitor, touchscreen, or three-dimensional display, or other visual display device.For example, the display 241 may be a flat panel display provided on a mobile device, multiple display screens embedded in an eyeglass form factor or headset device, or a monitor screen for a computing device.

[0038] Camera 243 may be any type of imaging device capable of capturing images and / or videos. In some embodiments, camera 243 captures images or videos, which I / O interface 239 transmits to media application 103.

[0039] The storage device 245 stores data related to the media application 103. For example, the storage device 245 may store a training data set including labeled images, a machine learning model, outputs from the machine learning model, etc.

[0040] Fig. 2 illustrates an example media application 103 stored in memory 237 and including a user interface module 202, a segmenter 204, a reconstructor module 206, and a diffusion module 208.

[0041] The user interface module 202 generates graphical data for presenting a user interface that includes images. In some embodiments, the user interface module 202 receives an output image. The output image may be received from the camera 243 of the computing device 200 or from the media server 101 via the I / O interface 239. The output image includes a subject, such as a person. In some embodiments, the user interface includes options for selecting various people and other objects in the output image. For example, a user may select a person by tapping the person, circling an object, swiping over an object, etc. In some embodiments, the user interface generates recommendations for modifying the image, such as presenting text asking if the user wants to remove a bystander from the object.

[0042] In some embodiments, user interface module 202 updates the graphical data to include a highlighted version of the selected object whenever a user selects an object. A user can modify the selected object. For example, the user can drag and drop the selected object from a first location to a second location, the user can resize the image, the user can select a button to delete the image, etc.

[0043] In some embodiments, the segmenter 204 generates a segmentation score that reflects a quality of identifying pixels associated with the selected object in the source image. The user interface may include various options for modifying the selected object based on the segmentation score. For example, if the segmentation score exceeds a threshold, the user interface module 202 provides an option to move the selected object, replace the selected object with another object, or delete the selected object. In another example, if the segmentation score does not exceed the threshold, the user interface module 202 does not provide an option to move the selected object, but instead provides the options to replace the selected object or delete the selected object.

[0044] In some embodiments, the user interface module 202 generates graphical data for displaying a reconstructed image, moving a selected object from a first position to a second position within the image, resizing the selected image, adding an additional object, etc. The user interface may also include options for editing the reconstructed image, sharing the reconstructed image, adding the reconstructed image to a photo album, etc.

[0045] The segmenter 204 segments a selected object from an input image by identifying pixels corresponding to the selected object. In some embodiments, the segmenter 204 uses an alpha map as part of a technique for distinguishing the foreground and background of the input image during segmentation. The segmenter 204 may also identify a texture of the selected object in the foreground of the input image. In some embodiments, the segmenter 204 generates a segmentation map that identifies pixels associated with one or more objects in the input image. For example, the segmentation map may include identification of pixels associated with the selected object.

[0046] The segmenter 204 may perform segmentation by detecting objects in a source image. The object may be a person, an animal, a car, a building, etc. A person may be a subject of the source image or not the subject of the source image (i.e., a bystander). A bystander may include people walking by, running by, riding a bicycle, standing behind the subject, or otherwise within the source image. In various examples, a bystander may be in the foreground (e.g., a person walking across the frame in front of the camera), at the same depth as the subject (e.g., a person standing to the side of the subject), or in the background. In some examples, more than one bystander may be in the source image. The bystander may be a human in any pose, e.g., standing, sitting, crouching, lying down, jumping, etc.The viewer can face the camera, stand at an angle to the camera, or face away from the camera.

[0047] Segmenter 204 may detect object types by performing object detection, which compares objects to object priors of people, vehicles, buildings, etc., to identify expected object shapes and determine whether pixels are associated with a selected object or background. Segmenter 204 may generate a region of interest for the selected object, such as a bounding box with x, y coordinates and a scale.

[0048] The segmenter 204 generates one or more object masks for one or more selected objects in the source image. The object mask represents a region of interest. The object mask is described in more detail below with reference to the diffusion model.

[0049] In some embodiments, one or more object masks are generated based on generating superpixels for the image and matching superpixel centroids with depth map values ​​(e.g., obtained by camera 243 using a depth sensor or by inferring depth from pixel values) to cluster detections based on depth. Specifically, depth values ​​in a masked area can be used to determine a depth range, and superpixels can be identified that fall within the depth range. Another technique for generating a mask involves weighting depth values ​​based on how close the depth values ​​are to the object mask, where weights have been represented by a distance transform map.

[0050] In some embodiments, the segmenter 204 uses a machine learning algorithm, such as a neural network or, more specifically, a convolutional neural network, to segment the output image and generate the object mask. The segmenter 204 may specify a circuit configuration (e.g., for a programmable processor, for a field-programmable gate array (FPGA), etc.) that enables the processor 235 to apply a machine learning model. In some embodiments, the segmenter 204 may include software instructions, hardware instructions, or a combination. In some embodiments, the segmenter 204 may provide an application programming interface (API) that may be used by the operating system 262 and / or other applications 264 to invoke the segmenter 204, for example, to apply the machine learning model to application data 266 to output the object mask.

[0051] Segmenter 204 uses training data to generate a trained machine learning model. For example, training data may include pairs of source images with one or more objects and output images with one or more corresponding object masks.

[0052] Training data may be obtained from any source, e.g., a data repository specifically marked for training, data for which consent is provided for use as machine learning training data, etc. In some embodiments, training may occur on the media server 101 providing the training data directly to the user device 115, training may occur locally on the user device 115, or a combination of both.

[0053] In some embodiments, segmenter 204 uses weights taken raw from / transferred to another application. For example, in these embodiments, the trained model may be generated on another device and provided as part of segmenter 204. In various embodiments, the trained model may be provided as a data file including a model structure or form (e.g., defining a number and type of neural network nodes, connectivity between nodes, and organization of the nodes into multiple layers) and associated weights. Segmenter 204 may read the data file for the trained model and implement neural networks with node connectivity, layers, and weights based on the model structure or form specified in the trained model.

[0054] The trained machine learning model may include one or more model forms or structures. For example, model forms or structures may include any type of neural network, such as a linear network, a deep learning network that implements multiple layers (e.g., "hidden layers" between an input layer and an output layer, where each layer is a linear network), a convolutional neural network (e.g., a network that divides and splits input data into multiple pieces or tiles, processes each tile separately using one or more neural network layers, and aggregates the results from processing each tile), a sequence-to-sequence neural network (e.g., a network that receives sequential data as input, such as words in a sentence, frames in a video, etc., and produces a sequence of results as output), etc.

[0055] The model shape or structure can specify the connectivity between different nodes and the organization of nodes into layers. For example, the nodes of a first layer (e.g., an input layer) can receive data as input data or application data. Such data can, for example, include one or more pixels per node, e.g., when the trained model is used to analyze, e.g., an initial image. Subsequent intermediate layers can receive as input the output of nodes of a previous layer according to the connectivity specified in the model shape or structure. These layers can also be referred to as hidden layers. For example, a first layer can output a segmentation between a foreground and a background. A final layer (e.g., output layer) produces an output of the machine learning model.For example, the output layer may receive the segmentation of the source image into a foreground and a background and output whether a pixel is part of an object mask or the rest of the source image. In some embodiments, the model shape or structure also specifies a number and / or type of nodes in each layer.

[0056] In various embodiments, the trained model may include one or more models. One or more of the models may include a plurality of nodes arranged in layers per model structure or shape. In some embodiments, the nodes may be memory-less compute nodes, e.g., configured to process an input unit to produce an output unit. The computation performed by a node may include, for example, multiplying each of a plurality of node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum with a bias or cut value to produce the node output. In some embodiments, the computation performed by a node may also include applying a step / activation function to the adjusted weighted sum.In some embodiments, the step / activation function may be a nonlinear function. In various embodiments, such a computation may include operations such as matrix multiplication. In some embodiments, computations may be performed in parallel by the plurality of nodes, e.g., using multiple processor cores of a multi-core processor, using individual processing units of a graphics processing unit (GPU), or special-purpose neural circuits. In some embodiments, nodes may include memory, e.g., capable of storing one or more previous inputs and using them when processing a subsequent input. For example, nodes with memory may include nodes with long short-term memory (LSTM).LSTM nodes can use memory to maintain a “state” that allows the node to act like a finite state machine (FSM).

[0057] In some embodiments, the trained model may include embeddings or weights for individual nodes. For example, a model may be initiated as a plurality of nodes organized into layers, as specified by the model shape or structure. Upon initialization, a respective weight may be applied to a connection between each pair of nodes connected according to the model shape, e.g., nodes in successive layers of the neural network. For example, the respective weights may be randomly assigned or initialized to default values. The model may then be trained, e.g., using training data, to produce a result.

[0058] Training may involve applying supervised learning techniques. In supervised learning, the training data may include a variety of inputs (e.g., source images, object masks, object masks, etc.) and a corresponding ground-truth output for each input (e.g., a ground-truth mask that correctly identifies the object in each image). Based on a comparison of the model's output with the ground-truth output, weight values ​​are automatically adjusted, e.g., in a way that increases the probability that the model will produce the ground-truth output for the image.

[0059] In various embodiments, a trained model includes a set of weights or embeddings that correspond to the model structure. In some embodiments, the trained model may include a set of weights that are fixed, e.g., downloaded from a server that provides the weights. In various embodiments, a trained model includes a set of weights or embeddings that correspond to the model structure. In embodiments where data is omitted, the segmenter 204 may generate a trained model based on previous training, e.g., by a developer of the segmenter 204, by a third party, etc. In some embodiments, the trained model may include a set of weights that are fixed, e.g., downloaded from a server that provides the weights.

[0060] In some embodiments, the trained machine learning model receives an input image with one or more selected objects. In some embodiments, the trained machine learning model outputs one or more object masks that include the one or more objects.

[0061] After the one or more object masks are output by the segmenter 204 (e.g., from the machine learning model), the segmenter 204 removes the one or more selected objects from the output image.

[0062] The reconstructor module 206 generates a reconstructed image that replaces object pixels corresponding to one or more objects with reconstructed pixels. The reconstructed pixels may be based on pixels from a reference image of the same location without the objects. Alternatively, the reconstructor module 206 may identify reconstructed pixels to replace the removed object based on a proximity of the reconstructed pixels to other pixels surrounding the object. The reconstructor module 206 may use a gradient of neighborhood pixels to determine properties of the reconstructed pixels. For example, where a bystander was standing on the ground, the reconstructor module 206 replaces the reconstructed pixels with pixels of the ground.Other reconstruction techniques are possible, including a machine learning-based reconstruction technique that outputs reconstructed pixels based on training data containing images of similar structures.

[0063] In embodiments where a user has decided to delete the selected object, the user interface module 202 may display the reconstructed image in which the selected object has been removed and the selected object pixels have been replaced with reconstructed pixels.

[0064] In embodiments where the user decides to move the selected object, replace the selected object, or add an object to an image, a diffusion module 208 performs blending of an object with an object mask and a reconstructed image using a diffusion model. The diffusion model may receive an object mask, an incomplete object, and a reconstructed image as input. The diffusion model may receive additional inputs, such as a text requirement, a number of pixels to fill to output a complete object based on an incomplete object, dimensions for a modified image, etc.

[0065] Diffusion models include a forward process, where the diffusion model adds noise to the data, and a backward process, where the diffusion model learns to recover the data from the noise. For example, when a selected object is moved from a first position to a second position, the diffusion module 208 applies the diffusion model by blending the selected object with progressively noisier versions and then progressively denoiser versions of the reconstructed image. In some embodiments, an object-stitch diffusion model is used to move an object from a first position to a second position. In some embodiments, a generational diffusion model is used when the object is an incomplete object and a portion of the object is generated and / or for new objects generated based on text prompts. Object stitch diffusion model

[0066] In some embodiments, the object stitch diffusion model is used when an object is moved from a first position to a second position. In some embodiments, the diffusion module 208 includes an object image encoder that extracts semantic features from a selected object, a diffusion model that blends an object with an image, and a content adapter that transforms a sequence of visual tokens into a sequence of text tokens to overcome a domain gap between image and text. In some embodiments, the diffusion module 208 trains the diffusion model using self-supervision based on training data, where the training data includes image and text pairs. In some embodiments, the diffusion model is trained with synthetic data that simulates real-world scenarios.The diffusion model can also be trained using data augmentation, which is generated by introducing random shifts and clipping augmentations during training while ensuring that the foreground object is contained within the clipping window.

[0067] The content adapter is trained during a first stage using image and text pairs to maintain high semantics of the object. During a second stage, the content adapter is trained within the diffusion model to encode key identity features of the object by promoting the visual reconstruction of the object in the original image. The diffusion model can be trained on an embedding produced by the content adapter using cross-attention blocks.

[0068] The diffusion model uses an object mask to blend the reconstructed image with the object. The diffusion model can denoise the masked region. The content adapter can transform the visual features from the object image encoder into text features (tokens) to use as conditioning for the diffusion model. Generative diffusion model

[0069] In some embodiments, the generative diffusion model is used to output a complete object based on an incomplete object or to generate a new object. The diffusion module 208 trains the generative diffusion model based on training data. The training data may include image and text pairs used to create an embedding space for images and text. The image and text pairs may include an image associated with a corresponding text, such as an image of a dog and text including "pit bull." The diffusion module 208 may be trained with a loss that reflects a cosine distance between an embedding of a text prompt and an embedding of an estimated unaltered image (i.e., with no text-generated objects).

[0070] The diffusion module 208 may use the training data to perform text conditioning, where text conditioning describes the process of outputting objects conditioned with a text prompt. The diffusion module 208 may train a neural network to output the object based on the text prompt provided by a user or by the media application. For example, the text prompt may be a suggestion generated by the media application based on the context of the source image (e.g., if the source image is a beach, the text prompt may be a beach ball, a turtle, etc.).

[0071] In some embodiments, the diffusion module 208 may output at least a portion of a missing part of an object based on receiving an incomplete object as input data, a position within the image to which the incomplete object is moved, and dimensions of the output (e.g., the original dimensions or modified dimensions if the object is resized). For example, if a user selects an object in a user interface that is partially clipped by a boundary and moves the object from a first position to a second position, where the second position also clips a portion of the object, the diffusion module 208 may output a modified object that includes more of the object visible in the image based on moving the object.In some embodiments, the diffusion module 208 may output a complete object based on an incomplete object selected by a user. For example, if a user selects a beach ball that is partially occluded by another object, the diffusion model may be trained to output a complete beach ball.

[0072] In some embodiments, the diffusion module 208 generates progressively noisier versions of the full object compared to a previous version and progressively noisier versions of the reconstructed image compared to a previous version. For example, a forward Markov process of denoising produces a series of noisy reconstructed images by gradually adding Gaussian noise until a nearly isotropic sample of Gaussian noise is obtained. The forward process of denoising defines a progression of image manifolds, where each manifold consists of noisy images.

[0073] The diffusion module 208 may spatially blend noisy versions of the full object with corresponding noisy versions of the reconstructed image using the object mask. For example, the diffusion module 208 may blend each noisy version of the full object with each corresponding noisy version of the reconstructed image using the object mask, where the object mask defines the boundaries of the full object such that the object mask limits the region modified during the blending process. In some embodiments, the diffusion process may include a local diffusion through the full object, where the image generation loss determined during the training process is used during the positional object generation diffusion under the object mask.

[0074] The diffusion module 208 may perform a diffusion step that denoises a latent space in a direction dependent on a text prompt. The diffusion module 208 generates progressively noisier versions of the full object compared to a previous version and progressively noisier versions of the reconstructed image compared to a previous version. For example, the backward Markov process transforms a sample of Gaussian noise by repeatedly denoising the reconstructed image using a learned posterior. Each step of the denoising diffusion process projects a noisy image onto the next, less noisy manifold.

[0075] The diffusion module 208 performs the denoising diffusion step after each blend to restore coherence by projecting onto the next manifold. After spatial blending is complete, the diffusion module 208 preserves the background by replacing an area outside the object mask with a corresponding area from the reconstructed image.

[0076] In some embodiments, the diffusion module 208 uses cross-domain compositing to apply an iterative refinement scheme to infuse an object with contextual information to match the style of the reconstructed image. For example, if the object is generated for an indoor environment and is added to a reconstructed outdoor environment image, the object may be modified to be brighter to match the reconstructed image. In another example, if the object is in a shadow at a first location and the second location is in full sun, the object may be modified to match the brightness of the second location. Object distance model

[0077] In some embodiments, instead of using the segmenter 204 to remove an object and using the reconstructor module 206 to add pixels to the removed region in an output image, the diffusion model 208 is trained to include an object removal model.

[0078] The diffusion module 208 generates counterfactual training data to train the diffusion model to include an object distance model. For each counterfactual image pair, the diffusion module 208 captures a factual image containing an object in a scene; physically removes the object while avoiding camera movement, lighting changes, or movement of other objects; captures a counterfactual image of the scene without the object; and segments the factual image to create an object mask. Segmenting the factual image involves creating a segmentation map (M o) for the object O that was removed from the factual image X.

[0079] The diffusion module 208 creates a combined image for each image pair that includes the factual image and the object mask and the counterfactual image. The object mask may be a binary object mask (M o (X)) and the counterfactual image pairs can be considered as an input pair of the factual image and the binary object mask (X, M o (X)) and the output counterfactual image (T cf ) are described.

[0080] The diffusion module 208 estimates the distribution of the counterfactual images P(X cf |X = x,M o (X)) using the factual image x and the binary object mask by training the diffusion model based on the use of the counterfactual image pairs. The diffusion module 208 determines the estimation by minimizing a loss function ℒ(θ) using the following equation: L(θ)=Et∼U([0,T]),ε∼N(0,I)[∑i=1N‖Dθ(αtxicf+σtε,xi,Mo(xi),t,p)−ε‖2] where D θ (x t , x cond , m, t, p) is a denoising network with the following inputs: noisy latent representation of the counterfactual image x̃ t , latent representation of the image containing the object to be removed, x cond , mask m, which specifies the position of the object, a timestamp t and encoding of an empty string (text prompt) p. x t is calculated using the following forward process equation: x˜t=αt⋅x+σt⋅ε where x represents the image without the object (the counterfactual), α t and σ t be determined by the noise plan, and ε∼N(O,I).

[0081] Once the diffusion model, which includes an object removal model, has been trained, the user interface module 202 may receive a request to remove a selected object from the first modified image. A source image and the request are provided as input to the object removal model, which outputs a modified image that does not include the selected object. Object insertion model

[0082] In some embodiments, instead of using the segmenter 204 to remove an object, using the reconstructor module 206 to add pixels to the removed region in an output image, and using the diffusion model 208 to blend the object with the pixels at a new position, the diffusion model 208 is trained to include an object insertion model.

[0083] In some embodiments, the object insertion model is trained with a number of image pairs that exceeds the number of available counterfactual image pairs. As a result, the diffusion module 208 generates synthetic training data. For each synthetic image pair, the diffusion module 208 selects original images that include objects, uses the object removal model to output modified images from the original images without the objects, generates an input image by inserting the object into the modified image, and segments the original image to create the object masks. The modified images that lack the objects are identified as z using the following equation: i designated: zi∼P(Xcf|xi,Mo(xi)) where the original images x1, x2,..., x n and the corresponding object masks M o (x1), M o (x2),..., M o (x n). The diffusion module 208 generates the input image by inserting the object into objectless scenes z i to obtain images without shadows and reflections using the following equation: yi=Mo(xi)⊙xi+(1−Mo(xi))⊙zi

[0084] The synthetic image pairs are (y i ,M o (x i )) and the corresponding targets are the original images x i While both the input and output images contain the object o, the input images do not contain the object's effects on the scene, while the output images do. In some embodiments, the diffusion module 208 trains the object insertion model with the diffusion objective shown in Equation 1.

[0085] For each synthetic image pair, the diffusion module 208 creates a second combined image that includes the original image, the object mask, and the input image. The diffusion module 208 pre-trains the diffusion model based on synthetic image pairs to include an object insertion model and fine-tunes the diffusion model to include the object insertion model based on the counterfactual image pairs used to train the object insertion model.

[0086] In some embodiments, the user interface module 202 generates graphical data for representing a user interface that provides the user with the option to set the position of the object and the option to resize the object. The diffusion module 208 adds a selected object removed from the source image to the new position. In some embodiments, the diffusion module 208 provides the selected object as input to a diffusion model, as well as a position where the selected object will be located in a modified image, and outputs a modified image that blends the selected object with the reconstructed image. For example, the diffusion module 208 may spatially blend noisy versions of the reconstructed image with noisy versions of the selected object.

[0087] In some embodiments, the diffusion module 208 may add a shadow to the selected object at the new location. The shadow may match a light direction in the image. For example, if the sun is casting rays from the upper left corner of the image, the shadow may be rendered to the right of the person and / or object. In some embodiments, the diffusion module 208 uses a machine learning model to output a shadow mask, which is used to generate the shadow associated with the object.

[0088] Once the selected object has been added to the reconstructed image, the user interface module 202 may include additional features for modifying the reconstructed image, such as an option for changing the lighting of the reconstructed image.

[0089] Fig. 3A illustrates an exemplary source image 300. The source image includes a person 301 who is a subject of the source image 300, a bystander 302 who is in the foreground of the source image 300, grass 303, a road 304, trees 305, and a cloudy sky 306.

[0090] Fig. Figure 3B illustrates an exemplary output image 310 in which the two objects from Fig. 3A for modification. The two objects are represented with outlines 311, 312, corresponding to how the user selected the two objects. A user may have selected the two objects using a finger, mouse, or other object to circle, swipe, double-tap the two objects, etc. The user interface module 202 may use an object selection tool, a lasso tool, an artificial intelligence segmentation tool, etc., to identify the object in the source image 310. In some embodiments, the user interface module 202 may have suggested selecting the two objects, and the two objects were highlighted in response to the user confirming the selection.

[0091] After the two objects have been selected, a segmenter 204 segments the objects from the source image 310 and creates object masks. Fig. Figure 3C illustrates an exemplary output image 320 with object masks 321, 322 surrounding the two objects. The person is surrounded by a first object mask 321, and the bystander is surrounded by a second object mask 322. The segmenter 204 removes the person and the bystander from the output image 320.

[0092] After the person and the bystander from the initial image 320 of the Fig. 3C, the reconstructor module 206 generates a reconstructed image that replaces object pixels corresponding to removed objects with reconstructed pixels. Fig. 3D illustrates an example reconstructed image 330 in which the bystander has been removed from the original image 320 and the person 331 has been moved and resized.

[0093] In some embodiments, the diffusion module 208 resizes objects to be larger or smaller than the object in the source image. For example, the object may be resized to be larger when moved forward or smaller when moved backward. In this example, a user provided input to resize person 331 to be smaller, and the diffusion module 208 resized person 331 and blended the person with the new position. In some embodiments, moving person 331 may be a separate action from resizing, or both moving and resizing may be part of the same action.

[0094] Additional changes can be made to the reconstructed image. Fig. Figure 3E illustrates an exemplary reconstructed image 340 in which the road 332 is Fig. 3D was replaced by grass 341. The reconstructed image 340 also includes a first tree species, which was replaced by a second tree species 342. The reconstructed image 340 also includes a clear sky with sunlight 343 coming from the upper right corner of the sky, through which the cloudy sky from Fig. 3D. The direction of the sun 343 causes the diffusion module 208 to output a shadow 344 that matches the person 345. In some embodiments, the road is replaced with grass 341 by outputting the grass with the diffusion module 208 and blending the grass with the reconstructed image.

[0095] In some embodiments, the diffusion module 208 receives an incomplete object as input and outputs a complete object. The diffusion module 208 may also add the complete object to a second location within a reconstructed image by blending pixels of the complete object corresponding to the complete object with reconstructed pixels.

[0096] In some embodiments, if an object was captured at an edge of the image and a portion of the complete object was missing, the diffusion module 208 may complete the missing portions of the object before performing blending. For example, if a woman is wearing a long dress and a portion of the dress is missing, the diffusion module 208 may output a complete dress based on the incomplete dress.

[0097] Fig. 4A illustrates an exemplary source image 400 of a child 405 sitting on a bench 410 and holding balloons 415 that are partially cropped by a boundary of the source image 400. In this example, a user interface module 202 provides a user interface with an option to select objects for a user. The user selects the child 405, the bench 410, and the balloons 415 at a first position where the balloons 415 represent an incomplete image. The segmenter 204 segments the child 405, the bench 410, and the balloons 415 to separate the objects from the source image 400.

[0098] The user interface module 202 includes an option to move the selected objects to a different position. The user selects a second position. The segmenter 204 removes the selected objects from the source image. A reconstructor module 206 generates a reconstructed image that replaces object pixels corresponding to removed objects with reconstructed pixels.

[0099] A diffusion module 208 receives as input the selected objects and coordinates for the second position and outputs balloons that are complete objects and a longer bank. Fig. 4B illustrates an example modified image 450 in which the child 455, the bench 460, and the balloons 465 have been moved to a second position. In this example, the diffusion module 208 outputs a modified image that blends one or more versions of the child 455, the bench 460, and the balloons 465 with one or more versions of the reconstructed image using the object mask.

[0100] In some embodiments, the user interface module 202 receives a command to expand the cropping of an image from the user interface. The command to expand the cropping may be performed on a modified image, a source image, etc. The command to expand the cropping may be a button that is part of a user interface that is a suggestion to help center an object, such as a person, in the center of an image. In some embodiments, the command may be based on a user specifying a new border for the image by directly expanding the edges of the image or by moving a selected object that expands the edges of the image.

[0101] The reconstructor module 206 receives a cropped image and cropped image dimensions as input and outputs a cropped image that replaces edges between the image and the cropped image with reconstructed pixels that match the image. For example, if the edge is water, the reconstructor module 206 may use water pixels for the cropped image portion of the image.

[0102] Fig. 5A illustrates an example user interface 500 of a source image 504 that includes a button 412 for changing an edge of a source image 504. In this example, the user interface 500 includes a first button 510 for editing an image and a second button 512 for changing the edges of the image. Other mechanisms for providing commands for expanding the image's field of view are possible. For example, a user can select an edge 506 of the source image 504 and drag and drop the edge 506 to indicate where the user wants a new edge to end.

[0103] The user may change the edge of the source image because the person 508 is not centered in the image and cropping the image to reduce the image on the right side would result in an image that is excessively narrow.

[0104] Fig. 5B illustrates an example user interface 515 with an indicator 521 used to expand an edge of the source image 517. In this example, the user has clicked and dragged the indicator 521 to expand the left edge of the source image 517. The expanded area 519 is illustrated with pixelated content while the reconstructor module 206 generates the expanded image.

[0105] In some embodiments, the reconstructor module 206 receives as input a cropped image and cropped image dimensions. For example, the dimensions include the length and width of the new border on the left side of the source image.

[0106] The reconstructor module 206 outputs a cropped image that includes reconstructed pixels between the edges of the cropped image of the modified image and the expanded edges based on the dimensions. In this case, the reconstructor module 206 copies pixels for bushes, a rock, water, and flowers. Fig. 5C illustrates an example user interface 530 of an enhanced cropped image 532 output based on the source image.

[0107] Fig. 5D illustrates an alternative example user interface 540 in which a selected person 544 is used to expand edges of the expanded image section of the source image 542. In this alternative example, instead of an indicator, such as indicator 521 in Fig. 5B, or an edge of the rim, such as edge 506 in Fig. 5A, to extend the edges of the expanded image section of the source image 542, select an object in the image and move the object to extend the edge. In Fig. 5D, a user has moved person 544 to the edge of the source image 542. The new position of person 544 defines the new edge to be created for an enlarged image, where 555 corresponds to the enlarged area. Exemplary procedures

[0108] Fig. 6 illustrates an exemplary flowchart of a method 600 for generating a modified image of a complete object from an incomplete object, according to some embodiments described herein. The method 600 may be performed by the computing device 200 in Fig. 2. In some embodiments, method 600 is performed by user device 115, media server 101, or partially on user device 115 and partially on media server 101.

[0109] The procedure 600 from Fig. 6 may begin at block 602. At block 602, it is determined whether a user has granted permission to access a source image. If no permission has been granted, the method 600 ends. If permission has been granted, block 602 may be followed by block 604.

[0110] At block 604, a selection of an incomplete object in a source image is received, the incomplete object being associated with a first position within the source image, and an omitted portion of the incomplete object being clipped by a boundary of the source image or obscured by another object. For example, the incomplete object may be a car clipped by the edges of the source image. A user may select the incomplete object by clicking on an object, circling an object, accepting a suggestion generated by a user interface to modify an object, etc. Block 604 may be followed by block 606.

[0111] At block 606, an object mask is generated that includes pixels of the incomplete object associated with the incomplete object. The pixels of the incomplete object may be determined by segmenting the output image to identify pixels associated with the incomplete object in the initial object. Block 606 may be followed by a block 608.

[0112] At block 608, the pixels associated with the incomplete object are removed from the original image. Block 608 may be followed by block 610.

[0113] At block 610, a reconstructed image is generated that replaces pixels of the incomplete object with reconstructed pixels. Block 610 may be followed by block 612.

[0114] At block 612, the object mask, the incomplete object, and the reconstructed image are provided as input to a diffusion model. Block 612 may be followed by block 614.

[0115] At block 614, the diffusion model outputs a complete object. For example, the diffusion model may receive the incomplete object, a second position where the complete object should be placed in a modified image, and dimensions of the complete object, including resized dimensions if a user has resized the incomplete object or if moving from a first position to a second position results in a resize of the incomplete object. Block 614 may be followed by a block 616.

[0116] At block 616, a modified image is created by blending one or more versions of the full object with one or more versions of the reconstructed image using the object mask, with the full object positioned at a second position in the modified image that differs from the first position in the source image. Continuing the above example, the full image may include a full version of the car. The car may be resized based on its being moved from a foreground to the background, and its size may be reduced to account for the distance reflected by its being positioned in the background.

[0117] In some embodiments, the modified image is a first modified image, and the method further includes receiving a request to remove a selected object from the first modified image; providing the first modified image and the request as input to an object removal model associated with the diffusion model; and outputting, with the object removal model, a second modified image that does not include the selected object.In some embodiments, the modified image is a first modified image, and the method further includes receiving a request to move a selected object from a third position to a fourth position; providing the first modified image and the request as input to an object insertion model associated with the diffusion model; and outputting, with the object insertion model, a second modified image including the selected object at a fourth position based on the request.

[0118] In some embodiments, the method further includes receiving a request to add an additional object to the source image; outputting, with the diffusion model, the additional object; and outputting, with the diffusion model, a modified image by blending one or more versions of the additional object with one or more versions of the reconstructed image, wherein the additional object is positioned at a third position in the reconstructed image that is different from the second position in the modified image. In some embodiments, the request to add the additional object includes a text prompt describing the additional object, and the diffusion model uses generative artificial intelligence to output the additional object.

[0119] In some embodiments, the size of the selected person in the reconstructed image can be changed to account for their movement forward or backward from the first position. For example, the selected person can be made smaller or larger than the person in the source image. In some embodiments, the diffusion model changes the size of the entire object based on a change from the first position in the source image to the second position in the modified image.

[0120] A shadow corresponding to the selected person at the other position may also be created, wherein the shadow matches a direction of light in the image. In some embodiments, a first image in the source image is replaced with a second object from the source image, wherein the modified image includes replacing the first object with the second object. In some embodiments, illumination of the reconstructed image is also modified. For example, the sky may be made brighter, darker, with denser clouds to reduce illumination, etc. In some embodiments, the method further includes modifying illumination of the modified image and adding a shadow to the entire object based on a direction of illumination of the modified image.

[0121] Fig. 7 illustrates an exemplary flowchart of a method 700 for outputting an enhanced image from an output image, according to some embodiments described herein. The method 700 may be performed by the computing device 200 in Fig. 2. In some embodiments, method 700 is performed by user device 115, media server 101, or partially on user device 115 and partially on media server 101.

[0122] The procedure 700 in Fig. 7 may begin at block 702. At block 702, an initial image is displayed in a user interface. Block 702 may be followed by block 704.

[0123] In block 704, a command to expand the image section of the source image is received to expand edges of the expanded image section of the source image into expanded edges, the command being based on at least one action selected from the group of selecting a button to expand the image section, moving an indicator to define the expanded edges, moving an edge of the source image to define the expanded edges, moving a selected object to define the expanded edges, and combinations thereof. Block 704 may be followed by a block 706.

[0124] At block 706, an extended crop image is output that includes reconstructed pixels between the edges of the extended crop of the source image and the extended edges based on the command.

[0125] Fig. Figure 8 illustrates an exemplary flowchart of a method 800 for training an object distance model according to some embodiments described herein. The method 800 may be performed by the computing device 200 in Fig. 2. In some embodiments, method 800 is performed by user device 115, media server 101, or partially on user device 115 and partially on media server 101.

[0126] The method 800 may begin at a block 802. At block 802, counterfactual training data is generated for each counterfactual image pair by: capturing a factual image containing an object in a scene; physically removing the object while avoiding camera movement, lighting changes, or movement of other objects; capturing a counterfactual image of the scene without the object; and segmenting the factual image to create an object mask. Block 802 may be followed by a block 804.

[0127] At block 804, a combined image is created for each counterfactual image pair, including the factual image, the object mask, and the counterfactual image. Block 804 may be followed by block 806.

[0128] At block 806, the diffusion model is trained to include an object distance model based on the use of counterfactual image pairs.

[0129] Fig. 9 illustrates an exemplary flowchart of a method 900 for training an object insertion model according to some embodiments described herein. The method 900 may be performed by the computing device 200 in Fig. 2. In some embodiments, method 900 is performed by user device 115, media server 101, or partially on user device 115 and partially on media server 101.

[0130] The method 900 may begin at a block 902. At block 902, counterfactual training data is generated for each counterfactual image pair by: capturing a factual image containing an object in a scene; physically removing the object while avoiding camera movement, lighting changes, or movement of other objects; capturing a counterfactual image of the scene without the object; and segmenting the factual image to create an object mask. Block 902 may be followed by a block 904.

[0131] At block 904, a combined image is created for each counterfactual image pair, including the factual image, the object mask, and the counterfactual image. Block 904 may be followed by block 906.

[0132] At block 906, the diffusion model is trained to include an object distance model based on the use of counterfactual image pairs. Block 906 may be followed by block 908.

[0133] At block 908, synthetic training data is generated for each synthetic image pair by: selecting original images that contain objects; using the object removal model to output modified images from the original images without the objects; generating an input image by inserting the object into the modified image; and segmenting the original image to create the object masks. Block 908 may be followed by a block 910.

[0134] At block 910, a second combined image is created for each synthetic pair, including the original image, the object mask, and the input image. Block 910 may be followed by a block 912.

[0135] At block 912, the diffusion model is pre-trained to include an object insertion model based on the use of synthetic image pairs. Block 912 may be followed by block 914.

[0136] At block 914, based on the use of counterfactual image pairs, the diffusion model is fine-tuned to include the object insertion model.

[0137] In addition to the foregoing descriptions, a user may be provided with controls that allow the user to choose whether and when the systems, programs, or features described herein enable the collection of user information (e.g., information about a user's social network, a user's social actions or activities, a user's occupation, a user's preferences, or a user's current location) and whether the user is sent content or communications from a server. In addition, certain data may be treated in one or more ways prior to storage or use so that personally identifiable information is removed.For example, a user's identity may be treated in a way that prevents personally identifiable information from being determined for the user, or a user's geographic location may be generalized when location information is obtained (such as at the city, ZIP code, or state level) so that a specific user location cannot be determined. Thus, the user can control what information is collected about them, how that information is used, and what information is provided to the user.

[0138] In the foregoing description, for the sake of explanation, numerous specific details are set forth in order to provide a thorough understanding of the description. However, it will be apparent to one skilled in the art that the disclosure may be practiced without these specific details. In some cases, structures and devices are shown in block diagram form to avoid obscuring the description. For example, the embodiments may be described above primarily with reference to user interfaces and specific hardware. However, the embodiments may apply to any type of computing device capable of receiving data and instructions, and to any peripheral devices that provide services.

[0139] Reference in the specification to "some embodiments" or "some cases" means that a particular feature, structure, or characteristic described in connection with the embodiments or cases may be included in at least one embodiment of the specification. The occurrences of the phrase "in some embodiments" in various places in the specification do not necessarily all refer to the same embodiments.

[0140] Some sections of the foregoing detailed descriptions are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing field to most effectively convey the gist of their work to others skilled in the art. An algorithm, here and generally, is conceived of as an internally consistent sequence of steps that leads to a desired result. The steps are those that require physical manipulation of physical quantities. Usually, though not necessarily, these quantities are electrical or magnetic data that can be stored, transferred, combined, compared, and otherwise manipulated.It has occasionally been found convenient, mainly because it is common practice, to refer to these data as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0141] It should be noted, however, that all these and similar terms are to be assigned to the corresponding physical quantities and are merely convenient designations applied to these quantities.Unless expressly indicated otherwise in the following discussion, it is to be understood that throughout the specification, discussions using terms including "processing" or "computing" or "calculating" or "determining" or "displaying" or the like refer to the action and processes of a computer system or similar electronic computing device that manipulates and converts data represented as physical (electronic) quantities within the registers and memories of the computer system into other data similarly represented as physical quantities within the memories or registers of the computer system or other such information storage, information transfer, or information display devices.

[0142] Embodiments of the description may also relate to a processor for performing one or more steps of the methods described above. The processor may be a special-purpose processor that is selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored on a non-transitory computer-readable storage medium, including, but not limited to, any type of disk, including optical disks, ROMs, CD-ROMs, magnetic disks, RAMs, EPROMs, EEPROMs, magnetic or optical cards, flash memories including USB flash drives with non-volatile memory, or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.

[0143] The description may take the form of some embodiments implemented entirely in hardware, some embodiments implemented entirely in software, or some embodiments that include both hardware and software elements. In some embodiments, the description is implemented in software, including, but not limited to, firmware, resident software, microcode, etc.

[0144] Furthermore, the description may take the form of a computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable medium may be any device that can contain, store, communicate, distribute, or transport the program for use by or in connection with the instruction execution system, device, or apparatus.

[0145] A data processing system suitable for storing or executing program code will include at least one processor coupled directly or indirectly via a system bus to storage elements. The storage elements may include local memory used during the actual execution of the program code, mass storage, and cache memory that provides temporary storage of at least a portion of the program code to reduce the number of times code must be retrieved from mass storage during execution. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 63 / 465,230

[0001] US 63 / 562,634

[0001]

Claims

[1] Computer-implemented method comprising: Receiving a selection of an incomplete object in an output image, wherein the incomplete object is associated with a first position within the output image and an omitted portion of the incomplete object is clipped by a boundary of the output image or obscured by another object; Creating an object mask that includes pixels of an incomplete object that are associated with the incomplete object; Removing the pixels of the incomplete object associated with the incomplete object from the source image; Creating a reconstructed image that replaces the pixels of the incomplete object corresponding to the incomplete object with reconstructed pixels; Providing, as input to a diffusion model, the object mask, the incomplete object and the reconstructed image; Outputting, using the diffusion model, a complete object; and Creating a modified image by blending one or more versions of the full object with one or more versions of the reconstructed image using the object mask, wherein the full object is positioned at a second position in the modified image that is different from the first position in the original image. [2] The method of claim 1, wherein the modified image is a first modified image and further comprising: Receiving a request to remove a selected object from the first modified image; Providing the first modified image and the request as input to an object distance model associated with the diffusion model; and Output, using the object removal model, a second modified image that does not include the selected object. [3] The method of claim 1, wherein the modified image is a first modified image and further comprising: Receiving a request to move a selected object from a third position to a fourth position; Providing the first modified image and the requirement as input to an object insertion model associated with the diffusion model; and Outputting, with the object insertion model, a second modified image including the selected object at the fourth position based on the request. [4] The method of claim 1, wherein the modified image is a first modified image and further comprising: Receiving a request to add an additional object to the source image; Outputting, using the diffusion model, the additional object; and Outputting, using the diffusion model, a second modified image by blending one or more versions of the additional object with one or more versions of the reconstructed image. [5] The method of claim 4, wherein the request to add the additional object includes a text prompt describing the additional object. [6] The method of claim 1, wherein the size of the complete object is changed based on a change from the first position in the source image to the second position in the modified image. [7] The method of claim 1, further comprising: Receiving a modified image expand command to expand edges of an expanded image portion of the modified image to expanded edges; and Output a cropped image that includes reconstructed pixels between the edges of the cropped image of the modified image and the cropped edges, based on the command. [8] The method of claim 7, wherein the command for expanding the image section of the reconstructed image includes: a selection of a button to expand the image section; and either a command to directly extend the edges of the expanded crop of the modified image to the expanded edges, or a movement of the entire object that extends the edges of the expanded crop of the modified image to the expanded edges. [9] The method of claim 1, further comprising: modifying an illumination of the modified image; and Adding a shadow to the entire object based on a direction of lighting of the modified image. [10] A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations, comprising: Receiving a selection of an incomplete object in an output image, wherein the incomplete object is associated with a first position within the output image and an omitted portion of the incomplete object is clipped by a boundary of the output image or obscured by another object; Creating an object mask that includes pixels of an incomplete object that are associated with the incomplete object; Removing the pixels of the incomplete object associated with the incomplete object from the source image; Creating a reconstructed image that replaces the pixels of the incomplete object corresponding to the incomplete object with reconstructed pixels; Providing, as input to a diffusion model, the object mask, the incomplete object and the reconstructed image; Outputting, using the diffusion model, a complete object; and Creating a modified image by blending one or more versions of the full object with one or more versions of the reconstructed image using the object mask, wherein the full object is positioned at a second position in the modified image that is different from the first position in the original image. [11] The non-transitory computer-readable medium of claim 10, wherein the modified image is a first modified image, and the operations further include: Receiving a request to remove a selected object from the first modified image; Providing the first modified image and the request as input to an object distance model associated with the diffusion model; and Output, using the object removal model, a second modified image that does not include the selected object. [12] The non-transitory computer-readable medium of claim 10, wherein the modified image is a first modified image, and the operations further include: Receiving a request to move a selected object from a third position to a fourth position; Providing the first modified image and the requirement as input to an object insertion model associated with the diffusion model; and Outputting, with the object insertion model, a second modified image including the selected object at the fourth position based on the request. [13] The non-transitory computer-readable medium of claim 10, wherein the modified image is a first modified image, and the operations further include: Receiving a request to add an additional object to the source image, the request including a text prompt describing the additional object; Outputting, using the diffusion model, the additional object; and Outputting, using the diffusion model, a second modified image by blending one or more versions of the additional object with one or more versions of the reconstructed image. [14] The non-transitory computer-readable medium of claim 10, wherein the operations further include: Receiving a modified image expand command to expand edges of an expanded image portion of the modified image to expanded edges; and Output a cropped image that includes reconstructed pixels between the edges of the cropped image of the modified image and the cropped edges, based on the command. [15] System comprising: a processor; and a memory coupled to the processor having instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising: Receiving a selection of an incomplete object in an output image, wherein the incomplete object is associated with a first position within the output image and an omitted portion of the incomplete object is clipped by a boundary of the output image or obscured by another object; Creating an object mask that includes pixels of an incomplete object that are associated with the incomplete object; Removing the pixels of the incomplete object associated with the incomplete object from the source image; Generating a reconstructed image that replaces the pixels of the incomplete object corresponding to the incomplete object with reconstructed pixels; Providing, as input to a diffusion model, the object mask, the incomplete object, and the reconstructed image; Outputting, with the diffusion model, a full object; and generating a modified image by blending one or more versions of the full object with one or more versions of the reconstructed image using the object mask, wherein the full object is positioned at a second position in the modified image that is different from the first position in the output image. [16] The system of claim 15, wherein the modified image is a first modified image and the operations further include: Receiving a request to remove a selected object from the first modified image; Providing the first modified image and the request as input to an object distance model associated with the diffusion model; and Output, using the object removal model, a second modified image that does not include the selected object. [17] The system of claim 15, wherein the modified image is a first modified image and the operations further include: Receiving a request to move a selected object from a third position to a fourth position; Providing the first modified image and the requirement as input to an object insertion model associated with the diffusion model; and Outputting, with the object insertion model, a second modified image including the selected object at the fourth position based on the request. [18] The system of claim 15, wherein the modified image is a first modified image and the operations further include: Receiving a request to add an additional object to the source image, the request including a text prompt describing the additional object; Outputting, using the diffusion model, the additional object; and Outputting, using the diffusion model, a second modified image by blending one or more versions of the additional object with one or more versions of the reconstructed image. [19] The system of claim 15, wherein the size of the complete object is changed based on a change from the first position in the source image to the second position in the modified image. [20] The system of claim 15, wherein the operations further include: Receiving a modified image expand command to expand edges of an expanded image portion of the modified image to expanded edges; and Output a cropped image that includes reconstructed pixels between the edges of the cropped image of the modified image and the cropped edges, based on the command.

Images