Performing scene impact editing tasks using diffusion neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing training of a diffusion neural network for scene impact editing tasks.

Description

[0001]Attorney Docket No.56113-0700WO1 PERFORMING SCENE IMPACT EDITING TASKS USING DIFFUSION NEURAL NETWORKS CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No.63 / 562,634, filed on March 7, 2024. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application. BACKGROUND This specification relates to processing images using machine learning models. As one example, neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to another layer in the network, e.g., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of weights. SUMMARY This specification describes a system implemented as computer programs on one or more computers in one or more locations that trains a diffusion neural network to perform a scene impact editing task on input images. More specifically, the system can train the diffusion neural network to remove an object from or insert an object into a scene in an input image. The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. Diffusion models have shown success in performing various image editing tasks but often generate images that violate physical laws, particularly when handling the impact of an object on the scene. Understanding how objects interact with their surroundings, e.g., occlusions, shadows, reflections, is crucial for realistic image editing. Existing approaches for training diffusion models, e.g., self-supervised approaches, are poorly suited for this task and yield trained models that are not able to effectively model object interactions with the larger scene. As a result, these trained models frequently generate unrealistic images for tasks that require adding or removing the impact of an object in a scene. For example, shadows and Attorney Docket No.56113-0700WO1 reflections corresponding to the added / removed object may not be rendered correctly in the generated image. The techniques in this specification, on the other hand, modify the training of a diffusion model in order to allow the model to more accurately model the impact of removing an object or inserting an object into a scene. As one example, the system can train a diffusion model on a counterfactual data set to yield a diffusion model that more accurately models the impact of removing an object from or inserting an object into a scene. As another example, the system can train a diffusion model on a synthetic scene impact data set in order to yield a diffusion model that can accurately modify an input image to model the impact of an additional object on the scene. That is, by virtue of being trained using the described techniques, the diffusion model can effectively model the effect on the scene of inserting a new object at a particular location in the scene. Aspects of the disclosure can be implemented for a variety of different image-editing use cases. For example, the system can be at least partially implemented on user devices, such as smartphones, personal computers, wearable devices, and so on, for providing image- editing features to a camera application or image storage application loaded on the user devices. Features of the system can be integrated as user-facing options for editing images to remove, add, or modify objects and their corresponding scene impact to an input image. These and other features can be performed on a user device automatically or in response to user input. In some examples, a software application implementing features of the system can generate suggestions or recommendations for editing images as described herein. The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS FIG.1 shows an example training system. FIG.2 shows an example input and output of a trained scene impact editing diffusion model. FIG.3 showcases different example images that can be included in the data sets used to train the scene impact editing diffusion model. Attorney Docket No.56113-0700WO1 FIG.4 shows a training iteration of the training process for the diffusion neural network for an object removal implementation. FIG.5 is a flow diagram of an example process for training the diffusion neural network for an object removal implementation. FIG.6 is a flow diagram of sub-steps of one of the steps of the process of FIG.5. FIG.7 illustrates the difference in object removal output images between the object removal diffusion neural network and a stable diffusion inpainting model. FIG.8 shows an example process of generating a synthetic object insertion dataset to train an object insertion diffusion neural network. FIG.9 is a flow diagram of an example process for training the diffusion neural network for an object insertion implementation. FIG.10 is a flow diagram of sub-steps of one of the steps of the process of FIG.9. FIG.11 highlights the difference between the object insertion diffusion neural network and other techniques used in diffusion models for an object insertion task. Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION FIG.1 shows an example training system 100. The training system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components and techniques described below can be implemented. The system 100 trains a diffusion model 130 to perform a scene impact editing task on an input image 102 to generate an output image 152. Generally, performing “scene impact editing” on an input image 102 that depicts a scene refers to editing the image 102 to modify the impact of an object on the scene. As one example, scene impact editing can refer to removing an object and any associated impacts on the scene from the input image 102. As another example, scene impact editing can refer to modifying the image 102 to integrate a new object and any associated impacts into the scene. The “associated impact” of an object being removed from or added to a scene can include occlusions, shadows, and reflections that are modified within the scene by virtue of the removal or addition of an object. Thus, properly accounting for the associated impact of an object removal or addition to an image can require modifying areas of the image that are outside the pixels that contain the object within the image. Attorney Docket No.56113-0700WO1 When performing an object removal task, the scene impact editing diffusion model 130 can be interchangeably referred to as an object removal neural network. When performing an object insertion task, the scene impact editing diffusion model 130 can be interchangeably referred to as an object insertion neural network. Generally, the diffusion model 130 being trained is configured to receive an input that includes (i) a representation of an input image, (ii) a noisy representation of an output image, and (iii) context data characterizing a region of the input image to be edited and to process the input to generate a denoising output that defines an update to the noisy representation of the output image. For example, the denoising output can be an estimate of the noise component of the noisy representation, i.e., of the noise that has been added to a final representation of the output image to generate the noisy representation. As another example, the denoising output can be an estimate of the final representation of the output image. After training, the scene impact editing diffusion model 130 can be used to perform a scene impact editing task by gradually "de-noising" the noisy representation across multiple reverse diffusion steps. For example, at each reverse diffusion step, the system 100 can update the noisy representation using the denoising output. For example, at each iteration other than the last, the system 100 can generate an estimate of the representation using the denoising output and then apply a diffusion sampler to the estimate. For the last iteration, the estimate can be the updated representation, or the system 100 can use the sampler. After the last reverse diffusion step, the system 100 generates the output image 152 from the final representation of the output image. In any of the above examples, the output image 152 generated using the diffusion neural network can either be an output image 152 in the output space, i.e., so that the values in the output image are the values of image pixels or an output image in a latent space, i.e., so that the values in the output image are values in a latent representation of an output image 152 in the output space. When the output image is generated in a latent space, the system 100 can generate the output image 152 in output space by processing the output image in the latent space using a decoder neural network, e.g., one that has been pre-trained in an auto-encoder framework. During training, the system 100 can use an encoder neural network, e.g., one that has been pre-trained jointly with the decoder in the auto-encoder framework, to encode target images Attorney Docket No.56113-0700WO1 in the output space to generate target outputs for the scene impact diffusion model 130 in the latent space. The scene impact diffusion model 130 can be any appropriate diffusion neural network that is configured to receive an input that includes a current (noisy) representation of an image and a conditioning input and to generate a denoising output. As described above, in some implementations, the scene impact diffusion model 130 performs a diffusion process in output space, e.g., pixel space when the data items are images. In this example, when the data items are images, the data items, or, “representations” of the data items, operated on and generated by the diffusion neural network have values for each pixel that specify color values, e.g., RGB values or another color encoding scheme. Examples of such diffusion neural networks include Imagen. In some other implementations, the scene impact diffusion model 130 performs a diffusion process in latent space, e.g., in a latent space that is lower-dimensional than the output space. That is, the data items, or representations of the data items, operated on by the diffusion neural network are latent representations and the values in the representations are learned, latent values, e.g., rather than color values when the data items are images. Examples of such diffusion neural networks include Stable Diffusion. In some implementations, the scene impact diffusion model 130 can be a convolutional neural network, e.g., a U-Net or other architecture that maps one input of a given dimensionality to an output of the same dimensionality. As another example, the scene impact diffusion model 130 can be a transformer neural network that processes the diffusion input through a set of self-attention layers to generate the denoising output. As yet another example, the scene impact diffusion model 130 can include both convolutional layers and self-attention layers. The neural network can be conditioned on the conditioning input in any of a variety of ways. As one example, the system 100 can use an encoder neural network to generate one or more embeddings that represent the conditioning input and the scene impact diffusion model 130 can include one or more cross-attention layers that each cross-attend into the one or more embeddings. An embedding, as used in this specification, is an ordered collection of numerical values, e.g., a vector of floating point values or other types of values. Attorney Docket No.56113-0700WO1 For example, when the conditioning input is text, the system 100 can use a text encoder neural network, e.g., a Transformer neural network, to generate a fixed or variable number of text embeddings that represent the conditioning input. When the conditioning input is an image, the system 100 can use an image encoder neural network, e.g., a convolutional neural network or a vision transformer neural network, to generate a set of embeddings that represent the image. In some of these cases, the system 100 can generate one or more initial embeddings for each of the different types of inputs, i.e., using an appropriate encoder neural network as described above, and then process the initial embeddings for all of the different types of inputs using a transformer encoder neural network to update each of the initial embeddings to generate a set of final embeddings. The one or more cross-attention layers within the scene impact diffusion model 130 can then cross-attend into the set of final embeddings. In others of these cases, different cross-attention layers within the scene impact diffusion model 130 can cross-attend into embeddings of different types of conditioning inputs. In yet others of these cases, the system 100 can concatenate the initial embeddings of the different types of inputs along the sequence dimension and then the one or more cross- attention layers can cross-attend into the concatenated set of final embeddings. As another example, the scene impact diffusion model 130 can include one or more other types of neural network layers that are conditioned on the one or more embeddings. Examples of such layers include Feature-wise Linear Modulation (FiLM) layers, layers with conditional gated activation functions, and so on. As another example, the output(s) of the encoder(s) when encoding one or more of the conditioning inputs can be combined, e.g., through a weighted sum, with features of the representation of the output image, and the combined features can be processed by the remainder of the scene impact editing diffusion model 130. The diffusion input at any given updating iteration can also include data defining a noise level for the iteration. Generally, each updating iteration has a corresponding time step t and the noise level for the iteration depends on the time step. For example, the noise level can be a decreasing function of the time step t. Examples of such functions include a linear function, a cosine function, and a sigmoid function. In these cases, data identifying the noise level, the time step, or both can be embedded using an appropriate neural network, e.g., a multi-layer perceptron (MLP) and used to condition the scene impact diffusion model 130 as described above for the conditioning input. Attorney Docket No.56113-0700WO1 To perform the reverse diffusion process to generate an output image from a given conditioning input, the system 100 initializes a representation of the output image. For example, the system 100 can sample each value in each representation from a noise distribution, e.g., a Gaussian distribution. The system 100 then updates the representation at each of a plurality of reverse diffusion steps (also referred to as “iterations” or “updating iterations”) using the conditional diffusion neural network. Each reverse diffusion step is associated with a noise level for the iteration. Generally, each updating iteration has a corresponding time step t and the noise level for the iteration depends on the time step. For example, the noise level can be a decreasing function of the time step t. Examples of such functions include a linear function, a cosine function, and a sigmoid function. Thus, early iterations are associated with higher noise levels and later iterations are associated with lower noise levels, resulting in the diffusion neural network gradually “denoising” the representation to generate the final representation. As part of the updating at any given step, the system generates a denoising output for the reverse diffusion step. The system 100 then updates the representation of the output image using the denoising output for the reverse diffusion step. For example, the system 100 can map the denoising output to an initial updated representation and then apply a diffusion sampler, e.g., the DDPM (Denoising Diffusion Probabilistic Model) sampler, the DDIM (Denoising Diffusion Implicit Model) sampler or another appropriate sampler, to the initial updated representation to generate an updated representation. Optionally, after the last reverse diffusion iteration, the system 100 can refrain from using the diffusion sampler and can instead use the initial updated representation as the updated representation. To generate the denoising output, the system 100 processes a first denoising input for the reverse diffusion step that includes the representation of the output image and the conditioning input using the denoising neural network to generate a first denoising output. In some cases, the first denoising output is the denoising output. In some other cases, the system 100 also generates one or more additional denoising outputs and then combines the additional denoising output(s) with the first denoising output through classifier free guidance, i.e., by computing a weighted sum of the denoising outputs with the weight for each denoising output being determined by a guidance weight for the classifier free guidance. Attorney Docket No.56113-0700WO1 After updating the representation at each of the plurality of reverse diffusion steps, the system 100 generates the output image from the representation of the output image. For example, when the diffusion neural network is in the latent space, the system 100 can process the representation using the decoder neural network to generate the output image. When the diffusion neural network is in the output space, the system 100 can use the final representation as the output image. In some implementations, the scene impact editing scene impact diffusion model 130 is fine-tuned 115 from a pre-trained diffusion neural network 110. In some cases, the pre- trained diffusion neural network 110 has been pre-trained on one or more image editing tasks. For example, the one or more image editing tasks can include an image in-painting task. FIG.2 shows an example input and output of the example trained scene impact editing diffusion model 130 of FIG.1. After the scene impact editing diffusion model 130 is trained, the model 130 can receive a new input image and new context data for the new input image. When performing an object insertion task, the diffusion model 130, e.g., an object insertion diffusion neural network, can also receive an object image of a new object to be inserted into the new input image. The model 130 can then generate a modified image by performing the scene impact editing task, e.g., object insertion or object removal, specified by the input data using the trained scene impact diffusion model 130. The scene impact editing task, e.g., whether the model will be performing an object insertion or removal task, can be specified by the new context data of the new input image in both implementations. For the object insertion implementation, the task can also be specified by the object image, e.g., the specific object to be inserted can be the object that is depicted in the object image and the model 130 can operate as if the task were to insert the object depicted in the object image into the modified image. For example, in an object removal implementation as depicted in FIG.2, the scene impact editing diffusion model 130, can receive an input image 202 of a little boy on a rainy road. The model 130 can also receive the following context 207 as input: “a photo of an empty road” that specifies the object removal task. The scene impact editing diffusion model 130 can use the context 207 that specifies the scene impact editing task, e.g., removing an object (the little boy) from the scene, and the input image 202, to generate an output image 252 that is a photo of an empty road. In the output image 252, the object, e.g., the little boy, has been removed from the scene as well as Attorney Docket No.56113-0700WO1 any shadows and reflections from the little boy, while any occlusions have been filled by the model 130, e.g., the body of the mailbox that was previously covered by the little boy. This particular example illustrates the robustness of the trained model 130 as the rainy nature of the scene can make removing certain reflections and occlusions particularly difficult. FIG.3 shows example images included in datasets that can be used to train the scene impact editing diffusion model for different scene impact editing tasks. In an object removal task, the scene impact editing diffusion model can be trained on a counterfactual data set that can include image C 331, e.g. an image with an object, and image A 311, e.g., an image without an object. The counterfactual training data set is described in further detail below with reference to FIG.4. In an object insertion task, the scene impact editing diffusion model can be trained on a synthetic object insertion data set that includes image B 321, e.g., an image with an object in the scene, and image C 331, e.g., an image with object and the reflection and shadows of the object in the scene. The synthetic object insertion data set is described in further detail with reference to FIG.8. FIG.4 shows a training iteration of the training system 400 for an object removal diffusion neural network 420. To train a diffusion neural network 420 to perform an object removal task, the system 400 can obtain a counterfactual training data set 441 that includes one or more training examples, e.g., training example 451. Each training example can include (i) a factual image 443 that depicts a respective scene that includes one or more objects, a counterfactual image 445 that depicts the respective scene with a particular object of the one or more objects removed, and context data 447 that specifies the particular object that was removed from the respective scene. For example, the factual image 443 can be image C 331 of FIG.3 that depicts a basketball on a table while the counterfactual image 445 can be image A 311 of FIG.3 that depicts the table without the basketball on it. The context data 447, not depicted in FIG.3, can then specify that the basketball was removed from the scene or can be an object mask corresponding to the basketball in image C 331 of FIG.3. As another example, the factual image 443 can depict a baseball in a baseball glove while the counterfactual image 445 can depict an empty baseball glove. The context data 447 can then specify that the baseball was removed from the scene or can be an object mask corresponding to the baseball in the factual image 443. Attorney Docket No.56113-0700WO1 In some implementations, some or all of the images in the counterfactual dataset 441 can include photographs of a real-world scene. The factual image 443 can be an image of a respective real-world scene taken by a camera with a particular camera pose. The counterfactual image 445 is an image of the same real-world scene taken by the camera with the same camera pose with the particular object removed from the real-world scene. The aim of the dataset 441 is to provide the diffusion model 420 with a causal understanding of object removal, including the effect on shadows and reflections. In some implementations, the context data 447 can be an object. The object mask can be obtained using any appropriate method. For example, the context data 447 can be generated by utilizing a segmentation model, i.e., by processing the factual image 443 using a trained segmentation model to generate a segmentation mask over the factual image 443. In some implementations, the object / segmentation mask can be a binary mask (e.g., 0 for environment, 1 for object for each pixel of the factual image 443), or a ratio mask (values between 0 and 1 indicating the likelihood of a pixel being part of the object). The system 400 can train / fine-tune the diffusion neural network 420 on the one or more training examples, e.g., training example 451, of the counterfactual training data set 441 to perform an object removal task, e.g., on a score matching objective or other appropriate diffusion model training objectives. To begin an iteration of the training process, the training system 400 can sample one or more of the training examples, e.g., training example 451, of the counterfactual training data set 441. The below steps can be implemented for each of the one or more sampled training examples, e.g., training example 451. The object removal diffusion neural network 420 can receive the training example 451 and can generate a noisy representation of the counterfactual image 455 in the training example 451. For example, noise can be added to an initial representation of the counterfactual image 445. To generate the noisy representation of the counterfactual image 455, the training system 400 can randomly sample noise, e.g., Gaussian noise, to be added to the counterfactual image 445. The training system 400 can randomly sample a noise level and combine the initial representation of the counterfactual image 445 and the noise in accordance with the noise level. The initial representation of the counterfactual image can be generated using an encoder when the diffusion process occurs in the latent space, or the counterfactual image Attorney Docket No.56113-0700WO1 445 can be used as the initial representation when the diffusion process occurs in the pixel space. The diffusion neural network 420 can then process a diffusion input that includes (i) the noisy representation of the counterfactual image 455, (ii) a representation of the factual image 443, and (iii) a representation of the context data 447 to generate a denoising output 457 that defines an update to the noisy representation of the counterfactual image 455. For example, the denoising output 457 can be an estimate of the noise component of the noisy representation of the counterfactual image 455. The training system 400 can train the diffusion neural network 420 on an objective that measures an error 462 between, the denoising output 457, e.g. an estimate of the noise, and the actual noise added to the initial representation of the counterfactual image 455. At a high level, the diffusion model 420 can estimate the noise in the noisy representation of the counterfactual image 455 to help learn and differentiate what constitutes the actual environment, e.g., scene, versus the noise. This can help the model 420 seamlessly fill occlusions caused by the object. The error 462 can be any appropriate error measurement or metric. For example, the model 420 can calculate a mean-squared error estimate between the denoising output 457 and the noisy representation of the counterfactual image 455 to train the neural network 420 by evaluating how well the neural network 420 is able to estimate noise. FIG.5 is a flow diagram of an example process 500 for training the diffusion neural network on an object removal task. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 400 depicted in FIG 4, appropriately programmed in accordance with this specification, can perform the process 500. The system can obtain a counterfactual training data set that includes one or more training examples (step 502). Each of the one or more training examples can include (i) a factual image that depicts a respective scene that includes one or more objects, (ii) a counterfactual image that depicts the respective scene with a particular one of the one or more objects removed from the respective scene, and (iii) context data specifying the particular object that was removed from the respective scene. As described above, in some implementations, the images for the counterfactual training data set can be manually obtained. For example, the factual image in the training example can be an image of a respective real world scene taken by a camera with a particular Attorney Docket No.56113-0700WO1 camera pose before the particular object is removed from the real world scene and the counterfactual image is an image of the respective real world scene taken by the camera with the particular camera pose after the particular object has been removed from the real world scene. In some implementations, the context data can be an object mask specifying a region of the factual image that depicts the particular object. For example, the context data can be a mask generated using a segmentation model. The system can train a diffusion neural network on the counterfactual training data set to perform an object removal task (step 504). For example, the diffusion neural network can be trained to, for each training example, generate the counterfactual image in the training example conditioned on the factual image in the training example and the context data in the training example. The training process of the diffusion neural network is further detailed with reference to FIG.6. FIG.6 is a flow diagram of example sub-steps of step 504 of FIG.5 that further details a training process 600of the diffusion neural network for an object removal task. For convenience, the process 600 of FIG.6 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 400 depicted in FIG 4 appropriately programmed in accordance with this specification, can perform the process 600. The system can sample one or more of the training examples (step 602). The one or more training examples can be sampled using any appropriate method, including bootstrapping, random sampling, stratified sampling, mini-batch sampling, and sequential sampling. The below steps (604-606) can be implemented for each of the one or more training examples. The system can generate a noisy representation of the counterfactual image in the training example by adding noise to an initial representation of the counterfactual image (step 604). The system can add noise to the initial representation of the counterfactual image using any appropriate technique. In some implementations, to generate the noisy representation, the system can randomly sample noise, e.g., Gaussian noise, to be added to the counterfactual image. The system can randomly sample a noise level and combine the initial representation of the counterfactual image and the noise in accordance with the noise level. Attorney Docket No.56113-0700WO1 The system can then process a diffusion input that includes (i) the noisy representation of the counterfactual image, (ii) a representation of the factual image, and (iii) a representation of the context data using the diffusion neural network to generate a denoising output (step 606). The denoising output can define an update to the noisy representation of the counterfactual image. For example, in some implementations, the denoising output can be an estimate of the noise component of the noisy representation, e.g., the noise that has been added to a final representation of the counterfactual image to generate the noisy representation. In other implementations, the denoising output can be an estimate of the final representation of the counterfactual image. The system can train the diffusion neural network on an objective that measures an error between, for each training example, the denoising output and the noise added to the initial representation of the counterfactual image (step 608). That is, the error can be calculated between the estimate of the noise of the counterfactual image and the actual noise added to the counterfactual image to better differentiate between noise and the environment, e.g., scene, of the image. In object removal tasks, the model needs to understand the context provided by the surrounding scene, so by more accurately estimating the noise, the model can better reconstruct the areas where the object has been removed and blend those areas naturally with the rest of the image. The error can be any appropriate error measurement or metric. For example, the diffusion neural network can calculate a mean-squared error estimate between the denoising output and the noisy representation of the counterfactual image to train the neural network. After the error is calculated, the system can then use the error value to update any parameters of the diffusion neural network, e.g., fine-tune the diffusion neural network. FIG.7 illustrates the difference in object removal output images between the fine- tuned object removal diffusion neural network 730 and a stable diffusion inpainting model 760. As demonstrated in the figure, for each example, both models receive an input image and a mask of the object which is to be removed. For example, in example C 731, the input image depicts a photo of a drink on an outdoor table while the input mask outlines the drink. The resulting output images illustrate the difference between the two models. The object removal diffusion neural network 730, i.e., the model described in this disclosure, can successfully remove the object from the input image as well as account for any reflections, shadows and occlusions as depicted in the output images. By comparison, the stable diffusion Attorney Docket No.56113-0700WO1 inpainting model 760 does not remove the object successfully and / or does not account for its results. In some cases, associated impacts, e.g., occlusions, shadows and reflections, and actually replaces the object is replaced with a different one. For example, in example C 731, the drink on the table is replaced with a jar. Because the inpainting model 760 depends heavily on the input mask, the shadows and reflections of the object which are not part of the input mask, are considered as part of the scene by the inpainting model 760. The inpainting model 760 then replaces the object with another object consistent with the leftover shadows and reflections and considering the scene. If the input mask is chosen to include the shadows and reflections of the object as well as the object, then too much of the image is specified to the inpainting model 760 and the original image and scene cannot be preserved by the inpainting model 760. The success of the object removal diffusion neural network 730 is as a result of fine- tuning the model on the counterfactual training data set. Through fine-tuning, the object removal diffusion neural network 730 can learn and understand the associated impacts of the object, e.g., the occlusions, reflections, and shadows, by seeing the environment when the object is present and not present. The model can then better identify and remove these associated impacts when removing the object, leading to the improved output images seen for each of the examples. FIG.8 shows an example process 800 of generating a synthetic object insertion data set to train an object insertion diffusion neural network. For convenience, the process 800 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 100 depicted in FIG 1, appropriately programmed in accordance with this specification, can perform the process 800. To train a diffusion neural network to perform an object insertion task, the system can obtain an initial data set 871 that includes a set of initial images that each depict a respective scene. For example, the initial data set 871 can contain one or more initial images, including initial image 873 as seen in FIG.8. The initial data set 871 can be any appropriate data set of images. For example, the initial data set 871 can be a large-scale data set of images obtained from the Internet. Each of the initial images can depict a different respective scene. For example, initial image 873 can depict a driveway with a trashcan and two cars while a different initial image, e.g., image C 331 of FIG.3, in the initial data set 871 can depict a table with a basketball, a placemat and a jar or cookies on it. Attorney Docket No.56113-0700WO1 The system can then generate, from the initial training data set 871, a synthetic object insertion data set 891 to be used to train 823 an object insertion diffusion neural network 825 to insert an object into a scene. To generate the synthetic data set 891, the system can, for each initial image, e.g., initial image 873, identify one or more objects, e.g., identified object 883, identified object 885 and identified object 887, that are in the respective scene depicted in the initial image. For example, the system can identify the trashcan and both of the two cars in the initial image 873. As another example, the system can identify both the basketball, the place mat, and the jar of cookies in the image C 331 of FIG.3 that depicts a table with the previously described objects on it. For one or more of the identified objects, e.g., identified object 883, the system can generate, using an object removal neural network 820, e.g., the trained object removal neural network of FIG.4, a training example 898 for the synthetic object insertion data set 891 that includes (i) the initial image 873, (ii) a blended image 896, and (iii) context data 898 specifying the region of the initial image that depicts the identified object. For example, the system can generate a training example 895 for each of the one or more identified objects in an initial image, e.g., initial image 873, for each of the initial images in the initial data set. For example, in the set of initial images that includes an initial image 873 that depicts a driveway with a trashcan and two cars and image C 331 of FIG.3, the synthetic object insertion data set 891 can include six training examples, three for each image: one generated for the trashcan, one generated for the first car, and one generated for the second car for initial image 873 and one generated for the basketball, one generated for the placemat and one generated for the cookie jar in image C 331 of FIG.3. To generate the blended image 896, the object removal diffusion neural network can generate a modified image that has the identified object removed from the respective scene depicted in the initial image (and retaining any other objects not identified for removal in the respective scene). In this regard, the modified image can be considered to be “object-less” in that the modified image does not include the identified object. For example, image A 311 of FIG.3 can depict a modified, or object-less image, that has removed the basketball from image A 311, but has kept other remaining objects in the image, e.g., the placemat and the jar of cookies. As another example, for the trashcan, e.g., identified object 883 in the driveway, the object-less image 894 can be an image that has the trashcan removed from the respective scene in the initial image 873. The modified image will be referred to as the “object-less image” from herein. Attorney Docket No.56113-0700WO1 The object removal neural network 820 can be the, e.g., fine-tuned, object removal neural network of FIG.4 that has been trained on the counterfactual training data set to perform the object removal task. By using the object removal diffusion neural network, the identified object can be removed together with corresponding shadows and reflections and taking account of any other impacts the object has on the scene. The blended image 896 can then be generated from the object-less image 894 for the identified object and the region of the initial image 873 that depicts the identified object by combining (i) a masked image that masks out a region of the object-less image for the identified object that corresponds to the region of the initial image that depicts the identified object and (ii) the region of the initial image that depicts the identified object. That is, the blended image 896 combines the region of the object-less image 894 that corresponds to the region containing the object in the initial image 873, but that does not have any of the associated impacts of the object, and the mask of the identified object. This has the effect of pasting the object back into the object-less image but without any of the corresponding shadows, reflections and other impacts from the inclusion of the object in the scene. Thus, the blended image depicts the identified object in the region that it appears in the initial image 873 without any of the reflections, occlusions, or shadows etc. For example, image B 321 of FIG.3 can depict a blended image. As seen in FIG.3, the image B 321 can include the basketball back on the table but does not include any of the corresponding shadows or reflections of the basketball on the table, e.g., the reflection of the basketball on the glass table. The context data 898 can specify the region of the initial image that depicts the identified object, e.g., identified object 883. In some implementations, the context data is an object that specifies the region of the initial image that depicts the identified object. For example, the training example for the trashcan, e.g., identified object 883, can include the initial image of the driveway with the trashcan and two cars, the blended image of the driveway with the two cars and the added trashcan without any of the associated impacts, e.g., reflections, occlusions, shadows, and an object mask of the trashcan. The system can then train an object insertion diffusion neural network 825 on the synthetic object insertion data set 891, e.g., on the training example 898 and one or more other training examples in the synthetic object insertion data set, to perform an object insertion task. That is, the system can train the neural network 825 to generate the initial image 873, e.g., the image included the identified object, from the blended image 896 and the context data. Attorney Docket No.56113-0700WO1 In some implementations, the system can train the neural network 825 to generate the initial image 873, e.g., the image including the identified object, from the object-less image 894 and the context data. That is, the system can train the neural network 825 to generate the initial image 873 without generating the blended image 896 and using the object-less image 894 instead. The training process of the object insertion diffusion neural network 825 is described in further detail with reference to FIG.9 and FIG.10. In some implementations, the object insertion diffusion neural network 825 and the object removal diffusion neural network 820 are two separate models. That is, the system can include two models: an object removal diffusion neural network 820 that can be trained to perform an object removal task and an object insertion diffusion neural network 825 that can be trained to perform an object insertion task. In some implementations, the object insertion diffusion neural network 825 is the object removal diffusion neural network 820 further trained to perform an object insertion task as well as the object removal task. That is, the system can include a singular model that can be trained to perform both an object removal task and object insertion task. FIG.9 is a flow diagram of an example process 900 for training an object insertion diffusion neural network on an object insertion task. For convenience, the process 900 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 100 depicted in FIG 1, appropriately programmed in accordance with this specification, can perform the process 900. The system can obtain an initial dataset that includes a set of initial images that each depict a respective scene (step 902). The initial data set can include one or more images that can be filtered to identify the set of initial images used in the training process. The initial data set can be filtered based on one or more criteria, including, removing redundant images, images that do not have shadows, images with detected objects that are too small, images with detected objects that are too big, and any other relevant criteria that would make the image unsuitable or unideal for training. The resulting set of initial images can include one or more images that each depict a respective scene. The system can then generate, from the initial data set, a synthetic object insertion data set (step 904). The synthetic object insertion data set can then be used to train an object Attorney Docket No.56113-0700WO1 insertion diffusion neural network. The synthetic object insertion data set can include an initial image of the set of initial images, a blended image, and context data, which are described in more detail above with reference to FIG.8. The below steps (906-912) can be performed for each initial image in the initial data set to generate the synthetic object insertion data set. The system can identify one or more objects that are in the respective scene depicted in the initial image (step 906). The one or more objects can be identified by using any appropriate method. For example, in some implementations, the system can identify the one or more objects in the respective scene by processing the initial image using a trained detector neural network to generate a detection output that identifies the one or more objects. As a specific example, a foreground detector neural network can be used to detect a set of objects in the image and the system can select one or more of the detected objects, e.g., by selecting all of the objects, by selecting a certain proportion of the objects randomly, and so on. Then, for each of one or more of the identified objects, the system can generate the following: an object-less image, a blended image, and a training example to include in the synthetic object insertion data set. The system can generate, using an object removal neural network, the object-less image that has the identified object removed from the respective scene depicted in the initial image (step 908). The object removal neural network can be the trained object removal neural network 420 of FIG.4 that can remove the object and account for its associated impacts, e.g., reflections, occlusions, and shadows, from the initial image. The object removal neural network can be considered to model the distribution ^^൫^^ௗ^^,థ^ห^^,^^^^, where ^^^,^^^^ represents a dataset of input pairs of a source image mask of the object to be removed ^^^^^, and ^^ௗ^^,థ^represents the counterfactual image that has removed the object and its associated impacts, e.g., occlusions, reflections and shadows. In this example, The object removal neural network, ^^൫^^ௗ^^,థ^ห^^,^^^^, can use the input pairs of images in the set of initial images, ^^^,^^ଶ, … ^^^,, and the corresponding objectmasks, ^^^,^^ଶ,…^^^,, to specify the object to be removed from the respective initial images, e.g., the ^^^,mask is a mask of the object to be removed from ^^^,. The object-less images can be denoted as, ^^ ^^ … ௗ^^,థ^^, ଶ, ^^^,, where ^^^ ~ ^^^^^ |^^^ ,^^^^. Attorney Docket No.56113-0700WO1 The system can then generate the blended image from the object-less image for the identified object and the region of the initial image that depicts the identified object (step 910). The blended image can be generated by combining (i) a masked image that masks out a region of the object-less image for the identified object that corresponds to the region of the initial image that depicts the identified object and (ii) the region of the initial image that depicts the identified object, as seen in the below equation. ^^^ ൌ ^^^ ⊙ ^^^ ^ ^1 െ ^^^^ ⊙ ^^^The identified object image, resulting in an image that contains the object and the background, e.g., scene, without the associated impacts of the object, e.g., reflections, occlusions, and shadows. From the blended image, the system can generate the training example for the synthetic object insertion data set that includes (i) the initial image, (ii) the blended image, and (iii) context data specifying the region of the initial image that depicts the identified object (step 912). The system can train a diffusion neural network, e.g., an object insertion neural network, on the synthetic object insertion data set to perform an object insertion task (step 914). The object insertion diffusion neural network can be trained to, for each training example, generate the initial image in the training example, e.g., the image containing the object, conditioned on the blended image and the context data in the training example. The optimization objective of the diffusion model can be given by the equation below, where ^^^is the diffusion model that is conditioned on images (^^^) and masks (^^^), α௧and σ^are the noising schedule, ^^ is the time step and ^^ is the text prompt, which remains fixed for all images. ℒ^θ^ ൌ ^^ே || ^ ^ | ଶ௧∈ ்,^∈ ே^^,ூ^ ^^ୀ^^^^ α௧^^^ ^ σ^^,^^^ ,^^^ , ^^,^^ െ ^ | The training process will be described in further detail with reference to FIG.9. After training the diffusion neural network on the synthetic object insertion data set, the diffusion neural network can be further trained, e.g., fine-tuned, on the counterfactual training data set, e.g., the counterfactual training data set 441 of FIG.4. That is, at a high level, the object insertion diffusion neural network is pre-trained on the synthetic object Attorney Docket No.56113-0700WO1 insertion data set and then fine-tuned on the smaller, more realistic, ground-truth counterfactual dataset. In some cases, adding an object and its associated impacts on a scene is a more complex task than removing an object and its corresponding impacts. As such, training the object insertion neural network may require more data than training an object removal neural network. However, collecting real-world counterfactual data can be an expensive and time-consuming process. As such, in practice, the size of the counterfactual dataset could be too limited to train the object insertion neural network. The above-described method of generating further synthetic training data enables the effective training of the objection insertion neural network. As described above, the object removal neural network trained on the counterfactual dataset can be leveraged to generate the synthetic training dataset. FIG.10 is a flow diagram of step 914 of the process of FIG.9 and further details the training process of the diffusion neural network on an object insertion task. For convenience, the process 1000 of FIG.10 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 100 depicted in FIG.1, appropriately programmed in accordance with this specification, can perform the process 1000. The system can sample one or more of the training examples (step 1002). The one or more training examples can be sampled using any appropriate method, including random sampling, stratified sampling, mini-batch sampling, and sequential sampling. The below steps (1004-1006) can be performed for each of the one or more training examples. The system can generate a noisy representation of the initial image in the training example by adding noise to an initial representation of the initial image (step 1004). The system can add noise to the initial representation of the counterfactual image using any appropriate technique, which is further described above with reference to FIG.4. The system can process a diffusion input that includes (i) the noisy representation of the initial image, (ii) a representation of the blended image, and (iii) a representation of the context data using the diffusion neural network to generate a denoising output (step 1006). The denoising output can define an update to the noisy representation of the initial image. For example, in some implementations, the denoising output can be an estimate of the noise component of the noisy representation, e.g., the noise that has been added to a final representation of the initial image to generate the noisy representation. In other Attorney Docket No.56113-0700WO1 implementations, the denoising output can be an estimate of the final representation of the initial image. The system can then train the diffusion neural network on an objective that measures an error between, for each training example, the denoising output and the noise added to the initial representation of the initial image (step 1008). That is, the error can be calculated between the estimate of the noise of the initial image and the actual noise added to the initial image to better differentiate between noise and the environment, e.g., scene, of the image. In object insertion tasks, the model needs to understand the context provided by the object, e.g., the reflections and shadows caused by the object, so by more accurately estimating the noise, the model can better reconstruct the reflections and shadows where the object has been inserted. The error can be any appropriate error measurement or metric. For example, the diffusion neural network can calculate a mean-squared error estimate between the denoising output and the noisy representation of the initial image to train the neural network. FIG.11 highlights the difference between the object insertion diffusion neural network and other techniques used in diffusion models for an object insertion task. The figure compares the output images of three diffusion models: a diffusion model 1127, a diffusion model 1129, and the object insertion diffusion neural network 1125 described in this disclosure. The diffusion model 1127 can involve exemplar-based image editing, where a reference image (example image) is used to guide the editing process. Examples of such diffusion neural networks include Paint-by Example. The diffusion model 1129 can use a diffusion U-Net architecture and feature extraction to guide the object insertion process. Examples of such diffusion neural networks include AnyDoor. As demonstrated in the figure, for each example, the input to the diffusion model 1127 and the diffusion model 1129 is a background image of the scene and an object image of the object to be inserted into the background. For example, in example A 1111, the background image depicts a snowy mountain scene while the object image depicts a stuffed bear wearing a blue winter hat and scarf. The input to the object insertion diffusion neural network 1196 is an image of the object inserted into the background, but without any of the associated impacts of the object, e.g., reflections and shadows, or a blended image, e.g., the blended image of FIG.8. For example, in example A 1111, the input is a blended image of the stuffed bear inserted into the snow mountain scene but without any of the reflections. Attorney Docket No.56113-0700WO1 The resulting output images illustrate the difference between the models. The output image of the object insertion diffusion neural network 1125, i.e., the model described in this disclosure, depicts the (actual) object inserted into the scene as well as any reflections and shadows. For example, for example A 1111, the output image is a bear with a hat and scarf, e.g., the same bear specified in the object image, in a snow mountain scene with a shadow under the bear’s legs. The shadows and reflections of the objects can be seen in each of the output images of the object insertion neural network 1125 for each of the examples. Both of the output images generated by the other diffusion models, e.g., the diffusion model 1127 and the diffusion model 1129, do not accurately preserve the identity of the inserted object and insert an object that is semantically similar, but not the same. For example, in example B 1131, the bear inserted into the background of the bunny is not the same bear in the object image. As opposed to the object insertion diffusion neural network 1125 that is trained on images of the object, images of the object in the scene, and object-less images of the scene, the diffusion model 1129 is only trained to characterize the target object using identify features and uses the characterized features and learned scene context to generate the output image. As demonstrated in FIG.11, this can lead to an object that is semantically similar but not the same object that is specified in the object image. Similarly, diffusion model 1127, while trained on an example image of the particular object, focuses on the overall features and style of the example image and can lead to variations of the object. While the diffusion models 1127 and 1129 can insert objects into a scene with appropriate associated impacts, as demonstrated in the output images, it is the object insertion diffusion neural network 1125 that most accurately inserts the object and its associated impacts into a scene. This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied Attorney Docket No.56113-0700WO1 computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively, or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network. Attorney Docket No.56113-0700WO1 In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently. Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers. The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers. Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few. Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. Attorney Docket No.56113-0700WO1 To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return. Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads. Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a Jax framework. Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet. The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some Attorney Docket No.56113-0700WO1 embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device. While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. Similarly, while operations are corresponded to in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. The specification also provides the subject-matter of the following clauses: Clause 1. A method performed by one or more computers, the method comprising: obtaining a counterfactual training data set that comprises a plurality of training examples, each of the plurality of training examples comprising: a factual image that depicts a respective scene that includes one or more objects; a counterfactual image that depicts the respective scene with a particular one of the one or more objects removed from the respective scene; and Attorney Docket No.56113-0700WO1 context data specifying the particular object that was removed from the respective scene; and training a diffusion neural network on the counterfactual training data set to perform an object removal task. Clause 2. The method of clause 1, wherein the diffusion neural network has been pre-trained on one or more image editing tasks. Clause 3. The method of clause 2, wherein the one or more image editing tasks comprise an image in-painting task. Clause 4. The method of any one of clauses 1-3, wherein the context data is an object mask specifying a region of the factual image that depicts the particular object. Clause 5. The method of any preceding clause, wherein training the diffusion neural network on the counterfactual data set to perform the object removal task comprises training the diffusion neural network to, for each training example, generate the counterfactual image in the training example conditioned on the factual image in the training example and the context data in the training example. Clause 6. The method of any preceding clause, wherein training the diffusion neural network comprises: sampling one or more of the training examples; for each of the one or more training examples: generating a noisy representation of the counterfactual image in the training example by adding noise to an initial representation of the counterfactual image; and processing a diffusion input that comprises (i) the noisy representation of the counterfactual image, (ii) a representation of the factual image, and (iii) a representation of the context data using the diffusion neural network to generate a denoising output; training the diffusion neural network on an objective that measures an error between, for each training example, the denoising output and the noise added to the initial representation of the counterfactual image. Clause 7. The method of any preceding clause, further comprising, after the training: receiving a new input image and new context data for the new input image; and Attorney Docket No.56113-0700WO1 generating a modified image by performing an object removal task specified by the new context data on the new input image using the trained diffusion neural network. Clause 8. The method of any preceding clause, wherein, for each training example: the factual image in the training example is an image of a respective real- world scene taken by a camera with a particular camera pose before the particular object is removed from the real-world scene; and the counterfactual image in the training example is an image of the respective real-world scene taken by the camera with the particular camera pose after the particular object has been removed from the real-world scene. Clause 9. A method performed by one or more computers, the method comprising: obtaining an initial data set that comprises a set of initial images that each depict a respective scene; generating, from the initial data set, a synthetic object insertion data set, comprising, for each initial image in the initial data set: identifying one or more objects that are in the respective scene depicted in the initial image; for each of one or more of the identified objects: generating, using an object removal neural network, an object- less image that has the identified object removed from the respective scene depicted in the initial image; generating a blended image from the object-less image for the identified object and the region of the initial image that depicts the identified object; and generating a training example that comprises (i) the initial image, (ii) the blended image, and (iii) context data specifying the region of the initial image that depicts the identified object; and training a diffusion neural network on the synthetic object insertion data set to perform an object insertion task. Clause 10. The method of clause 9, wherein the initial data set comprises a plurality of images that includes the set of initial images and wherein generating, from the initial data set, the synthetic object insertion data set further comprises: identifying the set of initial images by filtering the plurality of images in the initial data set. Attorney Docket No.56113-0700WO1 Clause 11. The method of clause 9 or clause 10, further comprising after training the diffusion neural network on the synthetic data set to perform the object insertion task, further training the diffusion neural network on the counterfactual training data set of any one of clauses 1-8. Clause 12. The method of any one of clauses 9-11, wherein the object removal neural network is the diffusion neural network of any one of clauses 1-8 that has been trained on the counterfactual training data set to perform the object removal task. Clause 13. The method of any one of clauses 9-12, wherein identifying one or more objects that are in the respective scene depicted in the initial image comprises: processing the initial image using a detector neural network to generate a detection output that identifies the one or more objects. Clause 14. The method of any one of clauses 9-13, wherein generating a blended image from the object-less image for the identified object and the region of the initial image that depicts the identified object comprises: combining (i) a masked image that masks out a region of the object-less image for the identified object that corresponds to the region of the initial image that depicts the identified object and (ii) the region of the initial image that depicts the identified object. Clause 15. The method of any one of clauses 9-14, wherein the diffusion neural network has been pre-trained on one or more image editing tasks. Clause 16. The method of clause 15, wherein the one or more image editing tasks comprise an image in-painting task. Clause 17. The method of any one of clauses 9-16, wherein training the diffusion neural network on the synthetic data set comprises training the diffusion neural network to, for each training example, generate the initial image in the training example conditioned on the blended image and the context data in the training example. Clause 18. The method of any one of clauses 9-17, wherein the context data is an object mask specifying the region of the initial image that depicts the identified object. Clause 19. The method of any one of clauses 9-18, wherein training the diffusion neural network comprises: sampling one or more of the training examples; for each of the one or more training examples: generating a noisy representation of the initial image in the training example by adding noise to an initial representation of the initial image; and Attorney Docket No.56113-0700WO1 processing a diffusion input that comprises (i) the noisy representation of the initial image, (ii) a representation of the blended image, and (iii) a representation of the context data using the diffusion neural network to generate a denoising output; training the diffusion neural network on an objective that measures an error between, for each training example, the denoising output and the noise added to the initial representation of the initial image. Clause 20. The method of any preceding clause, further comprising, after the training: receiving a new input image, an object image of a new object to be inserted into the new input image, and new context data for the new input image; and generating a modified image by performing an object insertion task specified by the new context data and the object image on the new input image using the trained diffusion neural network. Clause 21. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the operations of the respective method of any preceding clause. Clause 22. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the respective method of any preceding clause. Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes corresponded to in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. What is claimed is:

Claims

Attorney Docket No.56113-0700WO1 CLAIMS 1. A method performed by one or more computers, the method comprising: obtaining a counterfactual training data set that comprises a plurality of training samples, each of the plurality of training examples, comprising: a factual image that depicts a respective scene that includes one or more objects; a counterfactual image that depicts the respective scene with a first object of the one or more objects removed from the respective scene; and context data specifying the particular object that was removed from the respective scene; and training a diffusion neural network on the counterfactual training data set to perform an object removal task.

2. The method of claim 1, wherein the diffusion neural network has been pre-trained on one or more image editing tasks.

3. The method of claim 2, wherein the one or more image editing tasks comprise an image in-painting task.

4. The method of any one of claims 1-3, wherein the context data is an object mask specifying a region of the factual image that depicts the first object.

5. The method of any preceding claim, wherein training the diffusion neural network on the counterfactual data set to perform the object removal task comprises training the diffusion neural network to, for each training example, generate the counterfactual image in the training sample conditioned on the factual image in the training example and the context data in the training example.Attorney Docket No.56113-0700WO1 6. The method of any preceding claim, wherein training the diffusion neural network comprises: sampling one or more of the training examples; for each of the one or more training examples: generating a noisy representation of the counterfactual image in the training example by adding noise to an initial representation of the counterfactual image; and processing a diffusion input that comprises (i) the noisy representation of the counterfactual image, (ii) a representation of the factual image, and (iii) a representation of the context data using the diffusion neural network to generate a denoising output; training a diffusion neural network on an objective that measures an error between, for each training example, the denoising output and the noise added to the initial representation of the counterfactual image.

7. The method of any preceding claim, further comprising, after the training: receiving a new input image and new context data for the new input image; and generating a modified image by performing an object removal task specified by the new context data on the new input image using the trained diffusion neural network.

8. The method of any preceding claim, wherein, for each training example: the factual image in the training example is an image of a respective real-world scene taken by a camera with a particular camera pose before the first object is removed from the real-world scene; and the counterfactual image in the training example is an image of the respective real- world scene taken by the camera with the particular camera pose after the first object has been removed from the real-world scene.

9. A method performed by one or more computers, the method comprising: obtaining an initial data set that comprises a set of initial images that each depict a respective scene; generating, from the initial data set, a synthetic object insertion data set, comprising, for each initial image in the initial data set: identifying one or more objects that are in the respective scene depicted in the initial image; for each of one or more of the identified objects:Attorney Docket No.56113-0700WO1 generating, using an object removal neural network, an object-less image that has the identified object removed from the respective scene depicted in the initial image; generating a blended image from the object-less image for the identified object and the region of the initial image that depicts the identified object; and generating a training example that comprises (i) the initial image, (ii) the blended image, and (iii) context data specifying the region of the initial image that depicts the identified object; and training a diffusion neural network on the synthetic object insertion data set to perform an object insertion task.

10. The method of claim 9, wherein the initial data set comprises a plurality of images that includes the set of initial images and wherein generating, from the initial data set, the synthetic object insertion data set further comprises: identifying the set of initial images by filtering the plurality of images in the initial data set.

11. The method of claim 9 or claim 10, further comprising, after training the diffusion neural network on the synthetic data set to perform the object insertion task, further training the diffusion neural network on the counterfactual training data set of any one of claims 1-8.

12. The method of any one of claims 9-11, wherein the object removal neural network is the diffusion neural network of any one of claims 1-8 that has been trained on the counterfactual training data set to perform the object removal task.

13. The method of any one of claims 9-12, wherein identifying one or more objects that are in the respective scene depicted in the initial image comprises: processing the initial image using a detector neural network to generate a detection output that identifies one or more objects.

14. The method of any one of claims 9-13, wherein generating a blended image from the object-less image for the identified object and the region of the initial image that depicts the identified object comprises:Attorney Docket No.56113-0700WO1 combining (i) a masked image that masks out a region of the object-less image for the identified object that corresponds to the region of the initial image that depicts the identified object and (ii) the region of the initial image that depicts the identified object.

15. The method of any one of claims 9-14, wherein the diffusion neural network has been pre-trained on one or more image editing tasks.

16. The method of claim 15, wherein the one or more image editing tasks comprise an image in-painting task.

17. The method of any one of claims 9-16, wherein training the diffusion neural network on the synthetic data set comprises training the diffusion neural network to generate, for each training example, the initial image in the training sample conditioned on the blended image and the context data in the training example.

18. The method of any one of claims 9-17, wherein the context data is an object mask specifying a region of the initial image that depicts the first object.

19. The method of any one of claims 9-18, wherein training the diffusion neural network comprises: sampling one or more of the training examples; for each of the one or more training examples: generating a noisy representation of the initial image in the training example by adding noise to an initial representation of the initial image; and processing a diffusion input that comprises (i) the noisy representation of the initial image, (ii) a representation of the blended image, and (iii) a representation of the context data using the diffusion neural network to generate a denoising output; training a diffusion neural network on an objective that measures an error between, for each training example, the denoising output and the noise added to the initial representation of the initial image.

20. The method of any preceding claim, further comprising, after the training: receiving a new input image, an object image of a new object to be inserted into the new input image, and new context data for the new input image; andAttorney Docket No.56113-0700WO1 generating a modified image by performing an object insertion task specified by the new context data and the object image on the new input image using the trained diffusion neural network.

21. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the operations of the respective method of any preceding claim.

22. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the respective method of any preceding claim.

Images