Enhancement of images from mobile devices to give a professional camera effect

Abstract

A system and method for processing images from a mobile device such that the images appear to have been captured by a camera with particular characteristics, for example, by a digital SLR camera with particular settings. The system uses a trained image augmentation neural network. The image augmentation neural network can be trained without having to rely on pairs of images of the same scene, and several training methods are described.

Description

[Technical Field]

[0001] This specification relates to enhancing images from a mobile device, such as a smartphone, to allow a user to apply camera settings so that the image appears to have been captured by a camera using those settings. [Background technology]

[0002] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of a respective set of parameters. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] O. Ronneberger et al., arXiv:1505.04597 [Non-patent document 2] Ashish Vaswani et al., “Attention is all you need” [Non-patent document 3] Dosovitskiy et al., arXiv:2010.11929 [Non-patent document 4] Wang et al., "Multi-Scale Structural Similarity for Image Quality Assessment," Proc. IEEE Asilomar Conference on Signals, Systems and Computers, 2004, pp. 1398-1402. [Non-Patent Document 5] Danielyan et al., “Cross-color BM3D filtering of noisy raw data,” Intern. Workshop on Local and Non-Local Approximation in Image Processing, 2009, pp. 125-129. [Non-patent document 6] Nichol et al., "GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models," Section 2 [Non-Patent Document 7] Dwibedi et al., arXiv:1904.07846 Summary of the Invention [Means for solving the problem]

[0004] This specification describes systems and methods for processing images from a mobile device so that the images appear to have been captured by a camera with particular characteristics, e.g., particular camera settings or a particular type of lens. Generally, this is achieved using machine learning models. More particularly, it is recognized that neural networks can be trained using data such as Exchangeable Image File (EXIF) data that is typically captured when a digital camera is used to take a photograph, to enhance images from a mobile device to appear like images captured by a professional camera.

[0005] Thus, in one aspect, a computer-implemented method is described that may be implemented as a computer program on one or more computers at one or more locations, e.g., that may be implemented on a mobile device. The method includes capturing an image with a camera on the mobile device, e.g., a mobile phone, and obtaining user input data from a user interface of the mobile device that defines a set of one or more specified characteristics of the digital camera. The set of one or more specified characteristics defines one or more characteristics of an exposure triangle of settings that includes an aperture setting, a shutter speed setting, and an ISO setting of the digital camera.

[0006] The method determines, from user input data, conditioning tensors that represent characteristics of one or more specified characteristics, and processes an image captured by a camera of the mobile device using a trained image augmentation neural network while conditioned on the conditioning tensors to generate an augmented image that has the appearance of an image captured by the digital camera with the specified characteristics. The augmented image may be displayed to the user on the mobile device, stored locally or remotely for the mobile device, or transmitted, for example, for viewing by others.

[0007] In the implementation, an image augmentation neural network was trained while being conditioned on conditioning tensors defined by camera-characterizing metadata, e.g., Exchangeable Image File (EXIF) data.

[0008] The digital camera may be a "full-fledged" camera, i.e., a camera that includes a camera body and interchangeable lenses. For example, the digital camera may be a DSLR (digital single-lens reflex) camera or a mirrorless interchangeable-lens camera (MILC).

[0009] The method may be implemented in particular embodiments to achieve various advantages. Counterintuitively, implementations of trained image augmentation neural networks can produce very high-quality images from mobile device cameras, e.g., smartphone cameras, exceeding the apparent physical limitations of the lenses and sensors originally used to capture the image. Also, lens effects that are otherwise difficult to achieve without using a professional camera can be achieved. For example, professional photographers can control the degree of bokeh using camera settings, but this is difficult to simulate well using, for example, a depth-masked blur filter. As another example, implementations of the method facilitate the simultaneous application of multiple effects, which are difficult to achieve through simulation.

[0010] One might assume that such a neural network would need to be trained on pairs of identical images, one captured using a mobile device and the other captured using a digital camera with specific settings. While this might be possible, it would involve laborious collection of training datasets. However, it is recognized that image augmentation neural networks can also be trained without using such paired training data. In implementations, image augmentation neural networks were trained using an objective that did not require images captured by a mobile device's camera to be paired with corresponding augmented images.

[0011] One way that an image augmentation neural network can be trained without using paired training data is by jointly training the image augmentation neural network with an image restoration neural network. In such an approach, during training, an image is sequentially processed using both the image augmentation neural network and the image restoration neural network to restore a version of the image. The parameters of the image augmentation neural network and the image restoration neural network are updated to increase the consistency between the image and the restored version of the image based, among other things, on the gradient of an objective function that depends on the difference between the image and the restored version of the image. This allows the image augmentation neural network to be trained using unpaired images.

[0012] The training dataset for the system described herein includes two sets of images: a set of source camera images captured by one or more source cameras of one or more mobile devices, and a set of digital camera images captured by one or more digital cameras, which have camera-characterizing metadata, e.g., EXIF ​​data, that defines, for the digital camera images, one or more characteristics or settings of the camera body and lens used to capture the digital camera images.

[0013] The image augmentation neural network is trained to generate augmented images using source camera images while conditioned on metadata characterizing the camera for generating the augmented images. Thus, the image augmentation neural network is trained to generate images from a distribution corresponding to the distribution of digital camera images. In some implementations, the image augmentation neural network is configured and trained to process source camera images to directly generate augmented images according to the metadata characterizing the camera. In other implementations, the image augmentation neural network is trained to denoise noisy versions of digital camera images while conditioned on camera-characterizing metadata for the digital camera images, and then used to process the source camera images to generate augmented images according to the metadata characterizing the camera.

[0014] Similarly, an image recovery neural network is trained to generate, from digital camera images, restored images that have the appearance of the source camera images. Thus, the image recovery neural network is trained to generate images from a distribution that corresponds to the distribution of the source camera images. In some implementations, the image recovery neural network is configured and trained to directly process the digital camera images to generate the restored images. In other implementations, the image recovery neural network is trained to denoise a noisy version of the source camera images, and then used to process the digital camera images to generate the restored images.

[0015] The methods and systems described herein can be implemented in particular embodiments to realize one or more of the following additional advantages.

[0016] Although smartphone cameras have improved, they are still limited by the lens and the amount of light they can capture. However, in digital photography, camera settings such as the size and type of lens, the response of the camera sensor, and shutter speed all play an important role in the captured image obtained. While smartphones can simulate the effects of some of these settings, such as blur, such simulations can be inaccurate and suffer from artifacts such as edge artifacts resulting from the low resolution of the sensor or compounding errors when multiple image processing models are used. Implementations of the described system can address these issues and provide a trained image augmentation neural network that can generate augmented images that simulate authentic camera images with fewer artifacts and improved image quality, even exceeding the apparent physical limitations of smartphones.

[0017] Training a neural network end-to-end using pairs of images of the same scene captured by a mobile phone and a digital, e.g., full-scale, camera would involve time-consuming collection of training image pairs. However, the described techniques instead allow image augmentation neural networks to be trained using unpaired images, both source camera images from the mobile device and digital camera images, which allows for leveraging a much larger corpus of training data and, therefore, improved results.

[0018] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 illustrates an example of a mobile device with an image enhancement system. [Figure 2a] FIG. 1 illustrates an example system for training an image augmentation neural network. [Figure 2b] FIG. 2b shows details of a particular example of the system of FIG. 2a. [Figure 3] FIG. 2b is a flow diagram of an exemplary process for training an image augmentation neural network using the system of FIG. 2a. [Figure 4] 2c is a flow diagram of an exemplary process for training an image augmentation neural network using the system of FIG. 2b. [Figure 5] 10 is a flow diagram of a further exemplary process for training an image augmentation neural network. [Figure 6] 1 is a flow diagram of an example process for enhancing an image from a mobile device so that the image appears to have been captured by a digital camera. [Figure 7] 1 is a flow diagram of an exemplary process for using an image augmentation neural network to process an image. DETAILED DESCRIPTION OF THE INVENTION

[0020] Like reference numbers and designations in the various drawings indicate like elements.

[0021] 1 shows an example of a mobile device 100 with an image enhancement system 102 for enhancing images captured by the mobile device, as described further below. The image enhancement system 102 may be implemented as one or more computer programs on one or more computers in one or more locations. More particularly, the image enhancement system 102 may be implemented on the mobile device 100, on a remote server, or partly on the mobile device 100 and partly on a remote server.

[0022] The mobile device 100 may be, for example, a mobile phone (cell phone) or a smartphone, or a tablet computing device. The mobile device 100 includes a camera 104, e.g., a front or rear camera, and a display screen 100a to provide a user interface 106. As some examples, the user interface 106 may include a touch interface implemented, for example, by the touch-sensitive display screen 100a, or a gesture interface implemented, for example, using the camera 104, or a spoken word user interface implemented by capturing speech from a microphone (not shown) of the mobile device.

[0023] The image enhancement system 102 includes an image enhancement neural network 110. The image enhancement neural network 110 has an image enhancement neural network input 112 and an image enhancement conditioning input 114 and is configured to process the image enhancement neural network input 112 according to current values ​​of parameters, e.g., weights, of the image enhancement neural network while conditioned on the image enhancement conditioning input 114, to generate an image enhancement neural network output 116. More specifically, the image enhancement neural network 110 is configured to obtain the image enhancement neural network input 112 from the camera 104 and process an image captured by the camera 104 accordingly, to generate an enhanced image at the image enhancement neural network output 116. The image may be a still image or a moving image.

[0024] The image augmentation system 102 also includes a conditioning tensor determination subsystem 108. The image augmentation system 102 is configured to receive user input data from the user interface 106, the data defining a set of one or more specified characteristics of the digital camera. The set of one or more specified characteristics defines one or more characteristics of an exposure triangle of settings, including the digital camera's aperture setting, the digital camera's shutter speed setting, and the digital camera's ISO setting (which roughly corresponds to the digital camera's film speed). The conditioning tensor determination subsystem 108 receives the user input data and processes the user input data, for example, according to parameters of the conditioning tensor determination subsystem, to generate a conditioning tensor that represents a characteristic of the one or more specified characteristics. As used herein, a conditioning tensor is a numerical tensor. The conditioning tensor determination subsystem 108 may be implemented, for example, using a learned encoding matrix or an embedding neural network, for example, a feedforward neural network.

[0025] The image augmentation neural network 110 generates an augmented image while conditioned on the conditioning tensor, such that the augmented image has the appearance of an image captured by a digital camera with specified characteristics, as will be further described below. An example of the operation of the image augmentation system 102, and more particularly, the image augmentation neural network 110, is described below with reference to Figures 6 and 7.

[0026] FIG. 1 also shows a block diagram illustrating some of the components of an exemplary mobile device 100. Thus, the mobile device 100 includes one or more processors 101, non-volatile storage 105, and one or more communication subsystems 103 for wireless communication with a computer or mobile telephone network. These and the camera 104 are coupled via a device bus. The storage 105 stores instructions and data used by the processor 101 to implement the image augmentation system 102. More specifically, the storage 105 stores image augmentation code 105a for implementing the conditioning tensor determination subsystem 108 and for implementing image augmentation using an image augmentation neural network 110, as well as operating system code 105d. If the image augmentation neural network 110 is implemented in the mobile device, the storage 105 also stores parameters 105b for the image augmentation neural network 110. The storage 105 may also include image storage 105c, for example, for storing captured or augmented images.

[0027] In some implementations of the system, the set of one or more specified characteristics defined by the user-input data includes at least two settings, for example, all three settings, of an exposure triangle of settings. In some implementations, the specified characteristics may define that one of an aperture setting and a shutter speed setting is preferred over the other. The set of one or more specified characteristics may also include an exposure compensation setting to allow the enhanced image to be underexposed or overexposed, or a flash setting to specify that the enhanced image appears to have been taken using flash lighting.

[0028] If the digital camera is a camera that includes a camera body and interchangeable lenses, system implementations also allow a user to specify camera characteristics including the body type of the camera body or the lens type of the interchangeable lenses, e.g., manufacturer or model or body type or lens type. The lens type may additionally or alternatively include the focal length of the lens, or the class of the lens, e.g., macro, fisheye, or telephoto. The set of one or more specified characteristics may specify that the camera is a digital SLR (DSLR) camera or a MILC.

[0029] In some implementations, the augmented images have a higher image resolution than the resolution of the images captured by the mobile device's camera, i.e., the image augmentation system 102 can provide super-resolution imaging. In particular, training techniques described below enable the image augmentation neural network 110 to add realistic, high-resolution detail to images captured at a lower resolution; the image augmentation neural network can do this because it has "seen" many different images. That is, a trained image augmentation neural network may be used to add image detail to images captured by the mobile device's camera. In implementations, the added detail i) is generated at a resolution corresponding to a specified image resolution (which may be indirectly specified, for example, by specifying a camera body type) and ii) matches the image detail at the resolution of the image captured by the mobile device, depicting the content of the captured image.

[0030] In some implementations, the user interface 106 may include a graphical user interface implemented, for example, by the touch-sensitive display screen 100 a, that simulates the appearance of a digital camera with settings to allow the user to define the characteristics of the exposure triangle.

[0031] As described further below, in an implementation, the trained image augmentation neural network was trained while conditioned on conditioning tensors defined by metadata characterizing the camera, such as Exchangeable Image File (EXIF) data. In principle, the image augmentation neural network may be trained end-to-end using pairs of images of the same scene captured by a mobile device and a digital, e.g., professional, camera, while conditioned on conditioning tensors defined by the EXIF ​​data. Images from the mobile device may be provided as input to the image augmentation neural network, and the neural network may be trained by backpropagating the gradient of an objective function that depends on the difference between an image of the scene generated by processing the captured image using the image augmentation neural network and an image of the same scene captured by a digital, e.g., professional, camera. However, in an implementation, the trained image augmentation neural network is trained (i.e., by backpropagating its gradient) using an objective that does not require that images captured by the mobile device's camera be paired with corresponding augmented images, e.g., a cycle consistency objective.

[0032] Thus, FIG. 2a illustrates an example of an image augmentation neural network training system 200, which may be implemented as one or more computer programs on one or more computers at one or more locations. System 200 may be used to train image augmentation neural network 110 to augment images from a mobile device so that the images appear to be images captured by a digital camera, such as a DSLR camera or MILC, with characteristics defined by image conditioning inputs. The digital camera may be a full-scale camera, which, as used herein, is a camera with a camera body and interchangeable lenses. The image conditioning inputs may define characteristics of the digital camera, such as body type, lens type, etc. As used herein, the “characteristics” of a digital camera further include digital camera settings, such as aperture setting, shutter speed setting, and ISO setting (equivalent to film sensitivity setting).

[0033] In some other applications, system 200 may be used to train image augmentation neural network 110 to process images from a digital, e.g., DSLR camera or MILC, to make the images appear to be images captured with a particular lens or camera setting not available to the user, e.g., to add a "virtual lens" to the user's digital, e.g., DSLR camera or MILC, or to virtually upgrade the user's digital, e.g., DSLR camera or MILC to a high-end camera.

[0034] The image to be processed may be a monochrome or color image and may be represented by a pixel value, e.g., an RGB (red-green-blue) pixel value, for each pixel of the image. The image may also include additional information, e.g., an image depth map containing representative pixel values ​​for each pixel. The image may be, e.g., a composite image derived from multiple sensors (cameras) with different resolutions, or the image may include, e.g., multiple image channels with different spatial resolutions. The image may be a still image or a moving image. That is, as used herein, reference to an "image" includes reference to an image comprising multiple video frames. For example, the image augmentation neural network 110 may be configured to process a video input to generate an augmented video output.

[0035] As described above, the image augmentation neural network 110 processes both the image augmentation neural network input 112 and the image augmentation conditioning input 114 to generate the image augmentation neural network output 116. The image augmentation neural network output 116 may have the dimensions of the image augmentation neural network input 112. The image augmentation neural network output 116 may define an image, or may define a correction to be applied to an image to augment the image (and may have the dimensions of the image to which the correction should be applied). The image augmentation neural network input 112 is configured to receive a vector defining an image, although, as described later, this may sometimes be a noise vector defining an image that is pure noise. The image augmentation conditioning input 114 may include metadata characterizing a camera, as described below, particularly data defining one or more characteristics of a digital camera by which the generated augmented image appears to have been captured.

[0036] The image augmentation neural network 110 has a number of image augmentation neural network parameters, e.g., weights, that are adjusted by the training engine 130 during training of the system 200 to train the image augmentation neural network 110 to perform image augmentation functions, as will be described later.

[0037] The image augmentation neural network training system 200 also includes an image restoration neural network 120 having an image restoration neural network input 122 and configured to process this input to generate an image restoration neural network output 126 that includes a restored image. In some implementations, the image restoration neural network 120 also has an image restoration conditioning input 124, and the image restoration neural network 120 is configured to process the image restoration neural network input 122 while conditioned on the image restoration conditioning input 124 to generate the image restoration neural network output 126. The image restoration neural network output 126 may have the dimensions of the image restoration neural network input 122. The image restoration neural network output 126 may define the image or may define a correction to be applied to the image to restore (“de-enhance”) a version of the image (and may have the dimensions of the image to which the correction should be applied). The image restoration neural network input 122 is configured to receive a vector defining the image, although, as will be explained later, this may sometimes be a noise vector defining an image that is pure noise.

[0038] The image restoration neural network 120 has a number of image restoration neural network parameters, e.g., weights, that are adjusted by the training engine 130 during training of the system 200 to train the image restoration neural network 120 to perform an image restoration function, as also described below.

[0039] The image augmentation neural network 110 and the image restoration neural network 120 may have any neural network architecture capable of accepting image input and processing it to provide image output. Generally, the neural networks may have any suitable number, e.g., 1 to 100 layers, and any suitable type of neural network layers, e.g., fully connected layers, attention layers, convolutional layers, etc., connected in any suitable configuration, e.g., as a linear sequence of layers. For example, the image augmentation neural network 110 and the image restoration neural network 120 may each have a U-Net neural network architecture (O. Ronneberger et al., arXiv:1505.04597) that includes multiple downsampling, e.g., convolutional “analysis” layers, followed by multiple upsampling, e.g., convolutional “synthesis” layers, optionally including one or more attention layers, with skip connections between the analysis and synthesis layers.

[0040] Conditioning may be applied at one or more or all of the layers of the image augmentation neural network 110 and the image restoration neural network 120. For example, in some implementations, the conditioned neural network inputs may be concatenated or summed with the neural network inputs or may provide additional channels for the image input. In some implementations, the conditioned neural network inputs may also or instead be applied at one or more intermediate layers. If it is necessary to match the dimensions of the conditioned neural network inputs with the dimensions of the layer to which they are applied, this may be done by encoding the conditioned neural network inputs with a learned encoding matrix.

[0041] The system 200 includes a data store that holds training data 140. In some implementations, the training data includes a set of source camera images captured by one or more source cameras, i.e., cameras of a mobile device. The training data 140 further includes a set of digital camera images captured by one or more digital cameras, e.g., DSLR or MILC cameras, and metadata characterizing the corresponding camera for each of the digital camera images. In some cases, camera-characterizing metadata may be available for some or all of these source camera images, even if the source camera images are from a camera of a mobile device, e.g., a mobile phone. Either or both of the source camera images and the digital camera images may include images from multiple sensors, as described above, or may include moving images, i.e., video. An advantage of the system implementation is that paired source camera images and digital camera images, i.e., two images of the same scene captured by a source camera and a digital camera, respectively, are not required.

[0042] Camera-characterizing metadata for a digital camera image defines one or more characteristics of the digital camera when it was used to capture the image. For example, the camera-characterizing metadata may include EXIF ​​(Exchangeable Image File) data defined in or compatible with JEITA Standard Version 1.x or 2.x or later, e.g., standard CP-3451C. The camera-characterizing metadata may define one or more of: lens focal length; lens type (e.g., wide-angle, zoom, or normal); lens aperture (e.g., F-stop); exposure time; light source (e.g., flash, sunlight, tungsten, or fluorescent); sensor sensitivity as, e.g., ISO speed rating; camera body type (e.g., camera make / model); or other information (e.g., scene type information, subject distance, subject brightness, image size, image resolution, and degree of compression).

[0043] In some cases, camera-characterizing metadata may be missing from some of the digital camera images. Accordingly, implementations of the system may include a metadata reconstruction neural network 142, e.g., a convolutional neural network, trained to reconstruct missing camera-characterizing metadata for one or more of the digital camera images. It may be trained to predict missing camera-characterizing metadata using images for which desired camera-characterizing metadata is present. For example, the metadata reconstruction neural network 142 may have inputs including images and partial camera-characterizing metadata and may be configured to process the inputs to generate outputs including additional camera-characterizing metadata, e.g., to provide complete camera-characterizing metadata for the image augmentation conditioning input 114 of the image augmentation neural network. The metadata reconstruction neural network 142 may be trained using digital camera images for which such complete camera-characterizing metadata is available, e.g., masking elements of the metadata to generate training inputs. In some cases, metadata characterizing the missing camera may be determined or inferred from a database; for example, the type of lens may be used to determine its focal length, or the "film speed" may be retrieved from a database using the make / model of the camera body.

[0044] Figure 2b shows one particular implementation of the image augmentation neural network training system 200 of Figure 2a. This particular implementation includes a training image classifier neural network 201 and a source image classifier neural network 210.

[0045] The training image classifier neural network 201 has a training image classifier input 202 for receiving training image classifier input images and is configured to process the training image classifier input images to generate a training image classifier output 206 that includes a prediction of whether the training image classifier input images are genuine digital camera images rather than augmented source camera images. For example, the training image classifier output 206 may generate a value representing the probability that the training image classifier input images are genuine digital camera images.

[0046] In some implementations, the training image classifier neural network 201 also has training image classifier conditioning inputs 204 and is configured to process training image classifier input images while conditioned on the training image classifier conditioning inputs 204 to generate a training image classifier output 206. For example, if the training image classifier input images include augmented source camera images, the training image classifier conditioning inputs 204 may include metadata characterizing a camera used to condition the image augmentation neural network 110 when generating the augmented source camera images. If the training image classifier input images include digital camera images, the training image classifier conditioning inputs 204 may include metadata characterizing a camera for the digital camera images. Generally, the metadata characterizing a camera for the digital camera images defines one or more characteristics of an exposure triangle of settings, including the aperture setting, shutter speed setting, and ISO setting of the digital camera used to capture the images.

[0047] The training image classifier neural network 201 has a number of training image classifier neural network parameters, e.g., weights, that are adjusted by the training engine 130 during training of the system 200 in order to train the training image classifier neural network to generate correct predictions, as will be described below.

[0048] The source image classifier neural network 210 has a source image classifier input 212 for receiving a source image classifier input image and is configured to process the source image classifier input image to generate a source image classifier output 216 that includes a prediction of whether the source image classifier input image is a genuine source camera image rather than a source camera image recovered (i.e., generated) from a digital camera image. For example, the source image classifier output 216 may generate a value representing the probability that the source image classifier input image is a genuine source camera image.

[0049] In some implementations, the source image classifier neural network 210 also has a source image classifier conditioning input 214 and is configured to process the source image classifier input images while conditioned on the source image classifier conditioning input 214 to generate a source image classifier output 216. For example, if the source image classifier input images include recovered source camera images, i.e., source camera images generated from digital camera images, the source image classifier conditioning input 214 may include metadata characterizing a camera for the digital camera images. If the source image classifier input images include source camera images, the source image classifier conditioning input 214 may include metadata characterizing a random camera or a null camera, or metadata characterizing a camera for the source camera images, if available.

[0050] The source image classifier neural network 210 has a number of source image classifier neural network parameters, e.g., weights, that are adjusted by the training engine 130 during training of the system 200 in order to train the source image classifier neural network to generate correct predictions, as will be described later.

[0051] In the implementation of Figure 2b, image augmentation neural network 110 receives a source camera image at image augmentation neural network input 112 and receives metadata characterizing the camera for the source camera image at image augmentation conditioning input 114. Image augmentation neural network 110 is trained to generate image augmentation neural network output 116 that includes an augmented image that looks like an image captured by a camera with the characteristics defined by the image augmentation conditioning input.

[0052] The image augmentation neural network input 112 uses the image augmentation conditioning input 114 to define the appearance of the augmented image it generates. That is, in implementations, the augmented image has an appearance defined by camera characteristics according to metadata characterizing the camera that is provided to the image augmentation conditioning input 114 while the augmented image is being generated. Optionally, stochasticity, i.e., noise, may be added when generating the augmented image.

[0053] The image restoration neural network 120 receives digital camera images and, optionally, metadata associated with the digital camera images that characterize the camera, and is trained to generate an image restoration neural network output 126 that includes a restored image that looks like the image captured by the source camera. Optionally, random elements, such as noise, may be added when generating the restored image.

[0054] While the image restoration neural network 120, the training image classifier neural network 201, and the source image classifier neural network 210 do not require camera-characterizing metadata to perform their respective functions, this data can help the neural networks learn to "undo" the effects of the camera settings represented by the camera-characterizing metadata. Generally, conditioning one or more of the image restoration neural network 120, the training image classifier neural network 201, and the source image classifier neural network 210 with camera-characterizing metadata, as described above, can improve overall system performance, e.g., reducing artifacts.

[0055] The training image classifier neural network 201 and the source image classifier neural network 210 may have any neural network architecture capable of accepting image input and processing it to provide a predicted output. In general, the neural networks may have any suitable number, e.g., 1 to 100 layers, of any suitable type of neural network layers, e.g., fully connected layers, attention layers, convolutional layers, etc., connected in any suitable configuration, e.g., as a linear sequence of layers.

[0056] In some implementations, the training image classifier neural network 201 and / or the source image classifier neural network 210 may each include two "virtual" classifiers, each configured to operate on different aspects of the input image. For example, a first such virtual classifier operates on global image features, while a second virtual classifier operates on local image patches.

[0057] Thus, in some implementations, the source (or training) image classifier neural network includes a first source (or training) image classifier and a second source (or training) image classifier. The first source (or training) image classifier is configured to process the source (or training) image classifier input images to generate a first intermediate source (or training) image prediction of whether the source (or training) image classifier input images are authentic source (or training) camera images. The second source (or training) image classifier is configured to process each of a plurality of source (or training) image patches, i.e., image regions, that divide the source (or training) image classifier input images into tiles to generate a second intermediate source (or training) image prediction of whether the source (or training) image classifier input images are authentic source (or training) camera images. The source (or training) image classifier neural net may then combine the first intermediate source (or training) image prediction with the second intermediate source (or training) image prediction to generate a source (or training) image classifier output, i.e., a prediction of whether the source (or training) image classifier image input is a genuine source (or training) camera image.

[0058] In some implementations, the training image classifier neural network 201 and / or the source image classifier neural network 210 may each include a self-attention mechanism, which may enable the classifier neural network to compare different parts of the image input and direct attention to the parts that are most suitable for comparison, and which may facilitate efficient processing of high-resolution images and videos.

[0059] Thus, in some implementations, the source (or training) image classifier neural network includes one or more attention neural network layers, each configured to apply an attention mechanism to attention layer inputs to generate an attention layer output. The source (or training) image classifier neural network is configured to process each of a plurality of source (or training) image patches, i.e., image regions, that divide the source (or training) image classifier input image into tiles to generate a set of source image (or training) patch encodings or feature maps. The source (or training) image classifier neural network is further configured to process a set of image patch encodings by applying an attention mechanism to the set of source (or training) image patch encodings to generate a set of transformed source (or training) image patch encodings. These may then be combined using one or more further neural network layers, such as, for example, an MLP (multilayer perceptron), to generate a prediction of whether the source (or training) image classifier image input is a genuine source (or training) camera image.

[0060] The patch encodings may be generated by processing the image patches using a trained embedding matrix or a shared feature encoder neural network, e.g., a convolutional neural network. Each patch encoding may then be combined with a 1D or 2D positional encoding representing the position of the image patch within the input image, e.g., by summing or concatenating the patch encoding and the positional encoding. In implementations, the positional encoding has the same dimensions as the patch encoding and is trained or predefined. The attention mechanism may be configured to generate a query vector and a set of key-value vector pairs from the attention layer input and calculate the attention layer output as a weighted sum of the values. The weight assigned to each value may be calculated by a compatibility (similarity) function of the query with each corresponding key, e.g., the dot product or the scaled dot product compatibility (similarity). The query vector and the set of key-value vector pairs may be determined by respective trained matrices applied to the attention layer input. In the implementation, the query vector and the set of key-value vector pairs are determined from the same attention layer input, and the attention mechanism is a self-attention mechanism. In general, the attention mechanism may be similar to the attention mechanism described in Ashish Vaswani et al., "Attention is all you need" and Dosovitskiy et al., arXiv:2010.11929.

[0061] In another specific implementation of the image augmentation neural network training system 200 of Figure 2a, the image augmentation neural network 110 and the image restoration neural network 120 are denoising neural networks, i.e., the image augmentation neural network 110 is trained to denoise a noisy version of a digital camera image received by the image augmentation neural network input 112, and the image restoration neural network 120 is trained to denoise a noisy version of a source camera image received by the image restoration neural network input 122.

[0062] In this particular implementation, the image augmentation neural network input 112 is configured to receive input images, including noisy digital camera images, during training, and the image augmentation conditioning input 114 is configured to receive camera-characterizing metadata for the input images during training. The image augmentation neural network output 116 may define a correction to be applied to the input image to obtain a reduced-noise input image, or the image augmentation neural network output may be the noise-reduced input image. In either case, the input image is iteratively processed to generate an augmented image with progressively less noise.

[0063] After training, the image augmentation neural network input 112 receives the input image to be processed, and the image augmentation conditioning input 114 receives metadata characterizing the camera that defines the properties of the digital camera such that the augmented image produced by the image augmentation neural network input 112 has the appearance of an image captured by the digital camera.

[0064] The image restoration neural network 120 operates similarly, receiving the noisy source camera image during training to denoise the noisy source camera image. The image augmentation neural network 110 and the image restoration neural network 120 are also applied sequentially to either the source camera image or the digital camera image and trained to recreate a version of this image, with the noisy digital camera image or the noisy source camera image being replaced by the source camera image or the digital camera image, respectively. This is described in more detail below.

[0065] In this implementation, the image augmentation neural network 110 and the image restoration neural network 120 may have the same architecture as previously described, for example, the U-Net neural network architecture.

[0066] Generally, in the implementation described herein, after the image augmentation neural network 110 and the image restoration neural network 120 are trained by the image augmentation neural network training system 200, only the trained image augmentation neural network 110 is retained. This provides a neural network that is configured to process images so that they appear to have been captured by a camera with particular characteristics, i.e., characteristics defined by the image augmentation conditioning inputs 114.

[0067] Figure 3 is a flow diagram of an exemplary process for using the system 200 of Figure 2a to train the image augmentation neural network 110. The process of Figure 3 is repeated multiple times during training.

[0068] In step 300, the process obtains first, second, and further training examples by selecting them from training data 140. The first training example includes the selected source camera images, the second training example includes the selected digital camera images, and the further training example includes further images selected from either one of the source camera images or one of the digital camera images.

[0069] The process then trains an image augmentation neural network using one of the first training examples and the second training examples to generate a first augmented image while conditioned on metadata characterizing the camera for generating the first augmented image (step 302).

[0070] Training the image augmentation neural network to generate the first augmented image may include training the image augmentation neural network to generate an output including the augmented image. Alternatively, training the image augmentation neural network to generate the first augmented image may include training the image augmentation neural network to generate an output including a partially de-noised version of the input image, such that augmented (de-noised) images may be iteratively generated. Alternatively, training the image augmentation neural network to generate the first augmented image may include training the image augmentation neural network to generate an output representing noise in the input image, which may be used to correct the input image to generate the augmented image. In this manner, the process trains the image augmentation neural network, so that after training, the image augmentation neural network can be used to generate the first augmented image directly (by outputting the augmented image) or indirectly (e.g., by outputting image data for combination with the input image to provide the augmented image).

[0071] Generally, the first augmented image has the appearance of (is similar to) a digital camera image. That is, the first augmented image has the appearance of an image drawn from a distribution of digital camera images. Depending on the implementation, the first augmented image may have the appearance of a particular digital camera image in the training data, but this is not necessarily the case.

[0072] The process also trains an image restoration neural network using the other of the first training example and the second training example to generate a first restored image (step 304). Generally, the first restored image has the appearance of (is similar to) the source camera image. That is, the first restored image has the appearance of an image drawn from a distribution of source camera images. Depending on the implementation, the first restored image may have the appearance of a specific source camera image in the training data, but this is not necessarily the case.

[0073] Training the image restoration neural network to generate the first recovered image may include training it to generate an output including the restored image, or training it to generate an output including a partially de-noised version of the input image, or training it to generate an output representing noise in the input image that can be used to correct the input image to generate the recovered image. The process trains the image restoration neural network so that, after training, the image restoration neural network can be used to generate the first recovered image directly (by outputting the restored image) or indirectly (e.g., by outputting image data for combination with the input image to provide the recovered image).

[0074] The process also uses the additional image to jointly train both the image augmentation neural network and the image restoration neural network (step 306). More specifically, the additional image is processed sequentially (in either order), first using one of the image augmentation neural network and the image restoration neural network, and then using the other, to reconstruct a version of the additional image. In some implementations, the processing using the image augmentation neural network and the image restoration neural network may include a process of iteratively refining the generated image. The process then updates the image augmentation neural network parameters and the image restoration neural network parameters to increase consistency between the additional image and the reconstructed version of the additional image.

[0075] This step provides additional constraints such that the image augmentation neural network and the image restoration neural network are subject to additional constraints when they are trained to perform their respective image augmentation and image restoration functions, with the goal of ensuring that the image augmentation neural network, when used to augment an input image, generates an image with similar content to the input image, and similarly for the image restoration neural network.

[0076] In some implementations, the image augmentation neural network parameters and the image restoration neural network parameters may be updated based on a gradient of an objective function that depends on the difference between the further image and a reconstructed version of the further image. The objective function may include, for example, an L1 loss or an L2 loss.

[0077] In some implementations, the difference comprises a SSIM (Structural Similarity Index Measure) index of a reconstructed version of the further image, calculated using the further image as a reference. This may comprise a weighted combination of a comparison of brightness l, contrast c, and structure s between one or more windows (aligned spatial patches) of the two images. For example, for each window x and y, the SSIM is

[0078]

number

[0079] may be determined from μ x and μ y is the average pixel intensity value of the window, and σ x and σ y is the variance, and σ xyis the covariance of x and y, and c1, c2, and c3 are constants. In some implementations, the SSIM index may be a variant of SSIM, such as the multi-scale SSIM (MS-SSIM) index described, for example, in Wang et al., "Multi-Scale Structural Similarity for Image Quality Assessment," Proc. IEEE Asilomar Conference on Signals, Systems and Computers, 2004, pp. 1398-1402. In some other implementations, the value of the objective function may be determined using the BM3D (Block-matching and 3D filtering) algorithm, Danielyan et al., "Cross-color BM3D filtering of noisy raw data," Intern. Workshop on Local and Non-Local Approximation in Image Processing, 2009, pp. 125-129. In some implementations, the objective function may include a combination of two or more of the above losses.

[0080] In implementations, the gradient may be determined from one or a mini-batch of training examples. Generally, any gradient-based learning rule may be used in the implementations described herein. References to training a neural network based on the gradient of an objective function generally relate to backpropagating the gradient of the objective function through the neural network being trained.

[0081] Figure 4 is a flow diagram of an exemplary process for using the implementation of the image augmentation neural network training system 200 shown in Figure 2b to train the image augmentation neural network 110. Again, the process of Figure 4 is repeated multiple times during training.

[0082] In step 400, the process obtains the aforementioned first, second, and further training examples, as well as third and fourth training examples, by selecting them from training data 140. The third training example includes one of the source camera images, and the fourth training example includes one of the digital camera images and, in implementations, metadata characterizing the corresponding camera.

[0083] The third training example is processed by the source image classifier neural network 210 to generate a first prediction of whether the third training example contains authentic source camera images or synthetic source camera images (i.e., source camera images generated using the image restoration neural network) (step 402). The source image classifier neural network parameters are then updated to reduce the error in the first prediction (step 404).

[0084] The source image classifier neural network parameters may be updated based on a gradient of an objective function that depends on D(x), where D(x) is the likelihood determined by the source image classifier neural network that the third training example includes the genuine source camera image, x. In particular, the source image classifier neural network parameters may be updated to maximize D(x) when the third training example includes the genuine source camera image.

[0085] The source image classifier neural network parameters may be updated based on the gradient of an objective function that depends on D(G(y)), where G(y) is an image generated by the image restoration neural network 120. In particular, the source image classifier neural network parameters may be updated to minimize D(G(y)), for example, when the source image classifier neural network is provided with images generated by the image restoration neural network 120 by processing a digital camera image, y, sampled from training data, optionally conditioned on metadata characterizing the camera for the digital camera image. Thus, the source image classifier neural network parameters may be updated based on the gradient of the combined objective function that depends on D(x) - D(G(y)) to maximize D(x) - D(G(y)). In another example, the combined objective function that is maximized may depend on log D(x) + log (1 - D(G(y))). Optionally, randomness may be added when generating G(z) using the image restoration neural network, for example, by using dropout or adding noise to the input of the image restoration neural network 120.

[0086] The fourth training example is processed by the training image classifier neural network 201 to generate a second prediction of whether the fourth training example is a real digital camera image or a synthetic digital camera image (step 406). In implementations, but not necessarily, the digital camera images of the fourth training example are processed by the training image classifier neural network while the training image classifier neural network is conditioned on metadata characterizing the camera associated with the digital camera image. The training image classifier neural network parameters are then updated to reduce errors in the second prediction (step 408).

[0087] The training image classifier neural network parameters may be updated based on a gradient of an objective function that depends on D(x), where D(x) is the likelihood determined by the training image classifier neural network that the fourth training example includes a genuine digital camera image, x. In particular, the training image classifier neural network parameters may be updated to maximize D(x) when the fourth training example includes a genuine digital camera image.

[0088] The training image classifier neural network parameters may be updated based on the gradient of an objective function that depends on D(G(y)), where G(y) is an image generated by the image augmentation neural network 110. In particular, the training image classifier neural network parameters may be updated to minimize D(G(y)), e.g., when the training image classifier neural network is provided with images generated by the image augmentation neural network 110 by processing source camera images, y, sampled from training data while conditioned on metadata characterizing, e.g., randomly selected cameras.

[0089] Thus, the training image classifier neural network parameters may be updated based on the gradient of the combined objective function that depends on D(x) - D(G(z)) to maximize D(x) - D(G(z)). In another example, the combined objective function that is maximized may depend on log D(x) + log (1 - D(G(z))). Optionally, randomness may be added when generating G(z) using the image augmentation neural network, for example, using dropout or by adding noise to the input of the image augmentation neural network 110.

[0090] Generally, when the source image classifier neural network parameters and the training image classifier neural network parameters are updated based on the gradient of the objective function determined by processing real source camera images or digital camera images, the neural network parameters of the image augmentation neural network and the image restoration neural network are not updated.

[0091] The image augmentation neural network 110 processes the selected source camera images of the first training example while conditioned on metadata characterizing the camera for generating the first augmented image to generate a first augmented image (step 410). The metadata characterizing the camera for generating the first augmented image may be determined randomly, for example, by sampling metadata characterizing the camera from a distribution.

[0092] The distribution may be a uniform distribution, but in some implementations, the method includes determining, i.e., modeling, a joint distribution of the characteristics defined by the metadata characterizing the cameras in the training dataset. The metadata characterizing the cameras for the image augmentation neural network 110 may then be obtained by sampling from the joint distribution. For example, a neural network may be configured to implement a regression model and trained to model the joint probability distribution. Fitting the joint distribution is helpful because camera settings may be correlated, and selecting a combination of settings that deviates from the training distribution may make the classifier's task too easy, which may hinder training of the generator neural network 110, 120. Fitting the joint distribution may also facilitate interpolation between settings.

[0093] The first augmented image is processed by the training image classifier neural network 201, optionally conditioned on metadata characterizing the camera used to generate the first augmented image, to generate a third prediction of whether the first augmented image is a genuine digital camera image (step 412). The image augmentation neural network parameters are then updated to increase the error in the third prediction (step 414).

[0094] The image augmentation neural network parameters may be updated based on the gradient of an objective function that depends on D(G(x)), where G(x) is the first augmented image generated by the image augmentation neural network 110. In particular, the image augmentation neural network parameters are updated to maximize D(G(x)), or in some implementations, to minimize log(1 - D(G(x))).

[0095] As described above, each classifier neural network may operate to process an input image to generate a first intermediate prediction of whether the input image is real or synthetic, for example, based on processing the entire input image, and to generate a second intermediate prediction of whether the input image is real or synthetic by processing image patches that divide the input image into tiles. These intermediate predictions may be combined to generate a prediction of whether the input image is real or synthetic. Additionally or alternatively, each classifier may operate by processing the input image by applying an attention mechanism to multiple image patches, in particular the image patch coding, to generate a prediction of whether the input image is real or synthetic.

[0096] The image restoration neural network 120 processes the selected digital camera images of the second training example, optionally conditioned on metadata characterizing the camera for the selected digital camera images, to generate a first restored image (step 416). The first restored image is processed by the source image classifier neural network 210, optionally conditioned on metadata characterizing the camera for the selected digital camera images, to generate a fourth prediction of whether the first restored image is a genuine source camera image (step 418). The image restoration neural network parameters are then updated to increase the error in the fourth prediction (step 420).

[0097] The image restoration neural network parameters may be updated based on the gradient of an objective function that depends on D(G(x)), where G(x) is the first restored image generated by the image restoration neural network 120. In particular, the image restoration neural network parameters are updated to maximize D(G(x)), or in some implementations, to minimize log(1 - D(G(x))).

[0098] As described above, the process also jointly trains both the image augmentation neural network and the image restoration neural network using the additional image (step 422). If the additional image is acquired by selecting one of the source camera images, metadata characterizing the camera for the additional image may be acquired, for example, by random sampling from the learned distribution as described above. The additional image is then processed using the image augmentation neural network, conditioned on the metadata characterizing the camera for the additional image, to generate an augmented additional image. The augmented additional image is then processed using the image restoration neural network, optionally conditioned on the metadata characterizing the camera for the additional image, to recreate a version of the additional image.

[0099] The image augmentation neural network and the image restoration neural network are then trained, as described above, to increase the consistency between the further image and a reconstructed version of the further image, i.e., based on an objective function that depends on the difference between the further image and its reconstructed version, e.g., a loss that may be referred to as cycle consistency loss. As training progresses, the augmented further image begins to have the appearance of an image from a distribution of digital camera images. In an implementation, the source camera image is an image captured by a mobile phone, and the augmented further image has the appearance of an image captured by a DSLR camera or MILC.

[0100] If an additional image is obtained by selecting one of the digital camera images, the camera-characterizing metadata for the additional image may include camera-characterizing metadata for the selected digital camera image obtained from the training data. The additional image is then processed using the image restoration neural network to generate a restored additional image, and the restored additional image is processed using the image augmentation neural network, conditioned on the camera-characterizing metadata of the additional image, to recreate a version of the additional image. The image augmentation neural network and the image restoration neural network are then trained as described above.

[0101] Generally, the plurality of additional training examples used to train the system 200 includes additional images obtained by selecting from both the source camera images and the digital camera images.

[0102] 5 is a flow diagram of an exemplary process for using the system 200 shown in FIG. 2a to train the image augmentation neural network 110 when the image augmentation neural network 110 and the image restoration neural network 120 are denoising neural networks. Again, the process of FIG. 5 is repeated multiple times during training.

[0103] In step 500, the process obtains first, second, and further training examples by selecting them from training data 140, as described above. The first training example includes the selected source camera image, the second training example includes the selected digital camera image, and the further training example includes a further image selected from either one of the source camera images or one of the digital camera images. The process also obtains metadata characterizing the camera for the selected digital camera image.

[0104] The process then trains the image augmentation neural network 110 using the second training examples to denoise noisy versions of the selected digital camera images while conditioned on the metadata characterizing the camera for the selected digital camera images (step 502). The process also trains the image restoration neural network 120 using the first training examples to denoise noisy versions of the selected source camera images (step 504). Optionally, metadata characterizing the camera for the selected source camera images may be obtained, for example, by random selection as described above or by obtaining it from training data if available. The image restoration neural network 120 may then be trained while conditioned on the metadata characterizing the camera for the selected source camera images. The process also jointly trains the image augmentation neural network 110 and the image restoration neural network 120 (step 506), as described further below. A classifier neural network is not required.

[0105] An example of this process can use an image augmentation neural network and an image restoration neural network to implement a type of denoising diffusion model, but in inference, the model is not used for denoising itself. The denoising diffusion model includes a neural network that is trained for the purpose of denoising so that it may be used iteratively to remove various levels of noise from an image.

[0106] Thus, in an implementation, training the image augmentation neural network may include generating noisy versions of selected digital camera images and processing the noisy versions of the selected digital camera images using the image augmentation neural network conditioned on camera-characterizing metadata for the selected digital camera images to generate image augmentation neural network outputs. Values ​​of an image augmentation objective function are then determined, and image augmentation neural network parameters are updated using gradients of the image augmentation objective function with respect to the image augmentation neural network parameters. An example of using additional data to guide the denoising process is described in Section 2 of Nichol et al., "GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models" (incorporated by reference), and similar techniques may be used to condition on camera-characterizing metadata or encodings thereof.

[0107] Generating the noisy version of the selected digital camera image may include adding a noise vector ε to the selected digital camera image or otherwise corrupting the selected digital camera image. The noise vector may be sampled from a multivariate unit Gaussian distribution, ε ∼N(0,I), and have dimensions corresponding to the dimensions of the selected digital camera image. The noisy version of the selected digital camera image may be processed by an image augmentation neural network while further conditioned on a scalar noise level parameter, γ. The noise level parameter may be sampled from a distribution, for example, a piecewise uniform distribution. The image augmentation objective function may depend on the difference between the image augmentation neural network output and either i) the selected digital camera image or ii) a noise vector representing noise added to the selected digital camera image to generate the noisy version of the selected digital camera image. That is, the image augmentation neural network output 116 may be regressed onto the added noise or the original image.

[0108] In an exemplary implementation, the noisy version of the digital camera image is

[0109]

number

[0110] where y represents the selected digital camera image, and the image augmentation objective function is

[0111]

number

[0112] may depend on,

[0113]

number

[0114] where ε represents the p-norm, f(·) represents the image augmentation neural network output 116, and x represents the camera-characterizing metadata for the selected digital camera image. In the variant, the noise vector ε is replaced by the selected image y0.

[0115] Similarly, training the image restoration neural network may include generating a noisy version of the selected source camera image and processing it using the image augmentation neural network to generate an image restoration neural network output. The value of an image restoration objective function is then determined depending on the difference between the image restoration neural network output and either i) the selected source camera image or ii) a noise vector representing noise added to the selected source camera image to generate the noisy version of the selected source camera image, and the image restoration neural network parameters are updated using a gradient of the image restoration objective function with respect to the image restoration neural network parameters. The image restoration objective function may correspond to the example image augmentation objective function described above.

[0116] Image augmentation neural networks are trained similarly to denoising diffusion models, but are not used for denoising. When used for inference, image augmentation neural networks are given input images having the appearance of source camera images, i.e., the input images resemble images drawn from a distribution of source camera images, e.g., they resemble a mobile phone. The image augmentation neural network iteratively processes the input images to generate augmented images having the appearance of corresponding digital camera images, i.e., the augmented images resemble images drawn from a distribution of digital camera images, e.g., they resemble a DSLR or a MILC. During this iterative process, the image augmentation neural network is conditioned on metadata characterizing the camera that defines the appearance of the augmented image, more specifically, the camera's characteristics, e.g., settings, so that the augmented image appears to have been captured by a camera with those specific characteristics, e.g., those settings. During training, the image augmentation neural network learns about the distribution of digital camera images (while conditioned on the metadata characterizing the corresponding camera) and can therefore be used to iteratively process input images so that the input images acquire the characteristics of that distribution.

[0117] Similarly, the image restoration neural network is trained similarly to the denoising diffusion model, but is not used for denoising. In inference, the image restoration neural network is given input images that have the appearance of digital camera images, i.e., the input images resemble images drawn from a distribution of digital camera images, and the image restoration neural network iteratively processes them to generate restored images that have the appearance of the corresponding source camera images, i.e., the restored images resemble images drawn from the distribution of source camera images.

[0118] Thus, for example, if an additional image is acquired by selecting one of the source camera images, processing the additional training examples may include iteratively processing the additional image using an image augmentation neural network while conditioned on metadata characterizing the camera for the additional image to generate an augmented image. As described above, the metadata characterizing the camera for the additional image may be acquired, for example, by random sampling from a learned distribution or by retrieving it from the training data (if available). The augmented image is then iteratively processed using an image restoration neural network to reconstruct a version of the additional image. The image restoration neural network may (but need not) be conditioned on the metadata characterizing the camera for the additional image, i.e., the data used to generate the augmented image. The image augmentation neural network and the image restoration neural network are then jointly trained by updating the image augmentation neural network parameters and the image restoration neural network parameters, as described above, to increase consistency between the additional image and a reconstructed version of the additional image.

[0119] In a particular exemplary implementation, using the image augmentation neural network to generate an augmented image includes determining an initial input image from the additional image and then updating the initial input image in each of a plurality of update iterations. Each update iteration includes processing the input image using the image augmentation neural network while conditioned on metadata, x, characterizing the camera for the additional image to generate a modified input image. Each update iteration, except the last iteration, also includes adding noise to the modified input image to generate an input image for the next iteration. For example, for T iterations t = T, .., 1 and an initial input image y T Using the input image y at iteration t - 1, t-1 teeth,

[0120]

number

[0121] may be determined from z ∼ N(0, I) and a scalar parameter α 1:T is a hyperparameter in the range [0, 1], f(·) is the image augmentation neural network output 116,

[0122]

number

[0123] In a variant, if the noise vector ε is replaced by the image y0 during training, the same approach can be used, but with different hyperparameters.

[0124] The further image may be obtained by selecting one of the digital camera images and selecting the corresponding camera-characterizing metadata as the camera-characterizing metadata for the further image. Processing the further training examples may then include iteratively processing the further image using an image restoration neural network, optionally conditioned on the camera-characterizing metadata for the further image, to generate a restored augmented image. This may employ the same specific exemplary implementation as described above with respect to the image augmentation neural network. The restored image is then iteratively processed using the image augmentation neural network, conditioned on the camera-characterizing metadata for the further image, to reconstruct a version of the further image. The image augmentation neural network and the image restoration neural network are then jointly trained to increase consistency between the further image and the reconstructed version of the further image, as described above. Generally, the further training examples include both the source camera image and the digital camera image.

[0125] Figure 6 is a flow diagram of an exemplary process that may be implemented in a mobile device for processing images from the mobile device so that the images appear to have been captured by a digital camera with particular characteristics. The process uses the image augmentation neural network 110 of the image augmentation system 102 to process the captured images after the neural network has been trained by any of the processes of Figures 3-5. The steps of Figure 6 may be performed by a processor of the mobile device under control of stored instructions.

[0126] In step 600, an image is captured with a camera of a mobile device. The process also obtains data from a user interface of the mobile device defining a set of one or more specified characteristics of the digital camera, for example, one or more characteristics of an exposure triangle of settings including the digital camera's aperture setting, shutter speed setting, and ISO setting. This camera may be referred to as a target camera and may, but does not necessarily, correspond to a real camera. The process then determines values ​​of a conditioning tensor defined by the one or more specified characteristics (step 602).

[0127] The image augmentation neural network 110 then processes the captured image while conditioned on the conditioning tensors to generate an augmented image that has the appearance of an image captured by a digital camera with specified characteristics (step 604). The processing of the captured image by the image augmentation neural network 110 may be performed by a processor of the mobile device, or the processor of the mobile device may communicate with a remote server implementing the image augmentation neural network to process the captured image (in which case the augmented image may be received back from the remote server). The augmented image may be displayed on the mobile device, stored locally or remotely, or transmitted to, for example, another mobile device (step 606).

[0128] Figure 7 is a flow diagram detailing an exemplary process for using the image augmentation neural network 110 to process captured images to generate augmented images after the image augmentation neural network has been trained by the process of Figure 5. The steps of Figure 7 may be performed by a processor of the mobile device or, for example, by a remote server.

[0129] In step 700, the process determines an initial input image for the image augmentation neural network 110 from the captured image. The initial input image may include the captured image, or noise may be added, for example, to attenuate information that may be changed during the augmentation process. The image augmentation neural network then processes the input image (e.g., the initial input image or the modified input image) at the time of the update iteration, conditioned on the conditioning tensor and, optionally, the value of the noise level parameter for the update iteration, to generate a modified input image (step 702), for example, as described above. If the last update iteration has not yet been reached, the process then adds noise to the modified input image (step 704), for example, also as described above, and returns to step 702. If the last update iteration has been reached, for example, after a defined number of iterations, T, no noise is added and the modified input image becomes the output image (step 706).

[0130] The image augmentation neural network 110 used in the example processes of FIGS. 6 and 7 may be implemented on the mobile device or on a server remote from the mobile device.

[0131] As mentioned above, implementations of the systems and methods described herein may be used to process moving images, i.e., video, where one or more of the image augmentation neural network 110, image restoration neural network 120, training image classifier neural network 201, and source image classifier neural network 210 may have 3D neural network inputs and outputs rather than 2D neural network inputs and outputs, where 3D input refers to a time sequence of image frames.

[0132] The processing in the image augmentation neural network 110, the image restoration neural network 120, the training image classifier neural network 201, or the source image classifier neural network 210 may similarly operate on data having a temporal dimension and two spatial dimensions, for example, by performing spatiotemporal convolution or other processing. In some implementations, one or both of the image augmentation neural network 110 and the image restoration neural network 120 are configured to generate a temporal sequence of frames in which subsequent frames are conditioned on earlier frames.

[0133] In some implementations, one or both of the image augmentation neural network 110 and the image restoration neural network 120 have one or more attention neural network layers, e.g., self-attention layers. For example, they may include two (or more) factorized self-attention neural network layers, i.e., factorized self-attention neural network layers each configured to apply an attention mechanism to only a portion of the input image sequence. For example, a first factorized self-attention neural network layer may apply attention only to time-varying features of the input image sequence, and one or more second factorized self-attention neural network layers may apply attention only to spatial features of the image frames of the input image sequence. That is, spatial feature maps may be generated from the image frames and processed separately from temporal feature maps generated from the input image sequence, reducing system memory requirements.

[0134] Similarly, one or both of the image classifier neural networks may include a temporal classifier neural network for classifying based on temporal features of the sequence of image frames and a spatial classifier neural network for classifying based on spatial features of the image frames. In implementations, for computational efficiency, the spatial classifier neural network may be configured to process image frames with reduced temporal resolution (compared to the sequence of input image frames), and the temporal classifier neural network may be configured to process image frames with reduced spatial resolution (compared to the sequence of input image frames).

[0135] If the images are video, a temporal cycle consistency loss may be included when training the image augmentation neural network 110 and the image restoration neural network 120 for consistency when reconstructing further images of that version, as described, for example, in Dwibedi et al., arXiv:1904.07846.

[0136] This specification uses the term "configured" in reference to systems and components of computer programs. One or more computer systems being configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that causes the system to perform that operation or action during operation. One or more computer programs being configured to perform a particular operation or action means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform that operation or action.

[0137] Embodiments and functional operations of the subject matter described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, including the structures disclosed herein and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or to control the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a receiver device suitable for execution by a data processing apparatus.

[0138] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatuses, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus can also be or further include special-purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). Optionally, an apparatus can include, in addition to hardware, code that creates an execution environment for a computer program, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0139] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted, or declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, a single file dedicated to the program in question, or multiple organized files, e.g., files storing one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and connected together by a data communications network.

[0140] The term "database" is used broadly herein to refer to any collection of data, which need not be structured in any particular way, or at all, and which may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different way.

[0141] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines can be installed and running on the same computer or computers.

[0142] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by performing operations on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or a combination of special purpose logic circuitry and one or more programmed computers.

[0143] A computer suitable for running a computer program can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from a read-only memory or a random-access memory, or both. The elements of a computer are a central processing unit for performing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special-purpose logic circuitry. Generally, a computer also includes one or more mass storage devices, e.g., magnetic, magneto-optical, or optical disks, for storing data, or is operatively coupled to receive data from, transfer data to, or both of, such mass storage devices. However, a computer need not have such devices. Additionally, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few.

[0144] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0145] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer can also interact with a user by sending text messages or other forms of messages to a personal device, e.g., a smartphone, running a messaging application, and receiving a reply message from the user in return.

[0146] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units, for example for handling common computationally intensive parts of machine learning training or generation, i.e., inference, workloads.

[0147] The machine learning model can be implemented and deployed using a machine learning framework, for example, the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0148] Embodiments of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., as a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, web browser, or app through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0149] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data, e.g., HTML pages, to a user device for the purpose of, e.g., displaying data to and receiving user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., a result of user interaction, may be received from the device at the server.

[0150] While the specification contains many specific implementation details, these should not be considered limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features that are described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as working in a particular combination, and may even be initially claimed as such, one or more features of a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination or variations of the subcombination.

[0151] Similarly, while operations are illustrated in the figures and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order illustrated or in sequential order, or that all of the illustrated operations be performed, to achieve desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the division of various system modules and components in the above-described embodiments should not be understood as requiring such division in all embodiments, and it should be understood that the described program components and systems generally can be integrated together in a single software product or packaged in multiple software products.

[0152] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0153] 100 mobile devices 100a display screen 101 processors 102 Image Augmentation System 103 Communication Subsystem 104 Camera 105 Non-volatile Storage 105a Image Extension Code 105b Parameters 105c Image Storage 105d Operating System Code 106 User Interface 108 Conditioning Tensor Decision Subsystem 110 Image Augmentation Neural Network 112 Image Augmentation Neural Network Input 114 Image Augmentation Conditioning Input 116 Image Augmentation Neural Network Output 120 Image Restoration Neural Network 122 Image Restoration Neural Network Input 124 Image Recovery Conditioning Input 126 Image Restoration Neural Network Output 130 Training Engine 140 training data 142 Metadata Reconstruction Neural Network 200 Image Augmentation Neural Network Training System 201 Training Image Classifier Neural Network 202 Training image classifier input 204 Training Image Classifier Conditioning Input 206 Training image classifier output 210 Source Image Classifier Neural Network 212 Source image classifier input 214 Source Image Classifier Conditioning Input 216 Source image classifier output

Claims

1. Capturing an image with a camera of a mobile device; obtaining user input data from a user interface of the mobile device defining a set of one or more specified characteristics of a digital camera, the set of one or more specified characteristics defining one or more characteristics of an exposure triangle of settings including an aperture setting, a shutter speed setting, and an ISO setting of the digital camera; determining conditioning tensors characteristic of the one or more specified properties from the user input data; processing the image captured by the camera of the mobile device using a trained image augmentation neural network while conditioned on the conditioning tensor to generate an augmented image having the appearance of the image captured by the digital camera with the specified characteristics; displaying the augmented image for a user on the mobile device, storing the augmented image, or transmitting the augmented image; A computer-implemented method comprising:

2. The method of claim 1 , wherein the set of one or more specified characteristics defined by the user input data includes at least two settings of the exposure triangle of settings.

3. The method of claim 2 , wherein the set of one or more specified characteristics defined by the user input data includes three settings of the exposure triangle of settings.

4. 4. The method of claim 1, 2, or 3, wherein the set of one or more specified characteristics defined by the user input data includes an exposure compensation setting to allow the enhanced image to be underexposed or overexposed.

5. 5. The method of claim 1, wherein the digital camera is a camera including a camera body and an interchangeable lens, and the specified characteristics of the camera defined by the user input data include a body type of the camera body or a lens type of the interchangeable lens.

6. The method of claim 5 , wherein the specified characteristics of the camera defined by the user input data include a make or model of the body type of the camera body or the lens type of the interchangeable lens.

7. The method of claim 5 or 6, wherein the specified characteristics of the camera defined by the user input data include a focal length of the interchangeable lens.

8. 8. The method of claim 1, wherein the augmented image has a higher image resolution than a resolution of the image captured by the camera of the mobile device, and wherein using the trained image augmentation neural network to generate the augmented image comprises using the trained image augmentation neural network to add image detail to the image captured by the camera of the mobile device.

9. The method of claim 1 , wherein the digital camera is a digital SLR (DSLR) camera.

10. The method according to any one of claims 1 to 8, wherein the digital camera is a mirrorless interchangeable lens camera (MILC).

11. 10. The method of claim 1, wherein the trained image augmentation neural network is trained while being conditioned on conditioning tensors defined by Exchangeable Image File (EXIF) data.

12. 12. The method of any one of claims 1 to 11, wherein the trained image augmentation neural network was trained using an objective that does not require that an image captured by a camera of the mobile device be paired with a corresponding augmented image.

13. 13. The method of any one of claims 1 to 12, comprising performing the processing using the trained image augmentation neural network on or controlled by the mobile device.

14. 14. The method of claim 1, wherein the user interface is a graphical user interface that simulates the appearance of the digital camera with settings that enable the user to define the characteristics of the exposure triangle.

15. The step of processing the image further comprises: determining an initial input image from the captured image, and in each of a plurality of update iterations, processing the input image at the update iteration using the image augmentation neural network while conditioned on the conditioning tensor to generate a modified input image; at each update iteration except the last update iteration, adding noise to the modified input image to generate an input image for a next update iteration; The method according to claim 1 , further comprising updating the initial input image by:

16. 1. A mobile device comprising: at least one processor; and at least one storage device communicatively coupled to the at least one processor, the at least one storage device providing to the at least one processor, when executed by the at least one processor: Capturing an image with a camera of a mobile device; obtaining, from a user interface of the mobile device, user input data defining a set of one or more specified characteristics of a digital camera, the set of one or more specified characteristics defining one or more characteristics of an exposure triangle of settings including an aperture setting, a shutter speed setting, and an ISO setting of the digital camera; determining conditioning tensors characteristic of the one or more specified properties from the user input data; processing the image captured by the camera of the mobile device using the trained image augmentation neural network while conditioned on the conditioning tensor to generate an augmented image having the appearance of the image captured by the digital camera with the specified characteristics; displaying the augmented image for a user on the mobile device, storing the augmented image, or transmitting the augmented image; A mobile device storing instructions for performing operations to perform the above.

17. using the trained image augmentation neural network to process the images captured by the camera of the mobile device; determining an initial input image from the captured image, and in each of a plurality of update iterations, processing the input image at the update iteration using the image augmentation neural network while conditioned on the conditioning tensor to generate a modified input image; at each update iteration except the last update iteration, adding noise to the modified input image to generate an input image for a next update iteration; 17. The mobile device of claim 16, further comprising updating the initial input image by

18. 1. A computer-implemented method of training an image augmentation neural network to provide a photographic image, the image augmentation neural network having a plurality of image augmentation neural network parameters and configured to receive an input image from a mobile device and process the input image in response to the image augmentation conditioning inputs to generate an augmented image that appears to be an image captured by a digital camera having characteristics defined by the image augmentation conditioning inputs, the method comprising: For each of the first plurality of training examples, the second plurality of training examples, and the further plurality of training examples, obtaining the first training examples, the second training examples, and the further training examples from a training dataset including a set of source camera images captured by one or more source cameras of one or more mobile devices, a set of digital camera images captured by one or more digital cameras, and camera characterizing metadata for each of the digital camera images, wherein the camera characterizing metadata for the digital camera images defines one or more characteristics of an exposure triangle of settings including an aperture setting, a shutter speed setting, and an ISO setting of the digital camera used to capture the images; the first training examples include selected source camera images, the second training examples include selected digital camera images, and the further training examples include further images that are either one of the source camera images or one of the digital camera images; training the image augmentation neural network using one of the first training examples and the second training examples to generate a first augmented image while conditioned on metadata characterizing a camera for generating the first augmented image; training an image restoration neural network having a plurality of image restoration neural network parameters using another of the first training examples and the second training examples to generate a first restored image; sequentially processing the further images using both the image augmentation neural network and the image restoration neural network to reconstruct a version of the further images, and updating the image augmentation neural network parameters and the image restoration neural network parameters to increase consistency between the further images and the reconstructed versions of the further images. A method comprising:

19. 20. The method of claim 18, wherein obtaining the training data set comprises processing digital camera images using a trained metadata reconstruction neural network to reconstruct missing camera-characterizing metadata for the digital camera images when the digital camera images are missing camera-characterizing metadata.

20. Obtaining metadata characterizing the camera for the further image, determining a joint distribution of the characteristics defined by metadata characterizing the cameras in the training data set; obtaining metadata characterizing the camera for the further image by sampling metadata characterizing the camera from the joint distribution; 20. The method of claim 18 or 19, comprising:

21. updating the image augmentation neural network parameters and the image restoration neural network parameters to increase consistency between the further image and the reconstructed version of the further image; 21. The method of claim 18, 19 or 20, comprising updating the image augmentation neural network parameters and the image restoration neural network parameters based on a gradient of an objective function that depends on a difference between the further image and the reconstructed version of the further image.

22. 22. The method of claim 21, wherein the difference comprises a SSIM (Structural Similarity Index Measure) index of the reconstructed version of the further image, calculated using the further image as a reference.

23. 23. The method of any one of claims 18 to 22, wherein the image augmentation neural network has a U-net architecture with skip connections.

24. 24. The method of any one of claims 18 to 23, wherein the source camera images include images captured by one or more mobile phones and the digital camera images include images captured by one or more DSLR cameras or MILC cameras.

25. The method further comprises, for each of the plurality of third training examples and the plurality of fourth training examples: obtaining the third training example comprising one of the source camera images; processing the third training examples using a source image classifier neural network having a plurality of source image classifier neural network parameters to generate a first prediction of whether the third training examples are genuine source camera images; updating the source image classifier neural network parameters to reduce an error in the first prediction; obtaining the fourth training example including one of the digital camera images and corresponding metadata characterizing the camera; processing the fourth training example using a training image classifier neural network having a plurality of training image classifier neural network parameters, the training image classifier neural network being conditioned on metadata relating to the digital camera image characterizing the camera, to generate a second prediction of whether the fourth training example is a genuine digital camera image; updating the training image classifier neural network parameters to reduce error in the second prediction; and wherein the method further comprises: processing the selected source camera image using the image augmentation neural network while conditioned on metadata characterizing the camera for generating the first augmented image to generate the first augmented image; and processing the first augmented image using the trained image classifier neural network conditioned on metadata characterizing the camera used to generate the first augmented image to generate a third prediction of whether the first augmented image is a genuine digital camera image; and updating the image augmentation neural network parameters to increase the error in the third prediction. training the image augmentation neural network using the first training examples by processing the selected digital camera images using the image restoration neural network to generate the first restored image; and processing the first recovered image using the source image classifier neural network to generate a fourth prediction of whether the first recovered image is a genuine source camera image; and updating said image restoration neural network parameters to increase the error in said fourth prediction. training the image restoration neural network using the second training examples by 25. The method of any one of claims 18 to 24, further comprising:

26. The step of obtaining the second training examples further comprises obtaining metadata characterizing the camera for the selected digital camera images, the method comprising:

26. The method of claim 25, further comprising processing the selected digital camera images using the image restoration neural network conditioned on metadata characterizing the camera for the selected digital camera images.

27. obtaining the further training examples includes selecting one of the source camera images for obtaining the further image; and obtaining metadata characterizing a camera for the further image; The step of processing the further training examples further comprises: processing the further image using the image augmentation neural network while conditioned on metadata characterizing the camera for the further image to generate an augmented further image; processing the augmented further image using the image restoration neural network to reconstruct the further image of the version; 27. The method of claim 25 or 26, comprising:

28. obtaining the further training examples includes selecting one of the digital camera images and metadata characterizing the corresponding camera to obtain the further image and metadata characterizing the camera of the further image; The step of processing the further training examples further comprises: processing the further image using the image restoration neural network to generate a restored further image; processing the recovered further images using the image augmentation neural network while conditioned on metadata characterizing a camera of the further images to recreate the version of the further images; 28. The method of any one of claims 25 to 27, comprising:

29. said source image classifier neural network comprising: processing the source image classifier input image using a first source image classifier to generate a first intermediate source image prediction of whether the source image classifier input image is a genuine source camera image; processing each of a plurality of source image patches using a second source image classifier to generate a second intermediate source image prediction of whether the source image classifier input image is a genuine source camera image, the plurality of source image patches dividing the source image classifier input image into tiles; combining the first intermediate source image prediction and the second intermediate source image prediction to generate a prediction of whether the source image classifier input image is a genuine source camera image; 29. The method of claim 25, configured to process the source image classifier input image to generate the prediction of whether the source image classifier input image is a genuine source camera image by:

30. The training image classifier neural network comprises: processing the training image classifier input images using a first training image classifier to generate a first intermediate training image prediction of whether the training image classifier input images are genuine digital camera images; processing each of a plurality of training image patches using a second training image classifier to generate a second intermediate training image prediction of whether the training image classifier input image is a genuine digital camera image, the plurality of training image patches dividing the training image classifier input image into tiles; combining the first intermediate training image prediction and the second intermediate training image prediction to generate a prediction of whether the training image classifier input image is a genuine digital camera image; 30. The method of claim 25, configured to process the training image classifier input images to generate the prediction of whether the training image classifier input images are genuine digital camera images by:

31. said source image classifier neural network comprising: processing each of a plurality of source image patches to generate a set of source image patch encodings, the plurality of source image patches dividing a source image classifier input image into tiles; processing the set of source image patch encodings by applying a self-attention mechanism to the set of source image patch encodings to generate a set of transformed source image patch encodings; combining the transformed source image patch encodings to generate a prediction of whether the source image classifier input image is a genuine source camera image; 31. The method of claim 25, configured to process the source image classifier input image to generate the prediction of whether the source image classifier input image is a genuine source camera image by:

32. The training image classifier neural network comprises: processing each of a plurality of training image patches to generate a set of training image patch encodings, the plurality of training image patches dividing a training image classifier input image into tiles; processing the set of training image patch encodings by applying a self-attention mechanism to the set of training image patch encodings to generate a set of transformed training image patch encodings; combining the transformed training image patch encodings to generate a prediction of whether the training image classifier input image is a genuine digital camera image; and 32. The method of claim 25, configured to process the training image classifier input images to generate the prediction of whether the training image classifier input images are genuine digital camera images by:

33. The step of obtaining the second training examples further comprises obtaining metadata characterizing the camera for the selected digital camera images, the method comprising: training the image augmentation neural network using the second training examples by training the image augmentation neural network using the second training examples to denoise noisy versions of the selected digital camera images while conditioned on metadata characterizing the camera for the selected digital camera images; training the image restoration neural network using the first training examples by training the image restoration neural network to denoise the noisy versions of the selected source camera images; 25. The method of any one of claims 18 to 24, further comprising:

34. obtaining the first training examples further comprises obtaining metadata characterizing a camera for the selected source camera images; 34. The method of claim 33, wherein training the image restoration neural network to denoise the noisy version of the selected source camera image is conditioned on metadata characterizing the camera for the selected source camera image.

35. obtaining the further training examples includes selecting one of the source camera images for obtaining the further image; and obtaining metadata characterizing a camera for the further image; The step of processing the further training examples further comprises: processing the further image using the image augmentation neural network while conditioned on metadata characterizing the camera for the further image to generate an augmented image; processing the augmented image using the image restoration neural network to reconstruct the further image of the version; 35. The method of claim 33 or 34, comprising:

36. 36. The method of claim 35, comprising processing the augmented image using the image recovery neural network while conditioned on metadata characterizing the camera for the further images to reconstruct the further images of the version.

37. obtaining the further training examples includes selecting one of the digital camera images and corresponding metadata characterizing the camera to obtain the further image and metadata characterizing the camera of the further image; The step of processing the further training examples further comprises: processing the further image using the image restoration neural network to generate a restored image; processing the recovered images using the image augmentation neural network while conditioned on metadata characterizing a camera of the further images to recreate the version of the further images; 37. The method of any one of claims 33 to 36, comprising:

38. 38. The method of claim 37, comprising processing the further images using the image restoration neural network while conditioned on metadata characterizing a camera of the further images to generate the restored image.

39. training the image augmentation neural network, generating a noisy version of the selected digital camera image; processing the noisy versions of the selected digital camera images using the image augmentation neural network conditioned on metadata characterizing the camera for the selected digital camera images to generate an image augmentation neural network output; determining a value of an image augmentation objective function that depends on a difference between the image augmentation neural network output and either i) the selected digital camera image, or ii) a noise vector representing noise added to the selected digital camera image to generate the noisy version of the selected digital camera image; determining a gradient of the image augmentation objective function with respect to the image augmentation neural network parameters; updating the image augmentation neural network parameters using the gradient of the image augmentation objective function; 39. The method of any one of claims 33 to 38, comprising:

40. training the image restoration neural network, generating a noisy version of the selected source camera image; processing the noisy versions of the selected source camera images using the image augmentation neural network to generate an image restoration neural network output; determining a value of an image restoration objective function that depends on the difference between the image restoration neural network output and either i) the selected source camera image, or ii) a noise vector representing noise added to the selected source camera image to generate the noisy version of the selected source camera image; determining a gradient of the image restoration objective function with respect to the image restoration neural network parameters; updating the image restoration neural network parameters using the gradient of the image restoration objective function; 40. The method of any one of claims 33 to 39, comprising:

41. sequentially processing the further training examples using both the image augmentation neural network and the image restoration neural network to reconstruct the further images of the version; iteratively processing the further images using one of the image augmentation neural network and the image restoration neural network to progressively create intermediate images; and then iteratively processing the intermediate images using the other of the image augmentation neural network and the image restoration neural network to progressively reconstruct the further images of the version; Including, 41. The method of any one of claims 33 to 40, wherein the image augmentation neural network is conditioned on metadata characterizing a camera for augmented images progressively produced by the image augmentation neural network.

42. 1. A method for providing a mobile device with the capability to generate images that look like images captured by a digital camera, comprising: training an image augmentation neural network according to a method according to any one of claims 18 to 41 to obtain a trained image augmentation neural network; providing the mobile device with access to the trained image augmentation neural network; providing a user interface to enable a user of the mobile device to input a set of one or more user-defined characteristics of the digital camera; enabling the mobile device to pass i) the image captured by the camera of the mobile device, and ii) the data representing the user-defined characteristics of the digital camera to the trained image augmentation neural network, to process an image captured by a camera of the mobile device using the trained image augmentation neural network while conditioned on the data representing the user-defined characteristics of the digital camera to generate an augmented image for user viewing, transmission, or storage that appears to be an image captured by a digital camera with the user-defined characteristics; A method comprising:

Images