Image Extension Neural Network

Abstract

Methods, systems, and apparatus for generating realistic image extensions, including computer programs encoded on computer storage media. In one aspect, a method includes providing an input including a provided image to a generative neural network having a plurality of generative neural network parameters. The generative neural network processes the input according to trained values ​​of the plurality of generative neural network parameters to generate an extended image. The extended image has (i) more rows, more columns, or more rows and more columns than the provided image, and (ii) is predicted to be a realistic extension of the provided image. The generative neural network is trained using an adversarial loss objective function.

Description

Technical Field

[0001] This specification relates to processing images using machine learning models. Background Art

[0002] A machine learning model receives input and generates an output, such as a predicted output, based on the received input. Some machine learning models are parametric models and generate output based on the received input and the values ​​of the model parameters.

[0003] Some machine learning models are deep models that employ multi-layer models to generate outputs for received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a transformation to the received input to generate an output. Summary of the invention

[0004] This specification describes a system implemented as a computer program on one or more computers in one or more locations that performs image expansion.

[0005] According to a first aspect, a method performed by one or more data processing devices is provided, the method comprising: providing an input including a provided image to a generative neural network having a plurality of generative neural network parameters, wherein: the generative neural network processes the input according to trained values ​​of the plurality of generative neural network parameters to generate an extended image; the extended image has (i) more rows, more columns, or more rows and more columns than the provided image, and (ii) is predicted to be a true extension of the provided image; and the generative neural network has been trained using an adversarial loss objective function.

[0006] In this way, the additional rows and / or columns provide an extension beyond one or more original boundaries of the provided image and provide a realistic extension of the prediction of the provided image, such as, for example, preserving high-level semantic characteristics and low-level structure and texture of the provided image.

[0007] The method may include receiving a request for an image to be presented in a tile; determining that the provided image is responsive to the request; and after generating an extended image, providing the extended image in response to the request.

[0008] The method may include determining that the provided image qualifies for image expansion based on a size of the block being different than a size of the provided image, and in response, providing an input including the provided image to a generative neural network.

[0009] The size of the block may specify the aspect ratio of the block.

[0010] The request may specify additional elements to be presented with the image in the tile, and providing the extended image in response to the request may include overlaying the additional elements over the extended portion of the extended image.

[0011] The block can be presented next to search results or on a third-party web page.

[0012] A generative neural network can include multiple convolutional neural network layers.

[0013] A generative neural network can include multiple skip connections.

[0014] A generative neural network can include multiple instance normalization layers.

[0015] An input to the generative neural network may include: a baseline image having the same number of rows and columns as an extended image, wherein the baseline image includes: (i) a first portion corresponding to a provided image, and (ii) a second portion having default pixel values; and a mask image having the same number of rows and columns as the extended image, wherein the mask image identifies the first portion and the second portion of the baseline image.

[0016] Pixels in the mask image corresponding to a first portion of the baseline image may each have a first pixel value, and pixels in the mask image corresponding to a second portion of the baseline image may each have a second pixel value different from the first pixel value.

[0017] The generative neural network can be jointly trained with a discriminative neural network having a plurality of discriminative neural network parameters, the discriminative neural network being configured to process a given image to generate a discriminative output representing a likelihood that the given image was generated using the generative neural network.

[0018] Training a generative neural network using an adversarial loss objective function may include: using a generative neural network and processing a training input including a training image according to current values ​​of generative neural network parameters to generate a training extended image that extends the training image; generating a discriminative neural network input based on the training extended image; using a discriminative neural network and processing the discriminative neural network input according to current values ​​of the discriminative neural network parameters to generate a discriminative output representing the likelihood that the discriminative neural network input was generated using the generative neural network; and adjusting the current values ​​of the generative neural network parameters based on the adversarial loss objective function, wherein the adversarial loss objective function depends on the discriminative output representing the likelihood that the discriminative neural network input was generated using the generative neural network.

[0019] Generating a discriminative neural network input based on the training extended image may include overwriting a portion of the training extended image corresponding to the training image with the training image.

[0020] The method may include adjusting current values ​​of parameters of a generative neural network based on a reconstruction loss objective function characterizing similarity between a training extended image and a target image, wherein the training image is a cropped representation of the target image.

[0021] The discriminative neural network can be conditioned on a semantic feature representation of a target image, where the training images are cropped representations of the target image.

[0022] By processing the target image, the semantic feature representation of the target image is determined using the intermediate output of the classification neural network.

[0023] The discriminative output can be based on: (i) the output of the last layer of the discriminative neural network, and (ii) a similarity measure between the intermediate output of the discriminative neural network and the semantic feature representation of the target image.

[0024] According to another aspect, a system is provided, the system comprising one or more computers and one or more storage devices storing instructions which, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding method of the first aspect.

[0025] According to another aspect, one or more computer storage media storing instructions are provided that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of the first aspect.

[0026] Optional features of one aspect may be combined with another aspect where appropriate.Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0027] The image expansion system described in this specification can use a generative neural network to generate an extended image (e.g., an extended image that extends an input image beyond its original boundaries). The generative neural network is jointly trained with a "discriminative" neural network, which is configured to process the input image to generate an output that defines a prediction of whether the input image is (i) a "real" image or (ii) an extended image generated by the generative neural network. In order to facilitate the joint training of the generative neural network and the discriminative neural network, the system can adjust the discriminative neural network based on semantic features corresponding to the input image processed by the discriminative neural network. Adjusting the discriminative neural network in this way can enable the system to be trained to an acceptable performance level with fewer training iterations, thereby causing the system to consume less computing resources (e.g., memory and computing power) during training than some conventional systems.

[0028] Some conventional in-painting systems "fill in" a portion of an image that is enclosed in all directions with the original image data. In contrast, the image extension system described in the present specification extends an image beyond its original boundaries to generate an extended image that extends the high-level semantic properties of the image as well as the low-level image structure and texture. In some cases, directly applying a conventional in-painting system to perform an image extension task results in an extended image that is lower in quality than an extended image generated by the image extension system described in the present specification. For example, a conventional in-painting system may generate an extended image with blurred or repeated pixels and inconsistent semantics, while an extended image created by the currently described image extension system will not have the same blurred, repeated pixels, or inconsistent semantic properties.

[0029] The image expansion system described in this specification generates an expanded image that can be used for any of a variety of applications (e.g., virtual reality applications, computational photography applications, and digital component distribution applications). In a digital component distribution application, the digital component distribution system sends a digital component to be presented in a block, for example, next to a search result or on a third-party web page. In some cases, the image included in the digital component may have a size that prevents the image from filling a defined block, for example, because the aspect ratio of the image and the block is different. In this case, the image expansion system described in this specification can be used to adjust the image size so that the image can fill the block, thereby enabling more efficient use of resources (particularly more efficient use of the space available in the block) and preventing the appearance of blank spots and / or distorted images in the user interface. In other words, the system described in this application not only "fills" the image, but also extends the image beyond its boundaries, which enables the image to be modified to fit multiple blocks of different sizes even when only a single image size is available. This reduces the number of images that need to be stored because a single image can be modified to fit blocks of various sizes while retaining the characteristics of the original image. The image expansion system can change the aspect ratio of the image by stretching or compressing the image along a specific dimension to make the image fill the block to avoid the need for distorted images. Furthermore, in digital component distribution applications, some digital component vendors may require that images included in the digital component be presented without distortion (eg, by stretching or compressing along a particular dimension).

[0030] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 An example image expansion system is shown.

[0032] Figure 2Illustration of an example data flow for jointly training a generative neural network with a discriminative neural network using an adversarial loss objective function.

[0033] Figure 3 is an illustration of an example of an expanded image generated by an image expansion system.

[0034] Figure 4 is a block diagram of an example environment in which a digital component distribution system uses an image extension system.

[0035] Figure 5 A digital component is shown being displayed on a screen of a user device, wherein an image included in the digital component is expanded using an image expansion system.

[0036] Figure 6 A digital component is shown being displayed in a "banner" block, wherein an image included in the digital component is expanded using an image expansion system.

[0037] Figure 7 is an illustration of an example user interface that may be presented to an entity providing a digital component for delivery by a digital component distribution system.

[0038] Figure 8 is a flow diagram of an example process for providing an extended image in response to a request.

[0039] Fig. 9 is a block diagram of an example computer system.

[0040] Like reference numbers and designations throughout the various drawings represent like elements. DETAILED DESCRIPTION

[0041] This specification describes techniques for processing an image to generate an "extended" image that realistically extends the image beyond its original boundaries, e.g., extending the image beyond its original boundaries while preserving the high-level semantic properties and low-level structure and texture of the image. Such processing can be used for image enhancement. For example, if the size of an image does not match the block in which the image is to be placed, the image size can be extended to match the block while preserving the content characteristics within the original image. In another example, if an image is damaged, such as a portion of the image is missing, aspects of the present disclosure can be used to automatically generate the missing portion. For example, a captured version of a damaged image that was captured incorrectly or incomplete can be regenerated using the techniques discussed herein.

[0042] In order to expand an image, the image expansion system described in this specification processes the image using a generative neural network, which is trained using an adversarial loss objective function to generate a corresponding expanded image. The adversarial loss objective function evaluates the expanded image generated using the generative neural network based on the output generated by the discriminative neural network by processing the expanded image. The discriminative neural network is trained to predict whether the input image is (i) a "real" (i.e., natural) image, or (ii) an expanded image generated using the generative neural network. Jointly training the generative neural network and the discriminative neural network facilitates the image expansion system to generate an expanded image that is difficult to distinguish from a real image.

[0043] The image expansion system may be used in any of a variety of applications, such as resizing an image included in a digital component to be presented in defined blocks of various sizes so that the image fills the block without distortion.

[0044] As used throughout this document, the phrase digital component refers to a discrete unit of digital content or digital information, which may include one or more of the following: for example, an image, a video clip, an audio clip, a multimedia clip, a text segment, or a uniform resource locator (URL). A digital component may be electronically stored in a physical memory device as a single file or a collection of files, and a digital component may take the form of a video file, an audio file, a multimedia file, an image file, or a text file, and include streaming video, streaming audio, social network posts, blog posts, and / or advertising information, whereby an advertisement is a type of digital component. Typically, a digital component is defined (or provided) by a single provider or source (e.g., an advertiser, publisher, or other content provider), but a digital component provided from one source may be enhanced with digital content from another source (e.g., weather information, real-time event information, or other information obtained from another source).

[0045] These and other features are described in greater detail below.

[0046] Figure 1 An example image expansion system 100 is shown. The image expansion system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0047] Image expansion system 100 is configured to process image 102 to generate an expanded image 104 that actually expands image 102 beyond its original boundaries. More specifically, a two-dimensional (2D) pixel array representing expanded image 104 typically has more rows, more columns, or more rows and more columns than image 102. Figure 1, the expanded image 104 has one or more additional columns 106 on the right side thereof compared to the original image 102. In a specific example, the image 102 may be 100×100 in size, while the expanded image 104 may be 100×120 in size. In another specific example, the image 102 may be 80×120 in size, while the expanded image may be 90×125 in size. The units of the sizes may be any suitable units, such as the number of pixels. Figure 3 An example of an extended image generated using system 100 is shown in FIG.

[0048] In some cases, the image expansion system 100 may uniformly reduce or expand the original image 102 (i.e., while maintaining its aspect ratio and thereby avoiding distortion) before generating the expanded image 104. In a specific example, to generate an output image of size 50×120 from an input image of size 100×200, the image expansion system may first uniformly reduce the input image to size 50×100, and then generate an expanded image of size 50×120 from the reduced image of size 50×100.

[0049] Image 102 (and extended image 104) may be represented in any suitable format, for example, as a 2D array of pixels, each pixel being associated with a corresponding grayscale intensity or with a color vector representing the pixel color in a suitable color space. The color space may be, for example, a red-green-blue (RGB) color space or a CIELAB color space.

[0050] To generate extended image 104, system 100 determines generative network input 108 from image 102 and processes generative network input 108 using generative neural network 110. Several examples of generative network inputs are as follows.

[0051] In one example, the generative network input 108 includes a baseline image 112 and a mask image 114, both of which have the same size (i.e., number of rows and columns) as the extended image to be generated by the generative neural network 110. The baseline image 112 includes: (i) a first portion 112-A corresponding to the image 102, and (ii) a second portion 112-B having a default pixel value (e.g., a black or white pixel value). That is, the baseline image 112 includes the image 102, and can be understood as extending the image 102 by one or more rows or columns of the default pixel value. The mask image 114 identifies which portions of the baseline image correspond to: (i) the image 102, and (ii) the default pixel value. For example, pixels in the mask image 114 corresponding to a true subset of the baseline image 112 that matches the image 102 can have a first value (e.g., value 0), and the remaining pixels in the mask image 114 can have a second value (e.g., value 1). In this example, the baseline image 112 and the mask image 114 may be spatially concatenated channel-by-channel to form the generative network input 108 .

[0052] In another example, the generative network input 108 may include only the baseline image 112 , ie, without the mask image 114 .

[0053] In another example, generative network input 108 may correspond directly to image 102 , ie, without including baseline image 112 or mask image 114 .

[0054] The generative neural network 110 is configured to process the generative network input 108 according to trained values ​​of a set of generative neural network parameters to generate the extended image 104. The generative neural network 110 typically has a convolutional neural network architecture, i.e., a neural network architecture that includes one or more convolutional neural network layers (and optionally other suitable types of neural network layers).

[0055] The adversarial training system 116 is configured to determine trained values ​​of the generative neural network parameters that enable the generative neural network 110 to generate the realistic extended image 104. The adversarial training system 116 jointly trains the generative neural network 110 together with the discriminative neural network using an adversarial loss objective function, which will refer to Figure 2 Describe in more detail.

[0056] In some cases, the system 100 can create a "panoramic" extended image by repeatedly extending the original image. For example, the system 100 can extend the original image to generate a first extended image, then extend a cropped portion of the first extended image to generate a second extended image, and so on. Finally, the system can cascade the extended images to generate a panoramic image that significantly extends the original image, for example, by a factor of 2×, 3×, 5×, or any other suitable factor.

[0057] The expanded image 104 generated by the system 100 can be used in any of a variety of applications. The following are a few examples.

[0058] In one example, in virtual reality applications, it is sometimes necessary to simulate an image captured from a camera orientation different from the camera orientation actually used to capture the original image. The image expansion system 100 can be used to simulate an image by filling in the content outside the boundaries of the original image.

[0059] In another example, in a panoramic stitching application (e.g., where multiple images are "stitched" together to generate a single combined image), some conventional techniques require cropping jagged edges of the stitched projections to achieve a rectangular panorama. The image expansion system 100 can instead avoid the need to crop jagged edges by generating an expanded image that fills the gaps between the jagged edges, thereby achieving more efficient panoramic stitching.

[0060] In another example, the image expansion system 100 can be used to enable a video shot with a specific aspect ratio to be displayed on a screen with a different aspect ratio without resorting to stretching or cropping. In particular, the image expansion system 100 can be used to process each video frame in the video to generate a corresponding expanded video frame, for example, having the same aspect ratio as the screen on which the video is to be displayed.

[0061] In another example, a digital component distribution system may use the image expansion system 100 to expand an image to be displayed in a defined block next to a search result or on a third-party website so that the image fills the block without resorting to stretching or cropping. Figure 4 The use of image expansion system 100 as part of a digital component distribution system is described in greater detail.

[0062] Figure 2 2 is an example data flow 200 illustrating operations that may be performed by the adversarial training system 116 for jointly training the generative neural network 110 with the discriminative neural network 202 using an adversarial loss objective function 204. In general, the generative neural network 110 is trained to generate an extended image that realistically extends the input image, and the discriminative neural network 202 is trained to generate a discriminative output (i.e., different from a natural image) that characterizes the likelihood that the input image was generated using the generative neural network 110. Next, the training of the generative neural network 110 will be described in detail, followed by the training of the discriminative neural network 202.

[0063] The generative neural network 110 is trained through multiple training iterations, and at each training iteration, the current parameter values ​​of the generative neural network parameters are adjusted based on the current batch (i.e., set) of training examples. Each training example includes: (i) a training generative network input, and (ii) a target expanded image that should be generated by the generative neural network 110 by processing the corresponding training generative network input.

[0064] The target extended image 206 included in the training examples is typically a real (ie, natural) image. The training generative network input corresponding to the target extended image 206 can be represented in any of a variety of ways, such as reference Figure 1 In one example, the training generative network input corresponding to the target extended image 206 includes a baseline image 208 and a mask image 210. The baseline image 208 is generated by "masking" a portion of the target extended image 206 (e.g., by setting the pixel values ​​in a portion of the target extended image 206 to a default value (e.g., a value of 0)). The mask image 210 identifies the portion of the target extended image 206 that has been masked in the baseline image. The portion of the target extended image included in the training generative network input may be referred to as a "training image" in this document.

[0065] Typically, the current batch of training examples may include multiple training examples. However, for convenience, the following description will refer to a specific training example in the current batch.

[0066] The generative neural network 110 processes the training generative network input included in the training example to generate the corresponding training extended image 212. After generating the training extended image 212, the adversarial training system 116 processes the training extended image 212 using the discriminative neural network 202 to generate a discriminative output that characterizes the likelihood that the training extended image is generated by the generative neural network 110. Therefore, the discriminative output characterizes whether the training extended image 212 generated by the generative neural network 110 is a real image that realistically extends the training image. As will be described in more detail below, the adversarial loss objective function 204 depends on the discriminative output generated by the discriminative neural network 202.

[0067] In some cases, rather than directly processing the training extended image 212, the adversarial training system 116 first overwrites the training image onto the corresponding portion of the training extended image 212. That is, the adversarial training system 116 overwrites the training image onto the portion of the training extended image 212 that should match the training image, without modifying the portion of the training extended image 212 that extends beyond the boundary of the training image. Modifying the training extended image 212 in this manner facilitates the discriminative output to characterize not only whether the training extended image is a real image in nature, but also whether the training extended image is a real extension of the original training image.

[0068] Before providing the training extended image 212 to the discriminative neural network 202 , the mask image 210 may be spatially concatenated to the training extended image 212 channel by channel.

[0069] In addition to processing the training extended image 212 (or input based on the training extended image 212), the discriminative neural network 202 can be adjusted according to the semantic feature representation of the target extended image 206 of the training example. That is, the discriminative output generated by the discriminative neural network 202 for the training extended image 212 can depend on the semantic feature representation of the target extended image 206.

[0070] The semantic feature representation of the target extended image 206 refers to an ordered set of values, such as a vector or matrix of values, which implicitly or explicitly characterizes the content of the target extended image 206. The adversarial training system 116 determines the semantic feature representation of the target extended image 206 as an intermediate output generated by the pre-trained image processing neural network by processing the target extended image 206. The intermediate output of the neural network refers to the output generated by one or more intermediate layers of the neural network (i.e., the layers after the input layer but before the output layer).

[0071] The image processing neural network can have any suitable neural network architecture (e.g., the InceptionV3 architecture) and can be pre-trained to perform any of a variety of image processing tasks, such as a classification task or a regression task. In a specific example, the image processing neural network can be pre-trained to perform a classification task by generating an output representing the corresponding likelihood that the input image depicts objects from a predetermined number of object categories (e.g., a person, a vehicle, a bicycle, etc.). In another specific example, the image processing neural network can be pre-trained to perform a regression task by generating an output representing the location of a bounding box predicted to surround an object depicted in the input image. An image processing neural network is "pre-trained" if the network has been previously trained to perform an image processing task.

[0072] In some cases, the adversarial training system 116 uses the normalization engine 214 to normalize the semantic representation of the target extended image before using the semantic representation of the target extended image to adjust the discriminative neural network 202. For example, the normalization engine 214 may normalize the semantic representation C norm Determined as:

[0073]

[0074] Among them, C is the unnormalized semantic representation, is the expected value of the semantic representation of an image (e.g., from a set of training images), and |·| 2 L 2 In addition, the adversarial training system 116 can process the normalized semantic representation using one or more neural network layers 216 (e.g., fully connected layers) that are jointly trained with the generative and discriminative neural networks before using the semantic representation to adjust the discriminative neural network.

[0075] The discriminative neural network 202 may be adjusted according to the semantic representation of the target extended image in any of a variety of ways. In one example, the adversarial training system 116 may provide the semantic representation of the target extended image as an additional input to the discriminative neural network, for example, to an input layer or an intermediate layer of the discriminative neural network. In another example, the adversarial training system 116 may determine the discriminative output based at least in part on a similarity measure between (i) the semantic representation of the target extended image and (ii) the intermediate output of the discriminative neural network, for example, the discriminative output D may be given by:

[0076] D=D O + <D N ,S> (2)

[0077] Among them, D O is the scalar output of the last neural network layer of the discriminative neural network, D N is an intermediate output of the discriminative neural network (e.g., generated by the intermediate layer 218), S is a semantic representation of the target extended image, and <·,·> refers to a dot product operation.

[0078] Conditioning the discriminative neural network 202 according to the semantic representation may stabilize the joint training of the generative neural network 110 and the discriminative neural network 202, and also improve the quality of the extended image generated by the trained generative neural network 110. In particular, conditioning the discriminative neural network 202 according to the semantic representation may enable the discriminative neural network 202 to generate a discriminative output that depends on the entire semantic content of the target extended image (including masked portions from the target extended image). The semantic representation may provide effective conditioning information (i.e., conditioning information that enables the discriminative neural network to generate a more accurate discriminative output) as a result of characterizing the "global" context of the target extended image. In contrast, conditioning the discriminative network directly according to the pixel values ​​of the target extended image may be less effective, for example, because the pixel values ​​are "local" in nature and may be noisy. Conditioning the discriminative neural network according to the semantic representation may also train the discriminative neural network more efficiently (e.g., through fewer training iterations) by reducing the burden on the discriminative neural network to directly consider the semantic content in the output of its final layer. For example, referring to Equation (2), the discriminative output can take into account the semantic content through the inner product between the intermediate output of the discriminative neural network and the semantic representation.

[0079] After generating the discriminative output, the adversarial training system 116 may use an adversarial loss objective function 204 and reconstruction objective function 220 objective function The gradient of is used to adjust the current value of the generative neural network parameters, for example:

[0080]

[0081]

[0082]

[0083] Where λ is a scalar hyperparameter, x is the target expanded image, is the training expanded image generated by the generative neural network, |·| 1 YesL 1 norm, and It is the discriminative output generated by the discriminator neural network by processing the training extended image (or based on the input of the training extended image).

[0084] The objective function represented by equations (3)-(5) is provided as an illustrative example of an objective function, but other objective functions are possible. In general, the reconstruction objective function The target expanded image x and the training expanded image x can be represented in any of a variety of ways The similarity between them and the adversarial loss objective function The discriminant output may be determined in any of a number of ways

[0085] The adversarial training system 116 jointly trains the discriminative neural network 202 and the generative neural network 110 , for example, by alternating between training the discriminative neural network 202 and the generative neural network 110 for a predetermined number of training iterations.

[0086] At each training iteration during the training of the discriminative neural network, the current parameter values ​​of the discriminative neural network are adjusted based on the current batch of training examples. As previously mentioned, the current batch of training examples can include multiple training examples, but for convenience, the following description will refer to a specific training example.

[0087] To train the discriminative neural network 202, the adversarial training system 116 processes the training generative network input of the training example using the generative neural network 110 to generate the corresponding training extended image. The adversarial training system 116 processes the training extended image (or input based on the training extended image) and the corresponding target extended image using the discriminative neural network to generate the corresponding discriminative output. The adversarial training system 116 then adjusts the current parameter values ​​of the discriminative neural network 202 based on the discriminative objective function, which depends on the discriminative output generated for the training extended image and the target extended image. For example, the discriminative objective function It can be given by the following formula:

[0088]

[0089] where ReLU(·) is the rectified linear unit function, D(x) is the discriminant output generated by the discriminative neural network for the target expanded image, and is the discriminative output generated by the discriminative neural network for the training extended image generated by the generative neural network. The discriminative objective function represented by equation (6) is for illustrative purposes only, and other discriminative objective functions are also possible.

[0090] The discriminative neural network 202 can have any suitable neural network architecture. In one example, the discriminative neural network 202 has six strided convolutional layers with leaky ReLU activation functions, followed by fully connected layers.

[0091] During the training of the generative neural network and the discriminative neural network, the adversarial training system 116 can use, for example, back-propagation techniques to determine the gradient of the objective function. Using any suitable gradient descent optimization process (e.g., Adam or RMSprop), the adversarial training system 116 can use the gradient of the objective function to adjust the current parameter values ​​of the generative neural network and the discriminative neural network.

[0092] The adversarial training system 116 may continue to jointly train the generative neural network 110 and the discriminative neural network 202 until a training termination criterion is met, for example, until a predetermined number of training iterations have been performed for each neural network, or until a performance criterion is met.

[0093] Figure 3 By reference Figure 1 The depicted image expansion system 100 is a diagram of an example of an expanded image 300 generated by processing an original image 302. It can be appreciated that the expanded image 300 is a true expansion of the original image 302, which preserves the high-level semantic features and low-level structure and texture of the original image 302.

[0094] Figure 4 is a block diagram of an example environment 400 in which a digital component distribution system 410 uses the image extension system 100 in sending a digital component from a digital component database 416 for presentation with an electronic document. As will be described in greater detail below, the digital component distribution system can send the digital component in response to a request to present the digital component with the electronic document at a user device (e.g., in a block next to a search result or on a third-party website).

[0095] After identifying a digital component to be sent in response to the request, the distribution system can identify that an image included in the digital component qualifies for image extension. In one example, the distribution system can identify that the image qualifies for image extension based on a size of the image that is different (e.g., different by at least a threshold amount) from a size of a block in which the image will be rendered (e.g., aspect ratio). In this example, the digital component request can include data characterizing the size of the block in which the digital component will be rendered. In another example, the distribution system can identify that the image qualifies for image extension based on that a provider of the digital component has specified that other visual elements (e.g., text, logos, and interactive elements) of the digital component should not be overlaid on the image of the digital component.

[0096] In response to identifying that an image included in the digital component is eligible for expansion, the distribution system can process the image using the image expansion system 100 to generate an expanded image. In one example, the distribution system can generate the expanded image with an aspect ratio that matches the aspect ratio of the block in which the expanded image will be displayed. In another example, the distribution system can generate the expanded image and then overlay additional visual elements (e.g., text, logos, and interactive elements) of the digital component only on the expanded portion of the image (i.e., without modifying the original image).

[0097] After expanding the image included in the digital component, the distribution system may send the digital component together with the expanded image for presentation in blocks at the user device that generated the component request.

[0098] The example environment 400 and operations performed by the distribution system are described in more detail below.

[0099] Example environment 400 includes a network 402, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. Network 402 connects electronic document servers 404, client devices 406, digital component servers 408, and digital component distribution systems 410 (also referred to as "distribution systems" 410). Example environment 400 may include many different electronic document servers 404, client devices 406, and digital component servers 408.

[0100] Client device 406 is an electronic device capable of requesting and receiving resources over network 402. Example client devices 406 include personal computers, mobile communication devices (e.g., mobile phones), and other devices that can send and receive data over network 402. Client device 406 typically includes a user application, such as a web browser, to facilitate sending and receiving data over network 402, although native applications executed by client device 406 may also facilitate sending and receiving data over network 402.

[0101] An electronic document is data that presents a set of content at a client device 406. Examples of electronic documents include web pages, word processing documents, portable document format (PDF) documents, images, videos, search result pages, and feeds. Native applications (e.g., "apps"), such as those installed on mobile, tablet, or desktop computing devices, are also examples of electronic documents. Electronic documents can be provided to client devices 406 by electronic document servers 404 ("electronic Doc servers"). For example, electronic document servers 404 can include servers that host publisher websites. In this example, client device 406 can initiate a request for a given publisher web page, and electronic server 404 that hosts the given publisher web page can respond to the request by sending machine-executable instructions that initiate presentation of the given web page at client device 406.

[0102] In another example, the electronic document server 404 may include an app server from which the client device 406 can download the app. In this example, the client device 406 can download the files required to install the app at the client device 406 and then execute the downloaded app locally.

[0103] An electronic document may include a variety of content. For example, an electronic document may include static content (e.g., text or other specified content) that is within the electronic document itself and / or does not change over time. An electronic document may also include dynamic content that may change over time or upon request. For example, a publisher of a given electronic document may maintain data sources for populating various portions of the electronic document. In this example, a given electronic document may include one or more tags or scripts that cause the client device 406 to request content from a data source when the given electronic document is processed (e.g., rendered or executed) by the client device 406. The client device 406 integrates the content obtained from the data source into a given electronic document to create a composite electronic document that includes the content obtained from the data source.

[0104] In some cases, a given electronic document may include one or more digital component tags or digital component scripts that reference digital component distribution system 410. In these cases, when a given electronic document is processed by client device 406, the digital component tags or digital component scripts are executed by client device 406. Execution of the digital component tags or digital component scripts configures client device 406 to generate a request for one or more digital components 412 (referred to as a "component request"), which is sent to digital component distribution system 410 via network 402. For example, the digital component tags or digital component scripts may enable client device 406 to generate a packetized data request including header and payload data. Component request 412 may include event data that specifies characteristics such as the name (or network location) of a server from which a digital component is requested, the name (or network location) of a requesting device (e.g., client device 406), and / or information that digital component distribution system 410 may use to select one or more digital components to be provided in response to the request. Component request 412 is sent by client device 406 to a server of digital component distribution system 410 via network 402 (e.g., a telecommunications network).

[0105] Component request 412 may include event data specifying other event characteristics, such as the characteristics of the electronic document being requested and the location of the electronic document where the digital component can be presented. For example, event data may be provided to digital component distribution system 410 that specifies a reference (e.g., a URL) to an electronic document (e.g., a web page) in which the digital component is to be presented, an available location of the electronic document that can be used to present the digital component, the size of the available location, and / or the type of media that is eligible to be presented in the location. Similarly, event data specifying keywords associated with the electronic document ("document keywords") or entities (e.g., people, places, or things) referenced by the electronic document may also be included in component request 412 (e.g., as payload data) and provided to digital component distribution system 410 to facilitate identifying digital components that are eligible to be presented with the electronic document. Event data may also include a search query submitted from client device 406 to obtain a search results page, and / or data specifying search results and / or text, audible, or other visual content included in the search results.

[0106] The component request 412 may also include event data related to other information, such as information that has been provided by a user of the client device, geographic information indicating the state or region in which the component request was submitted, or other information that provides context for the environment in which the digital component will be displayed (e.g., the time of day of the component request, the day of the week of the component request, the type of device that will display the digital component, such as a mobile device or a tablet device). The component request 412 may be sent, for example, over a packetized network, and the component request 412 itself may be formatted as packetized data having a header and payload data. The header may specify a destination for the packet and the payload data may include any of the information described above.

[0107] Component distribution system 410 selects a digital component to be presented with a given electronic document in response to receiving component request 412 and / or using information included in component request 412. In some embodiments, a digital component is selected (using the techniques described herein) in less than one second to avoid errors that may be caused by delayed selection of the digital component. For example, a delay in providing a digital component in response to component request 412 may cause page loading errors at client device 406 or cause some portions of the electronic document to remain unpopulated, even after other portions of the electronic document are presented at client device 406. In addition, as the delay in providing a digital component to client device 406 increases, it is more likely that the electronic document will no longer be presented at client device 406 when the digital component is delivered to client device 406, thereby negatively affecting the user's experience of the electronic document. In addition, a delay in providing a digital component may cause a failure in the delivery of the digital component, for example, if the electronic document is no longer presented at client device 406 when the digital component is provided.

[0108] In some embodiments, the digital component distribution system 410 is implemented in a distributed computing system that includes, for example, a server and a set of multiple computing devices 414 that are interconnected and identify and distribute digital components in response to a request 412. The set of multiple computing devices 414 operate together to identify a set of digital components that are eligible for presentation in an electronic document from a corpus of millions of available digital components (DC1-x). For example, the millions of available digital components can be indexed in a digital component database 416. Each digital component index entry can reference a corresponding digital component and / or include distribution parameters (DP1-DPx) that facilitate (e.g., regulate or limit) the distribution / transmission of the corresponding digital component. For example, the distribution parameters can facilitate the transmission of the digital component by requiring the component request to include at least one criterion that matches (e.g., exactly or with some pre-specified similarity level) one of the distribution parameters of the digital component.

[0109] In some implementations, the distribution parameters for a particular digital component may include distribution keywords that must be matched (e.g., by an electronic document, document keywords, or terms specified in component request 412) in order for the digital component to be eligible for presentation. In other words, the distribution parameters are used to trigger the distribution (e.g., sending) of the digital component over network 402. The distribution parameters may also require that component request 412 include information specifying a particular geographic region (e.g., a country or state) and / or information specifying that component request 412 originates from a particular type of client device (e.g., a mobile device or a tablet device) in order for the digital component to be eligible for presentation.

[0110] The distribution parameters may also specify an eligibility value (e.g., a ranking score, a bid, or some other specified value) used to evaluate the eligibility of the digital component for distribution / delivery (e.g., through a component evaluation process) (e.g., among other available digital components). In some cases, the eligibility value may specify a maximum amount of compensation that the provider of the digital component is willing to submit in response to the delivery of the digital component (e.g., for each instance of a particular event attributed to the presentation of the digital component, such as a user interaction with the digital component).

[0111] The identification of eligible digital components may be split into a plurality of tasks 417a-417c, which are then distributed among the computing devices within the group of multiple computing devices 414. For example, different computing devices in the group 414 may each analyze different portions of the digital component database 416 to identify various digital components having distribution parameters that match the information included in the component request 412. In some implementations, each given computing device in the group 414 may analyze a different dimension (or set of dimensions) of data and communicate (e.g., send) the results of the analysis (Res 1-Res 3) 418a-418c back to the digital component distribution system 410. For example, the results 418a-418c provided by each computing device in the group 414 may identify a subset of digital components that are eligible to be distributed in response to the component request and / or a subset of digital components having specific distribution parameters. The identification of the subset of digital components may include, for example, comparing the event data to the distribution parameters, and identifying a subset of digital components having distribution parameters that match at least some features of the event data.

[0112] The digital component distribution system 410 aggregates the results 418a-418c received from the set of multiple computing devices 414 and uses information associated with the aggregated results to: (i) select one or more digital components to be provided in response to the request 412, and (ii) determine the requirements for sending the one or more digital components. For example, the digital component distribution system 410 can select a set of winning electronic digital components (one or more digital components) based on the results of one or more component evaluation processes. Then, the digital component distribution system 410 can generate and send reply data 420 (e.g., digital data representing the reply) over the network 402, which enables the client device 406 to integrate the set of winning digital components into a given electronic document so that the set of winning digital components and the content of the electronic document are presented together at the display of the client device 406.

[0113] In some embodiments, the client device 406 executes instructions included in the reply data 420, which configure and enable the client device 406 to obtain a set of winning digital components from one or more digital component servers. For example, the instructions in the reply data 420 may include a network location (e.g., a uniform resource locator (URL)) and a script that causes the client device 406 to send a server request (SR) 421 to the digital component server 408 to obtain a given winning digital component from the digital component server 408. In response to the request, the digital component server 408 will identify the given winning digital component specified in the server request 421 (e.g., within a database storing multiple digital components) and send digital component data (DC data) 422 to the client device 406, which presents the given winning digital component in an electronic document at the client device 406.

[0114] To facilitate searching of electronic documents, environment 400 may include a search system 450 that identifies electronic documents by crawling and indexing the electronic documents (e.g., indexing based on crawled content of the electronic documents). Data about electronic documents may be indexed based on the electronic documents associated with the data. Indexed and optionally cached copies of the electronic documents are stored in a search index 452 (e.g., hardware memory device(s)). Data associated with an electronic document is data representing content included in the electronic document and / or metadata of the electronic document.

[0115] The client device 406 may submit a search query to the search system 450 via the network 402. In response, the search system 450 accesses the search index 452 to identify electronic documents related to the search query. The search system 450 identifies the electronic documents in the form of search results and returns the search results to the client device 406 in a search results page. The search results are data generated by the search system 450 that identify electronic documents that are responsive to (e.g., related to) a particular search query and include active links (e.g., hypertext links) that cause the client device to request data from a specified network location (e.g., URL) in response to a user's interaction with the search results. Example search results may include a web page title, a portion of a text snippet or image extracted from a web page, and the URL of the web page. Another example search result may include a title of a downloadable application, a text snippet describing the downloadable application, an image depicting a user interface of the downloadable application, and / or a URL of a location from which the application can be downloaded to the client device 406. In some cases, search system 450 may be part of or interact with an application store (or online portal) from which applications may be downloaded for installation at client device 406, in order to present information about downloadable applications relevant to the submitted search query. As with other electronic documents, search results pages may include one or more slots in which digital components (e.g., advertisements, video clips, audio clips, images, or other digital components) may be presented.

[0116] In order to select a digital component to be sent in response to a component request, the distribution system 410 may identify a set of digital components that are eligible to be sent in response to the component request. The distribution system 410 may then select one or more eligible digital components to be sent, for example, through an auction process. In some embodiments, the distribution system 410 performs an auction process by ranking the eligible digital components according to their respective eligibility values ​​and selecting one or more highest-ranked digital components in response to the component request.

[0117] For example, distribution system 410 may identify digital components A, B, and C as eligible to be sent in response to a component request. In this example, digital component A has an eligibility value of $5, digital component B has an eligibility value of $1, and digital component C has an eligibility value of $5.5, where the eligibility values ​​of the digital components represent bids associated with the digital components. Distribution system 410 may rank the digital components (e.g., in descending order) according to their respective eligibility values ​​as: C, A, B. Finally, distribution system 410 may select the highest ranked digital component C to send in response to the component request.

[0118] After selecting a digital component to be sent in response to the digital component request, the distribution system 410 determines a sending requirement for the selected digital component. The sending requirement specifies an action to be performed by the provider of the digital component in response to the sending of the digital component. For example, the sending requirement may specify that the provider of the digital component submit a certain amount of compensation in response to the sending of the digital component. In some cases, the compensation amount specifies an amount to be submitted for each instance of a particular event attributed to the presentation of the digital component (e.g., a user interaction with the digital component).

[0119] Distribution system 410 may determine the sending requirement for the selected digital component based on the eligibility value of the selected digital component and / or the eligibility values ​​of other digital components determined to be eligible to be sent in response to the component request. For example, distribution system 410 may identify digital components A, B, and C as eligible for sending in response to the digital component request, where A, B, and C have corresponding eligibility values ​​of $5, $1, and $5.5. Distribution system 410 may select digital component C for sending (because it has the highest eligibility value), and may determine the sending requirement for digital component C to be the next highest eligibility value among the eligibility values ​​of the eligible digital components. In this example, the next highest eligibility value is $5 (i.e., the eligibility value of digital component A), and therefore distribution system 410 may determine the sending requirement for digital component C to be $5.

[0120] Figure 5 A digital component is shown being displayed on a screen of a user device (e.g., a smart phone). The digital component includes an image that fills the device screen and additional visual elements, including a logo 502, a text segment 504, and an interactive element 506 that overlays the image. In this example, the image includes a first portion 508 provided to the digital component distribution system and a second portion 510 generated using the image expansion system 100. After receiving a request to provide the digital component to the user device, the digital component distribution system generates an expanded image that fills the device screen and overlays the additional elements 502, 504, and 506 on the expanded portion 510 of the image. In this example, the "block" in which the digital component is displayed corresponds to the entire screen of the user device, but the block may occupy less than the full screen.

[0121] Figure 6 A digital component is shown that is displayed, for example, in a "banner" block along with search results or on a third-party website. Figure 6 , the digital component includes an image of the filler block and an additional element 602. In this example, the image includes a first portion 604 provided to the digital component distribution system and a second portion 606 generated using the image expansion system 100. After receiving a request to present the digital component in a banner block on a user device, the digital component distribution system generates an expanded image of the filler block and overlays the additional element 602 on the expanded portion 606 of the image.

[0122] Figure 7 is an illustration of an example user interface 700 that may be presented to an entity (e.g., an advertiser, publisher, or other content provider) that provides a digital component for delivery by a digital component distribution system (e.g., as described in reference to Figure 4 The user interface 700 presents an image 702 included in the provided digital component and a plurality of expanded images 704-AC that expand the image 702. Typically, each expanded image 704-AC is different because it is generated using a different image expansion process. For example, an expanded image may have been generated using a reference image. Figure 1 The other extended image may have been generated using an image expansion system described herein, while the other extended image may have been generated using an image expansion process based on partial differential equations (PDEs), which will be described in more detail below.

[0123] User interface 700 prompts the provider of the digital component to select a preferred one of the extended images (e.g., an extended component that is most visually appealing to the digital component provider). The digital component provider may select a particular extended image, such as by clicking on the particular extended image using mouse 706 (or in any of a variety of other ways). If the distribution system later determines that the image included in the provided digital component is eligible for expansion, the distribution system generates an expansion of the image using the image expansion process corresponding to the particular extended image selected by the user.

[0124] Images can be extended in a variety of ways, for example, using reference Figure 1 An image expansion system as described herein, or an image expansion process based on a PDE is used. In the PDE expansion method, pixel values ​​in the expanded portion of the image are identified as (approximately or exactly) solutions to a PDE that is subject to boundary conditions specified by pixel values ​​on a portion of the boundary of the original image. The PDE can be, for example, a diffusion PDE or any other suitable PDE. Any suitable numerical technique (e.g., finite element technique) can be used to obtain solutions to the PDE that is subject to boundary conditions.

[0125] Figure 88 is a flow chart of an example process 800 for providing an extended image in response to a request. For convenience, process 800 will be described as being performed by a system of one or more computers located in one or more locations. For example, a distribution system appropriately programmed according to the present specification, such as Figure 1 The distribution system 100 can perform process 800.

[0126] The system determines that a provided image is eligible for image expansion (802). In one example, the system can receive a request for a digital component to be presented in a tile, such as alongside search results or on a third-party webpage, and then determine a digital component including the provided image in response to the request. In this example, the system can determine that the provided image is eligible for expansion based on a size of the image being different than a size of the tile, such as if the image and the tile have different aspect ratios.

[0127] The system provides an input including an image to a generative neural network, which processes the input according to trained values ​​of generative neural network parameters to generate an extended image (804). The extended image has (i) more rows, more columns, or more rows and more columns than the provided image, and (ii) is predicted to be a true extension of the provided image.

[0128] The input processed by the generative neural network may include a baseline image and a mask image, both having the same number of rows and columns as the extended image. The baseline image includes: (i) a first portion corresponding to the provided image, and (ii) a second portion having default pixel values. The mask image identifies the first portion and the second portion of the baseline image. In one example, pixels in the mask image corresponding to the first portion of the baseline image all have a first pixel value, and pixels in the mask image corresponding to the second portion of the baseline image all have a different second pixel value.

[0129] The generative neural network includes multiple convolutional neural network layers and can have any suitable neural network architecture. For example, the generative neural network can include one or more skip connections, one or more instance normalization layers, or both.

[0130] The generative neural network has been jointly trained with the discriminative neural network using an adversarial loss objective function. The discriminative neural network is configured to process a given image to generate a discriminative output representing the likelihood that the given image was generated using the generative neural network.

[0131] The system trains a generative neural network using an adversarial loss objective function in multiple training iterations. At each training iteration, the system processes a training input including a training image using a generative neural network to generate a training extended image that extends the training image, and then generates a discriminative neural network input based on the training extended image. In one example, the system generates a discriminative neural network input by covering a portion of the training extended image corresponding to the training image with the training image. The system processes the discriminative neural network input using a discriminative neural network to generate a discriminative output that characterizes the likelihood that the discriminative neural network input is generated using a generative neural network. Thereafter, the system adjusts the current value of the generative neural network parameter based on the adversarial loss objective function, wherein the adversarial loss objective function depends on the discriminative output generated by the discriminative neural network.

[0132] In some cases, the training image is a cropped representation of the target image, and the discriminative neural network is adjusted according to the semantic feature representation of the target image. The system can determine the semantic representation of the target image as an intermediate output generated by the classification neural network by processing the target image. Adjusting the discriminative neural network according to the semantic feature representation of the target image can include, for example, determining the discriminative output based on a similarity measure between the intermediate output of the discriminative neural network and the semantic feature representation of the target image. The discriminative output can be additionally determined based on the output of the last layer of the discriminative neural network.

[0133] In addition to training the generative neural network using the adversarial loss objective function, the system can also train the generative neural network using the reconstruction loss objective function. More specifically, the system can adjust the current values ​​of the parameters of the generative neural network based on the reconstruction loss objective function that characterizes the similarity between the training extended image and the target image.

[0134] The system provides the expanded image in response to the request (806). In some cases, the request specifies additional elements to be presented with the image in the tile, and the system overlays the additional elements on the expanded portion of the expanded image. After the system provides the expanded image in response to the request, the expanded image can be presented, for example, next to the search results or on a third-party web page.

[0135] Fig. 9900 is a block diagram of an example computer system 900 that can be used to perform the above operations. System 900 includes a processor 910, a memory 920, a storage device 930, and an input / output device 940. Each of the components 910, 920, 930, and 940 can be interconnected, for example, using a system bus 950. Processor 910 is capable of processing instructions for execution within system 900. In one embodiment, processor 910 is a single-threaded processor. In another embodiment, processor 910 is a multi-threaded processor. Processor 910 is capable of processing instructions stored in memory 920 or on storage device 930.

[0136] The memory 920 stores information within the system 900. In one embodiment, the memory 920 is a computer readable medium. In one embodiment, the memory 920 is a volatile memory unit. In another embodiment, the memory 920 is a non-volatile memory unit.

[0137] The storage device 930 can provide mass storage for the system 900. In one embodiment, the storage device 930 is a computer-readable medium. In various embodiments, the storage device 930 may include, for example, a hard disk device, an optical disk device, a storage device shared by multiple computing devices over a network (e.g., a cloud storage device), or some other mass storage device.

[0138] Input / output device 940 provides input / output operation for system 900.In one embodiment, input / output device 940 can include one or more network interface devices, such as Ethernet card, serial communication device (such as and RS-232 port) and / or wireless interface device, such as and 802.11 card.In another embodiment, input / output device can include the driver device that is configured to receive input data and output data is sent to other input / output devices (such as keyboard, printer and display device 960).But, also can use other embodiments, such as mobile computing device, mobile communication device, set-top box TV client device etc.

[0139] Although already Fig. 9 An example processing system is described in the specification, but the subject matter and implementation of the functional operations described in this specification may be implemented in other types of digital electronic circuits or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them.

[0140] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination of software, firmware, hardware that, in operation, causes the system to perform those operations or actions. For one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0141] Embodiments of the subject matter and functional operations described in this specification may be implemented with digital electronic circuits, with tangibly implemented computer software or firmware, with computer hardware including the structures disclosed in this specification and their structural equivalents, or with a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more of them. Alternatively or in addition, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0142] The term "data processing apparatus" refers to data processing hardware and includes all kinds of apparatus, devices and machines for processing data, including by way of example a programmable processor, a computer or a plurality of processors or computers. An apparatus may also be or further include special-purpose logic circuitry, for example, an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). An apparatus may optionally include, in addition to hardware, code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0143] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script or code, may be written in any form of programming language including compiled or interpreted languages ​​or declarative or procedural languages; and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document; in a single file dedicated to the program or in multiple coordinated files, such as files storing one or more modules, subroutines or portions of code. A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0144] The term "engine" is used broadly in this specification to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines may be installed and run on the same computer or multiple computers.

[0145] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a special purpose logic circuit such as an FPGA or ASIC, or by a combination of a special purpose logic circuit and one or more programmed computers.

[0146] A computer suitable for executing a computer program may be based on a general-purpose microprocessor or a special-purpose microprocessor or both, or any other kind of central processing unit. Typically, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for executing or implementing instructions and one or more storage devices for storing instructions and data. The central processing unit and the memory may be supplemented by a dedicated logic circuit or incorporated in the dedicated logic circuit. Typically, a computer will also include one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or may be operationally coupled to receive data from the one or more large-capacity storage devices or to transfer data to the one or more large-capacity storage devices, or both for storing data. However, a computer need not have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game controller, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, etc.

[0147] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including by way of example semiconductor storage devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks.

[0148] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on the user's device in response to receiving a request from the web browser. In addition, the computer may interact with the user by sending a text message or other form of message to a personal device, such as a smart phone running a messaging application, and then receiving a response message from the user.

[0149] The data processing apparatus for implementing a machine learning model may also include, for example, dedicated hardware accelerator units for processing common and computationally intensive parts of machine learning training or production, i.e., inference, workloads.

[0150] The machine learning model can be implemented and deployed using a machine learning framework, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0151] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server; or includes a middleware component, such as an application server; or includes a front-end component, such as a client computer with a graphical user interface, a web browser, or an app that a user can use to interact with an implementation of the subject matter described in this specification; or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0152] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data such as an HTML page to the user device, for example, for the purpose of displaying data to a user interacting with the device as a client and receiving user input from the user. Data generated at the user device, such as the result of the user interaction, may be received at the server from the device.

[0153] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or that may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. In addition, although features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from a claimed combination may be removed from the combination in some cases, and a claimed combination may be directed to a subcombination or variations of a subcombination.

[0154] Similarly, although operations are depicted in the drawings and described in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0155] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims may be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more data processing devices, the method comprising: An input including a provided image is provided to a generative neural network having a plurality of generative neural network parameters, wherein: The generative neural network processes the input according to the trained values ​​of the plurality of generative neural network parameters to generate an extended image; The extended image has (i) more rows, more columns, or more rows and more columns than the provided image, and (ii) is predicted to be an extension of the provided image; and The generative neural network has been trained using an adversarial loss objective function; Training the generative neural network using the adversarial loss objective function includes processing a training input including a training image using the generative neural network; The generative neural network is jointly trained with a discriminative neural network having a plurality of discriminative neural network parameters, the discriminative neural network being configured to process a given image to generate a discriminative output representing a likelihood that the given image was generated using the generative neural network; and The discriminative neural network is adjusted according to a semantic feature representation of a target image, the training image being a cropped representation of the target image, wherein the semantic feature representation of the target image is provided as: an additional input to the discriminative neural network; or the discriminative output is determined at least in part based on a similarity measure between a semantic representation of a target extended image and an intermediate output of the discriminative neural network, and wherein the semantic representation of the target extended image is normalized by a normalization engine before the semantic representation of the target extended image is used to adjust the discriminative neural network.

2. The method according to claim 1, further comprising: receiving a request for an image to be rendered in a block; determining that the provided image is responsive to the request; and After the extended image is generated, the extended image is provided in response to the request.

3. The method according to claim 2, further comprising: The provided image is determined to be eligible for image expansion based on the size of the block being different from the size of the provided image, and in response, the input including the provided image is provided to the generative neural network.

4. The method according to claim 3, wherein: The size of the block specifies the aspect ratio of the block.

5. The method according to any one of claims 2 to 4, wherein: The block is to be presented next to the search results or on a third-party web page.

6. The method according to any one of claims 2 to 4, in, The request specifies additional elements to be presented with the image in the block; and Providing the extended image in response to the request includes overlaying the additional element on an extended portion of the extended image.

7. The method according to any one of claims 1 to 4, wherein: The generative neural network includes multiple convolutional neural network layers.

8. The method according to any one of claims 1 to 4, wherein: The generative neural network includes a plurality of skip connections.

9. The method according to any one of claims 1 to 4, wherein: The generative neural network includes multiple instance normalization layers.

10. The method according to any one of claims 1 to 4, wherein: The input of the generative neural network includes: a baseline image having the same number of rows and columns as the extended image, wherein the baseline image includes: (i) a first portion corresponding to the provided image, and (ii) a second portion having default pixel values; and A mask image having the same number of rows and columns as the extended image, wherein the mask image identifies the first portion and the second portion of the baseline image.

11. The method according to claim 10, wherein: (i) pixels in the mask image corresponding to a first portion of the baseline image each have a first pixel value, and (ii) pixels in the mask image corresponding to a second portion of the baseline image each have a second pixel value different from the first pixel value.

12. The method according to claim 1, wherein: Training the generative neural network using the adversarial loss objective function includes: processing a training input including a training image using the generative neural network and based on current values ​​of parameters of the generative neural network to generate a training extended image that extends the training image; generating a discriminative neural network input based on the training extended image; processing the discriminative neural network input using the discriminative neural network and according to current values ​​of the discriminative neural network parameters to generate a discriminative output representing a likelihood that the discriminative neural network input was generated using the generative neural network; and The current values ​​of the generative neural network parameters are adjusted based on the adversarial loss objective function, wherein the adversarial loss objective function depends on the discriminant output characterizing the likelihood that the discriminative neural network input was generated using the generative neural network.

13. The method according to claim 12, wherein: Generating the discriminative neural network input based on the training extended image comprises: A portion of the training extended image corresponding to the training image is overwritten with the training image.

14. The method according to any one of claims 12-13, further comprising: The current values ​​of the generative neural network parameters are adjusted based on a reconstruction loss objective function that characterizes the similarity of the training extended images to a target image, wherein the training images are cropped representations of the target image.

15. A system comprising one or more computers and one or more storage devices storing instructions which, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding method according to any one of claims 1-14.

16. A computer storage medium storing instructions which, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method according to any one of claims 1-14.

Images