Using combined inputs to power generative adversarial networks

Abstract

The goal is to improve the performance and / or training efficiency of GAN models. A method, system, and apparatus, including a computer program encoded on a computer storage medium, for generating a synthetic signal. A computer-implemented system obtains generator input data including at least an input signal having one or more first characteristics, processes the generator input data to generate output data including a synthetic signal having one or more second characteristics using a generator neural network, and outputs the synthetic signal to a device. The generator neural network is trained with a discriminator neural network based on a plurality of training examples. The discriminator neural network is configured to process the discriminator input data, which combines the discriminator input signal having the one or more second characteristics with at least a portion of the generator input data, to generate a prediction of whether the discriminator input signal is a real signal provided in one of the plurality of training examples or a synthetic signal.

Description

[Background technology]

[0001] A generative adversarial network (GAN) is a machine learning neural network that learns features of training data to generate new data with the same properties as the training data. A GAN includes a generator neural network that generates new data and a discriminator neural network that provides feedback to the generator neural network on how to improve the generated data. Summary of the Invention

[0002] In a generative adversarial network (GAN) model, the generator neural network generally aims to generate output signals that are likely to be derived from input data, and the discriminator neural network aims to classify the signals as either "real" (from a training dataset) or false (generated by the generator neural network). In one example, a computer-implemented system can use the generator neural network to process low-resolution input images to generate high-resolution synthetic images, and the discriminator neural network to predict whether a high-resolution image input to the discriminator neural network is a real (e.g., provided in one of the training examples) high-resolution image or a high-resolution image synthesized by the generator neural network. The system can use the predictions generated by the discriminator neural network to update the network parameters of the generator neural network, enabling the generator neural network to generate more realistic high-resolution synthetic images.

[0003] To train the discriminator neural network of the GAN model to make predictions, the system uses multiple training examples. Each training example includes at least a training signal in the same domain as the signal generated by the generator neural network. Typically, the system uses either the output of the generator neural network or the training signal in the training examples as input data to the discriminator neural network. The discriminator neural network processes the input to predict whether the input is the generated signal output by the generator neural network or the "real" (training) signal. The system updates the network parameters of the discriminator neural network based on a loss function that measures the comparison between the prediction output by the discriminator neural network and whether the input is the generated signal output by the generator neural network or the training signal. That is, the discriminator neural network is trained to distinguish between the signal generated by the generator neural network and the real signal.

[0004] This specification describes systems, methods, devices, and other techniques related to improving the performance and / or training efficiency of GAN models.

[0005] In one aspect of the present specification, a method for generating a synthetic signal using a GAN model is provided. The GAN model includes a generator neural network and a discriminator neural network. The system obtains input data (conveniently referred to as generator input data) for the generator neural network. The generator input data includes at least an input signal having one or more first characteristics. The system processes the generator input data to generate output data including a synthetic signal having one or more second characteristics using the generator neural network. The generator neural network is trained with a discriminator neural network based on a plurality of training examples. The input data to the discriminator neural network (conveniently referred to as discriminator input data) combines a discriminator input signal having one or more second characteristics with at least a portion of the generator input data. The discriminator neural network is configured to process the discriminator input data to generate a prediction of whether the discriminator input signal is a label signal (considered to be a "real" signal) provided in one of the plurality of training examples or a synthetic signal output by the generator neural network. The system further outputs the composite signal to a device.

[0006] In another aspect of the present specification, a method for training a GAN model is provided. The GAN model includes a generator neural network and a discriminator neural network. The generator neural network is configured to process generator input data including at least one input signal having one or more first characteristics to generate output data including a synthetic signal having one or more second characteristics. The discriminator neural network is configured to process discriminator input data combining the discriminator input signal having one or more second characteristics with at least a portion of the generator input data, and generate a prediction of whether the discriminator input signal is a signal provided in one of a plurality of training examples (considered to be a "real" signal) or a synthetic signal output by the generator neural network.

[0007] To train the generator neural network and the discriminator neural network, the system obtains a plurality of training examples. Each training example includes at least a first training signal having one or more first characteristics and a second training signal having one or more second characteristics. Based on each training example, the system generates training input data for the generator neural network by including at least the first training signal in the training example, and processes the training input data using the generator neural network to generate a synthetic signal. The system generates a first instance of discriminator input data by combining the synthetic signal with at least a portion of the training input data for the generator neural network, and processes the first instance of the discriminator input data using the discriminator neural network to generate a first prediction. The system also generates a second instance of discriminator input data by combining a second training signal with a portion of the training input data for the generator neural network, and processes the second instance of the discriminator input data using the discriminator neural network to generate a second prediction. The system updates a first set of network parameters of the discriminator neural network based on a first loss function that measures a comparison between a prediction output by the discriminator neural network and whether the discriminator input signal in the discriminator input data is a second training signal or a synthetic signal, and updates a second set of network parameters of the generator neural network based on a second loss function that includes at least an adversarial loss that measures a decreasing function of the prediction output by the discriminator neural network while the discriminator input signal in the discriminator input data is a synthetic image output by the generator neural network, where a larger value of the adversarial loss indicates that the discriminator neural network predicts a decreasing probability that the input high-resolution image in the discriminator input data is a "real" image.

[0008] In an exemplary implementation, the GAN model is used to generate a high-resolution synthetic image from a low-resolution input image. The input signal and the synthetic signal are both images. The first characteristic may be a first spatial resolution of the image. The second characteristic may be a second spatial resolution for the image that is different from the first spatial resolution. Specifically, in this example, the input signal is a low-resolution input image having a first spatial resolution. The synthetic signal is a high-resolution synthetic image having a second spatial resolution that is higher than the first spatial resolution. The discriminator input signal is a high-resolution image having a second spatial resolution, and may be either a high-resolution "real" image provided in one of a plurality of training examples, or a high-resolution synthetic signal output by the generator neural network.

[0009] Furthermore, the GAN model can be used to generate a high-resolution synthetic distribution map showing the fire distribution of the fire burning area. The generator input data includes a low-resolution input image (input signal) showing the fire distribution of the fire burning area and a reference input image showing the features of the area. The low-resolution input image has a first spatial resolution. The reference input image has a third spatial resolution higher than the first spatial resolution. The computer system then uses the generator neural network to process the low-resolution input image and the reference input image to generate a high-resolution synthetic image showing the fire distribution of the area at a second spatial resolution higher than the first spatial resolution, thus providing the high-resolution fire distribution features required to understand the wildfire spread behavior.

[0010] To train the generator neural network and the discriminator neural network, each of the training examples further includes low-resolution training images (first training signal) having a first SR, high-resolution training images (second training signal) having a second SR, and reference training images having a third spatial resolution. The discriminator input data includes the discriminator input signal (in this case the high-resolution synthetic images or the high-resolution training images) combined with at least a portion of the training input data for the generator neural network, including, for example, the low-resolution training images and / or the reference training images.

[0011] In some other examples of implementations, the GAN models may be used to generate high fidelity audio, high definition video, or other types of signals.

[0012] Generally, the described systems and associated methods improve the performance and / or training efficiency of GAN models, which can be used to generate signals, such as high-resolution synthetic images, based on at least an input signal, such as a low-resolution input image.

[0013] Typically, the discriminator neural network of a GAN model aims to distinguish between a signal generated by a generator neural network and a real signal provided in a training example, and therefore the input to the discriminator neural network usually includes only the signal output by the discriminator neural network or the real signal in the training example. The systems and methods described herein improve the performance and / or training efficiency of a GAN model by incorporating at least a portion of the input data to the generator neural network (generator input data) into the input data to the discriminator neural network (discriminator input data) during training of the discriminator neural network. In this way, instead of aiming to distinguish between a signal generated by the generator neural network and a real signal without reference, the trained discriminator neural network resulting from the process provided herein can determine whether an input signal to the discriminator neural network (e.g., a high-resolution image) is a reasonable processed version (e.g., with resolution upscaling) of another input signal (e.g., a low-resolution input image).

[0014] In an example implementation of generating a high-resolution synthetic distribution map showing the fire distribution of a fire burning area, by incorporating a low-resolution training image and a reference training image into the discriminator input data, the discriminator neural network is more effectively trained to determine whether the high-resolution image is a reasonable upscaling of the low-resolution input image according to the reference input image. The resulting improvements include better prediction accuracy of the discriminator neural network, better quality in the high-resolution synthetic image generated by the generator neural network, and / or improved training efficiency, such as requiring fewer training examples to achieve a certain performance metric of the GAN model, thus reducing the computational power required to perform the training.

[0015] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0016] [Figure 1] 1 illustrates an exemplary high resolution image generation system. [Diagram 2] 1 illustrates an exemplary process for training a GAN model for use in a high-resolution image generation system. [Diagram 3] FIG. 1 is a flow diagram of an example process for generating a synthetic signal using a GAN model. [Figure 4] FIG. 1 is a flow diagram of an example process for training a GAN model. [Diagram 5] 1 illustrates an exemplary computer system for performing operations described herein.

[0017] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Generative Adversarial Network (GAN) models are used in many applications for generating signals, such as for image resolution upscaling tasks. This specification describes systems, methods, devices, and other techniques for using GAN models to generate synthetic signals, such as high-resolution fire distribution maps, with improved performance and / or training efficiency for the GAN models. The GAN model includes a generator neural network and a discriminator neural network.

[0019] A feature of the described system and method is that during training of the GAN model, the system incorporates at least a portion of the input data to the generator neural network into the input data to the discriminator neural network. In this way, instead of aiming to distinguish between a signal generated by the generator neural network and a real signal without reference, the trained discriminator neural network can be more effective in determining whether an input signal to the discriminator neural network (e.g., a high-resolution image) is a reasonably processed version (e.g., with resolution upscaling) of another input signal (e.g., a low-resolution input image).

[0020] 1 illustrates an exemplary high resolution image generation system 120. System 120 is one example of a system that may implement the systems, components, and techniques described below, implemented as a computer program on one or more computers at one or more locations.

[0021] Although system 120 is focused on generating high resolution synthetic images showing fire distribution features, other types of synthetic signals can be generated using similar systems, including, for example, high fidelity audio and video with high spatial and / or temporal resolution. System 120 is one example of a system that generates various types of synthetic signals.

[0022] To build useful models of wildfire spread and behavior, accurate, high-resolution training data of actual real-world fires is needed. Unfortunately, most of the wildfire observation datasets available today are low resolution and / or collected infrequently. For example, many satellite-based remote sensing infrared (IR) imaging systems typically take survey infrared images at low resolution, e.g., at a spatial resolution of about 400 m / pixel or less. Systems that provide higher resolution survey images only need to acquire higher resolution infrared images every 12 hours, sometimes every two weeks. The low spatial and / or temporal resolution in available datasets makes them difficult to use to understand and predict wildfire spread using data-driven model-based predictions.

[0023] The system 120 can be used to automatically generate a high-resolution fire distribution map based on available fire-related data with low spatial resolution and pre-fire / post-fire geophysical maps of the corresponding area. The system 120 can receive inputs of a low-resolution distribution map showing the fire distribution of an area and a high-resolution reference input image of the same area, and outputs a high-resolution composite image showing the fire distribution of the area.

[0024] 1, a system 120 can obtain multiple training examples 110 and process the training examples 110 using the system's training engine 122 to update network parameters 124 of a machine learning model 121. Each training example can include a low-resolution training image 110a of a region, a reference training image 110b of the same region, and a high-resolution training image 110c of the same region.

[0025] 1, the system 120 can obtain input data 140, process the input data 140 using the machine learning model 121 having learned network parameters 124, and output a high-resolution synthetic image 155 based on the processing result to the output device 150. The input data can include a low-resolution input image 140a of a fire burning area and a reference input image 140b of the same area.

[0026] In this specification, "low resolution" and "high resolution" describe spatial resolution in a relative sense. For example, if the input image 140a has a first spatial resolution R1 (e.g., 400 m / pixel) and the high-resolution composite image 155 has a second spatial resolution R2 (e.g., 20 m / pixel), the second spatial resolution R2 is higher than the first spatial resolution R1, so the high-resolution composite image 155 is considered to be a high-resolution map and the input image 140a is considered to be a low-resolution map.

[0027] In the example shown in Figure 1, the low-resolution input image 140a may be a low-resolution infrared image. In general, the low-resolution input image 140a may include a distribution map or a data set showing the fire distribution of the fire burning area. The low-resolution infrared image is an example of a distribution map.

[0028] The satellite infrared imagery can show the spatial distribution of active fires, since active fires burning on the ground emit a spectral signal characterized by increased emissions of mid-infrared radiation that can be captured by satellite infrared sensors. The low-resolution infrared image 140a can be an infrared image of a single infrared band corresponding to the heat distribution, such as a mid-IR band with a central wavelength of 2.1 μm, 4.0 μm, or 11.0 μm. The low-resolution infrared image 140a can also include additional infrared data of other infrared bands, such as one or more near-IR bands with central wavelengths of 0.65 μm and / or 0.86 μm. These near-IR data can be used to calibrate artifacts such as sunlight and cloud reflections. The low-resolution infrared image 140a can include a multi-channel infrared image taken in multiple infrared bands, or a composite infrared image combining multiple channel infrared images. In addition to the infrared image, the low-resolution input image 140a can further include calibration and geolocation information, which can be used to pre-process the infrared image to ensure consistency between data sources and across different time points.

[0029] In some implementations, instead of receiving infrared images directly from instrument measurements or simply combining multiple channel infrared images, the input data low-resolution input image 140a may include derived products, such as fire distribution maps generated by processing multiple remotely sensed images using fire detection algorithms. Various fire products that map fire hotspots based on satellite remotely sensed images have been developed and are available from several agencies and can be used as the input low-resolution input image 140a.

[0030] A large number of maps showing fire distribution, whether directly received remote sensing measurements or fire maps derived using fire detection algorithms, can be retrieved from satellite remote sensing image archives or in near real time from satellite remote sensing image providers. These maps can include a series of images taken at multiple points in time for the same area and therefore can include information of the temporal characteristics of the fire spread behavior. However, these maps often have a low spatial resolution, i.e. each pixel in the map corresponds to a large area and cannot provide finer spatial details of the fire distribution.

[0031] On the other hand, the reference input image 140b can provide higher resolution features of the same area. In the example shown in FIG. 1, the reference input image 140b is a high-resolution aerial landscape image of the same area. In general, the reference input image 140b can include a reference input image that shows a particular feature of the area. The reference input image 140b has a higher spatial resolution than the spatial resolution of the low-resolution input image 140a. For example, the low-resolution input image 140a can have a spatial resolution of about 400 m / pixel or less, while the reference input image 140b can have a spatial resolution of about 20 m / pixel or more.

[0032] In addition to having different spatial resolutions, the reference input image 140b may be collected by a sensor or imaging device at a different time than when the low-resolution input image 140a is collected. For example, the low-resolution input image 140a may be collected during an active fire, while the reference input image 140b may be collected at a pre-fire time point or a post-fire time point, such as days, weeks, or months before the low-resolution input image 140a is collected. During an active fire burning, a series of images 140a may be collected at multiple time points for the same area, thus providing information about the fire's temporal spread behavior. The reference input image 140b may be used with each of the series of images 140a to form the input data 140.

[0033] Furthermore, the features shown in the reference input image 140b can be features other than fire and temperature distributions. That is, the reference input image 140b can have a modality different from that of the low-resolution input image 140a. For example, the low-resolution input image 140a can be an infrared image or a fire distribution map derived from remotely sensed infrared data, and the reference input image 140b can be an image in the visible wavelength range or a non-optical image. Examples of the reference input image 140b include satellite images in the visible band, e.g., center wavelength 0.65 μm, aerial photographs (e.g., collected by a drone), labeled survey maps, and vegetation index maps calculated from visible and near-IR images. The reference input image 140b can provide information about fire susceptibility at a higher resolution compared to the input image 140a in terms of features such as topographical features (e.g., elevation, slope, rivers, coastlines, etc.), man-made structures (roads, buildings, land, etc.), vegetation index, and / or soil moisture of the same area. The reference input image may also be a post-fire map showing burn scarring of the area, which also provides information indicative of fire susceptibility.

[0034] In some implementations, the system 120 can further perform pre-processing of the input data. For example, the system 120 can use the calibration data to calibrate the satellite infrared imagery and use the geolocation data to align and register the satellite infrared imagery with a reference input imagery. The system can further convert the set of satellite infrared images in the input data into a fire distribution map based on a fire detection algorithm. The fire detection algorithm can include processes such as cloud masking, background characterization and removal, sunlight rejection, and threshold application. The system 120 can then use the machine learning model 121 to process the pre-processed input data to generate output data including a high-resolution composite image 155.

[0035] In the example shown in FIG. 1, the high-resolution composite image 155 is a fire distribution map showing the distribution of fire burning locations at a higher spatial resolution. The fire distribution map can be a binary map with pixels having high or low intensity values. A pixel with a high intensity value in the map indicates an active fire burning at the corresponding location, and a pixel with a low intensity value in the map indicates no active fire burning at the corresponding location. Alternatively, the composite image 155 can have multiple or continuous distributions of pixel intensity values. A pixel with a higher intensity value can indicate a location with a high probability of active fire burning. Alternatively, a pixel with a higher intensity value can indicate a location with a higher intensity fire burning, e.g., different pixel intensity values ​​can be mapped to different levels of fire radiant power (FRP).

[0036] In some implementations, the high-resolution synthetic image 155 can include a fire distribution map derived from a probabilistic posterior distribution of possible fire distribution maps. The output of the machine learning model can also include a quantification of the model uncertainty at each output pixel.

[0037] The machine learning model 121 is based on a generative adversarial neural network (GAN), which includes a generator neural network 121a for generating synthetic data and a discriminator neural network 121b for distinguishing the synthetic data from "real" data.

[0038] GANs have been used in the past for resolution upscaling tasks, but those efforts were usually focused on designing an appropriate perceptual loss function to create visually realistic images with enhanced resolution. In contrast, the GAN model 121 provided herein aims to leverage additional information provided in the reference input images 140b in generating high-resolution fire distribution maps. Unlike past super-resolution GAN models, the system 120 does not aim to provide visually pleasing images. This allows for a training process focused on learning fire dynamics.

[0039] Specifically, as shown in step (C) of FIG. 1, the machine learning model 121 of the system 120 receives both the low-resolution input image 140a and the reference input image 140b as input and generates output data including a high-resolution composite image 155.

[0040] The GAN model 121 includes both a generator neural network 121a and a discriminator neural network 121b. The generator neural network 121a is used to process neural network inputs to generate output data. The neural network input to the generator neural network 121a can be a combination of a low-resolution input image 140a and a reference input image 140b. For example, the input can be formed by stacking a low-resolution distribution map and a reference input image.

[0041] The generator neural network 121a may include multiple network layers, including, for example, one or more fully connected layers, convolutional layers, parametric rectified linear unit (PReLU) layers, and / or batch normalization layers. In some implementations, the generator neural network 121a may include one or more residual blocks that include a skip connection layer.

[0042] The generator neural network 121a includes a set of network parameters, including, for example, weights and bias parameters of the network layers. These parameters are updated in a training process to minimize a loss characterization difference between the model's output and a desired output. The set of network parameters of the generator neural network 121a are part of the network parameters 124 of the machine learning model 121. The system 120 further includes a training engine 122 for updating these network parameters 124.

[0043] In the GAN configuration, the generator neural network 121a is trained together with the discriminator neural network 121b based on multiple training examples, as shown in step (B) of FIG. 1. The discriminator neural network 121b may include multiple network layers, including, for example, one or more convolutional layers, leaky rectified linear unit (ReLU) layers, dense layers, and / or batch normalization layers. The network parameters of the discriminator neural network 121b are also included in the network parameters 124 and are repeatedly and alternately updated during the training process together with the network parameters of the generator neural network 121a. The discriminator neural network 121b outputs a prediction of whether the input to the discriminator neural network 121b is a real distribution map or a synthetic distribution map.

[0044] The training data used to update the network parameters 122 includes multiple training examples 110. Each training example includes a set of three distribution maps, including a low-resolution training image 110a showing fire distribution in an area, a reference training image 110b showing the characteristics of the same area, and a high-resolution training image 110c as the "real" label data. In the example shown in FIG. 1, the low-resolution training image 110a is an infrared image, the reference training image 110b is an aerial landscape image, and the high-resolution training image 110c is a fire distribution map. In general, similar to the description of the data types in the input data 140 and the output map 155, the low-resolution training image 110a, the reference training image 110b, and the high-resolution training image 110c can be other types of images showing fire distribution or land characteristics. For example, the low-resolution training image 110a can be a derived fire distribution map, the high-resolution training image 110c can be a high-resolution infrared map, and the reference training image 110b can be a vegetation index map.

[0045] 1, multiple training examples are collected and used by the training engine 122 to update the network parameters 124. In each training example, the low-resolution training image 110a, the reference training image 110b, and the high-resolution training image 110c correspond to the same geographic region. Furthermore, in each training example, the low-resolution training image 110a and the high-resolution training image 110c correspond to the same time point.

[0046] In some cases, both high-resolution and low-resolution satellite measurements are available for the same region at the same time during an active fire. These measurements can be collected as high-resolution training image 110c and low-resolution training image 110a, respectively. In some other cases, when only high-resolution satellite measurements are available for a region under an active fire, low-resolution training image 110a can be generated by downsampling the corresponding high-resolution training image 110c to create additional training examples.

[0047] In some implementations, further resampling can be performed to ensure that the low-resolution training images 110a in the training examples have the same spatial resolution as the low-resolution input images 140a in the input data, the reference training images 110b in the training examples have the same spatial resolution as the reference input images 140b in the input data, and the high-resolution distribution map 110c in the training examples has the same spatial resolution as the high-resolution synthetic image 155 in the output data.

[0048] During training, the training engine 122 updates the network parameters 124 of the generator neural network 121a and the discriminator neural network 121b based on multiple training samples 110. In some implementations, the training engine 122 can update the network parameters 124 by repeatedly performing two alternating operations.

[0049] In a first step, the training engine 122 updates a first set of network parameters (e.g., weights and bias parameters) of the discriminator neural network 121b based on a loss function that measures the difference between the predictions output by the discriminator and whether the input to the discriminator neural network includes a high-resolution training image 110c in one of the training examples 110 or a high-resolution synthetic image 155 output by the generator neural network.

[0050] In a second step, the training engine 122 updates a second set of network parameters (e.g., weights and bias parameters) of the generator neural network 121a based on a second loss function. The second loss function includes an adversarial loss that measures a decreasing function of the predictions output by the discriminator neural network, and the input to the discriminator neural network includes the synthetic images output by the generator neural network. A larger value of the adversarial loss indicates that the discriminator neural network predicts a decreasing probability that the input high-resolution images in the discriminator input data are "real" images.

[0051] Further details of the training process performed by the training engine 122 are described with reference to Figures 2 and 4. Briefly, an important feature of the training process is that during training, the input data to the discriminator neural network includes at least a portion of the training input data to the generator neural network. That is, in addition to the high-resolution synthetic image output by the generator neural network or the high-resolution training image 110c in one of the training examples, the input data to the discriminator neural network further includes the low-resolution training images 110a and / or the reference training images 110b.

[0052] Typically, the discriminator neural network of a GAN model aims to distinguish between the signal generated by the generator neural network and the real signal provided in the training examples, and therefore the input to the discriminator neural network usually includes only the signal output by the discriminator neural network or the real signal in the training examples. By incorporating a portion of the input signal to the generator neural network into the input signal to the discriminator neural network during training, the training engine 122 improves the performance and / or training efficiency of the GAN model, since the discriminator neural network is more effectively trained to determine whether the high-resolution image is a reasonable upscaling of the low-resolution input image by the reference input image. The resulting improvements include better prediction accuracy of the discriminator neural network, better quality in the high-resolution synthetic image generated by the generator neural network, and / or improved training efficiency, such as requiring fewer training examples to achieve a particular performance metric of the GAN model, thus reducing the computational power required to perform the training.

[0053] Furthermore, in many conventional GAN ​​models for generating high-resolution images, noise images are passed to the generator neural network. In contrast, in the GAN model provided herein, noise images do not have a static value epoch, so noise images cannot be considered as one of the generator input reference images.

[0054] In some implementations, the noise image is not passed through either the generator or the discriminator neural network. In some other implementations, the noise image may be included in the input to the discriminator neural network. When training the discriminator neural network on "real" examples, the training engine can create fake noise images to be paired with the examples. In yet some other implementations, for a "real" example, the training engine can use the same noise image that was most recently included in that example's corresponding input to the generator.

[0055] 2 illustrates a training process for learning network parameters of a GAN model. The process illustrated in FIG. 2 is conveniently described as being performed by one or more computer systems located at one or more locations. For example, the high-resolution image generation system 120 of FIG. 1, suitably programmed in accordance with the present specification, can include a training engine 122 for performing the process. The training engine can learn network parameters of a generator neural network 230 and a discriminator neural network 250 based on a plurality of training examples 210.

[0056] In the particular example shown in FIG. 2, each training example includes a low-resolution training image 210a having a first spatial resolution (e.g., showing the fire distribution of an area with active fire burning), a high-resolution training image 210c (e.g., showing the fire distribution of an area with a higher spatial resolution), and a reference training image 210b (e.g., showing features related to fire susceptibility of the area). The training engine uses the high-resolution input training image 210c as the “real” data label.

[0057] The training engine includes a generator input data generator 220 that generates training input data 225 for the generator neural network. For each training example, the training input data 225 for the generator neural network includes a low-resolution training image 210a and a reference training image 210b. The generator input data generator 220 can generate the training input 225 by combining, e.g., stacking, the low-resolution training image 210a with the reference training image 210b.

[0058] The training engine also includes a discriminator input data generator 240 that generates discriminator input data 245. The discriminator input data 245 is the input data to the discriminator neural network. An important feature of the training process is that the discriminator input data 245 includes at least a portion of the generator training input data 225, such as the low-resolution training images 210a and / or the reference training images 210b.

[0059] In some implementations, the discriminator input data 245 can include all of the generator training input data 225, i.e., both the low-resolution training images 210a and the reference training images 210b. The training engine can combine the generator training input data 225 with high-resolution discriminator input images to form the discriminator input data 245. The high-resolution discriminator input images can be the high-resolution synthetic images 235 output by the generator neural network or the high-resolution training images 210c ("real" images).

[0060] Specifically, the discriminator input data generator 240 may generate a first instance of the discriminator input data 245 by combining (e.g., by stacking) the high-resolution synthetic image 235 output by the generator neural network with the generator training input data (i.e., the low-resolution training images 210a and / or the reference training images 210b). The discriminator input data generator 240 may also generate a second instance of the discriminator input data 245 by combining (e.g., by stacking) the high-resolution training image 210c with the generator training input data (i.e., the low-resolution training images 210a and the reference training images 210b).

[0061] The training engine uses the discriminator neural network 250 to process the first and second instances of the discriminator input data 245, respectively, to generate a prediction 255 that distinguishes between the high resolution synthetic image 235 and the high resolution training image 210 (a "real" image) included in the discriminator input data 245. In one example, the prediction 255 may be a score that measures the probability that the discriminator input data 245 includes a "real" image. For example, an output score of "1" indicates that the discriminator neural network predicted the discriminator input data 245 to include the high resolution training image 210c, and an output score of "0" indicates that the discriminator neural network predicted the discriminator input data 245 to include the high resolution synthetic image 235.

[0062] The training engine may update the network parameters of the discriminator neural network 250 based on a first loss function that measures the prediction error of the discriminator neural network, i.e., a comparison difference between the outputted prediction 255 and whether the discriminator input data 245 includes the high-resolution synthetic image 235 (i.e., the input data of the discriminator neural network is a first instance of the discriminator input data) or the discriminator input data 245 includes the high-resolution training image 210c (i.e., the input data of the discriminator neural network is a second instance of the discriminator input data). The training engine may update the network parameters of the discriminator neural network 250 to minimize the first loss function using any suitable backpropagation-based machine learning technique, for example, by using an Adam or AdaGrad optimizer.

[0063] The training engine can update the network parameters of the generator neural network 230 based on a second loss function. The second loss function can include an adversarial loss that measures a decreasing function of the predictions 255 output by the discriminator neural network, and the discriminator input data includes the high-resolution synthetic images 235 output by the generator neural network. A larger value of the adversarial loss indicates that the discriminator neural network predicts a decreasing probability that the input high-resolution images in the discriminator input data 235 are “real” images. The training engine can update the network parameters of the generator neural network 230 to minimize the second loss function using any suitable backpropagation-based machine learning technique, for example, by using an Adam or AdaGrad optimizer.

[0064] The training engine can repeat the alternating update operations until a stopping criterion is reached, e.g., until the difference between the high-resolution synthetic image and the high-resolution training image is below a threshold. Both the network parameters of the generator neural network 230 and the discriminator neural network 250 improve over time during the repeated alternating training process.

[0065] 3 is a flow chart illustrating an exemplary process 300 for generating a synthetic signal using a GAN model. For convenience, the process 300 is described as being implemented by one or more computer systems located at one or more locations. For example, a signal generation system suitably programmed in accordance with this specification, such as the high-resolution image generation system 120 of FIG. 1, can perform the process 300.

[0066] In general, when performing the process 300, the system uses a GAN model to generate a synthetic signal based on an input signal. The input signal has one or more first characteristics. The synthetic signal has one or more second characteristics that are different from the one or more first characteristics. For example, the input signal can be a low-resolution input image having a first spatial resolution. The synthetic signal can be a high-resolution synthetic image having a second spatial resolution that is higher than the first spatial resolution.

[0067] The GAN model includes a generator neural network and a discriminator neural network. The system processes at least an input signal to generate output data including a composite signal. The generator neural network is trained with the discriminator neural network based on a plurality of training examples.

[0068] As shown in FIG. 3, the process 300 includes the following operations.

[0069] In operation 310, the system obtains generator input data including at least an input signal. The generator input data is input data to the generator neural network. The system can obtain the generator input data from various sources, such as a computing device, a data acquisition device, or a storage device, via a data transmission interface or network. The input signal has one or more first characteristics. The input signal can be an audio signal, an image, a video, or another type of signal.

[0070] In one exemplary implementation, the input signal is a low-resolution input image having a first spatial resolution that maps a fire spatial distribution of a fire burning area. An example of a data type of a low-resolution input image includes a low-resolution satellite infrared image of one or more bands. Another example of a low-resolution input image includes a fire distribution map derived from satellite infrared measurements. In an exemplary example, the first spatial resolution may be a resolution of about 400 m / pixel or less.

[0071] The generator input data may include other data, such as a reference signal, in addition to the input signal. The system may generate the generator input data by combining, e.g., concatenating or stacking, the input signal with the reference signal. For example, in the exemplary implementation described above, the generator input data may further include a reference input image of the same area. The reference input image may have a higher spatial resolution than the first spatial information and may include information indicative of features of the area. For example, the reference input image may be a satellite image in the visible band, an aerial photograph (e.g., collected by a drone), a labeled survey map, and a vegetation index map calculated from the visible and near-IR images. The reference input image may be collected before the fire and may provide information on fire susceptibility with respect to features such as topographical features (e.g., elevation, slope, rivers, coastlines, etc.), man-made structures (roads, buildings, land, etc.), vegetation index, and / or soil moisture of the same area at a higher resolution compared to the low-resolution input image. The reference input image may also be a post-fire image showing burn scars of the area, which also provides information indicative of fire susceptibility. The system can generate the generator input data by combining, for example, stacking a low-resolution input image with a reference input image.

[0072] At operation 320, the system processes the generator input data using a generator neural network of the GAN model to generate output data including a synthetic signal. The synthetic signal may be an audio signal, an image, a video, or another type of signal. The synthetic signal has one or more second characteristics that differ from one or more first characteristics of the input signal.

[0073] In an exemplary implementation, the input signal is a low-resolution input image having a first spatial resolution, and the synthetic signal is a high-resolution synthetic image having a second spatial resolution higher than the first spatial resolution. That is, the generator neural network generates a synthetic image having an upscaled resolution and providing finer spatial details compared to the input image. For example, the first spatial resolution of the low-resolution input image can be a resolution of around 400 m / pixel or less, and the second resolution of the high-resolution synthetic image can be a resolution of higher than 20 m / pixel.

[0074] When the generator input data includes additional data, the system can combine the input signal with the additional data (e.g., a reference signal) to form an input to a generator neural network and use the generator neural network to process the combined input to generate a composite signal. For example, the generator input data can include a low-resolution input image having a first spatial resolution (the input signal having a first characteristic) and a reference input image having a higher spatial resolution than the first spatial information (the reference signal). The system can use the generator neural network to process an input that combines the low-resolution input image and the reference input image to generate output data including a high-resolution composite image (the composite signal having a second characteristic).

[0075] The generator neural network used to process the generator input data may include multiple network layers, including, for example, one or more fully connected layers, convolutional layers, parametric rectified linear unit (PReLU) layers, and / or batch normalization layers. In some implementations, the generator neural network may include one or more residual blocks that include a skip connection layer.

[0076] The generator neural network includes a set of network parameters including weights and bias parameters of the network layers. The set of network parameters of the generator neural network is conveniently referred to as a first set of network parameters. The first set of network parameters is updated in a training process to minimize a loss characterization difference between the model's output and a desired output.

[0077] The training process for obtaining the network parameters is described with reference to FIG. 4. Briefly, the generator neural network is trained with the discriminator neural network of the GAN model based on a plurality of training examples. The discriminator neural network is configured to process discriminator input data including a discriminator input signal. The discriminator input signal has one or more second characteristics as a synthetic signal output by the generator neural network. For example, if the synthetic signal is a high-resolution synthetic image having a second spatial resolution, the discriminator input signal of the discriminator neural network is also a high-resolution image having a second spatial resolution. The discriminator neural network processes the second input to generate a prediction of whether the discriminator input signal in the discriminator input data is a real signal provided in one of the plurality of training examples or a synthetic signal output by the generator neural network.

[0078] An important feature of the training process is that the discriminator input data (input of the discriminator neural network) includes at least a portion of the data used as input to the generator neural network during the training process. For example, the training examples can include low-resolution training images having a first spatial resolution, high-resolution training images having a second spatial resolution higher than the first spatial resolution, and reference training images having a third spatial resolution higher than the first spatial resolution. The training input data for the generator neural network can include low-resolution training images and reference training images. By using the generator neural network to process the training input data for the generator neural network that combines the low-resolution training images and the reference training images, the system can generate a high-resolution synthetic image (synthetic signal) having a second spatial resolution. The system then generates the discriminator input data by combining the high-resolution discriminator input image with at least a portion of the training input data for the generator neural network, for example, including the low-resolution training images and / or the reference training images. The high-resolution discriminator input image can be a high-resolution synthetic image or a high-resolution training image output by the generator neural network. In this case, the portion of the training input data for the generator neural network that is also included in the discriminator input data includes the low-resolution training images and / or the reference training images.

[0079] At operation 330, the system outputs the composite signal to a device. In an example implementation, the composite signal is a high-resolution composite image, and the system can output the high-resolution composite image to a display device for displaying the image, to a storage device for storing the image data, or to a network device for transmitting the image data to a user device.

[0080] 4 is a flow chart illustrating a method 400 for training a GAN model. For convenience, the process 400 is described as being performed by one or more computer systems located at one or more locations. For example, a signal generation system suitably programmed in accordance with the present specification, such as the high-resolution image generation system 120 of FIG. 1, can include a training engine 122 for performing the process 400.

[0081] In general, when performing the process 400, the system trains a GAN model including a generator neural network and a discriminator neural network based on a plurality of training examples. The generator neural network is configured to process generator input data including an input signal to generate output data including a composite signal. The input signal has one or more first characteristics, and the composite signal has one or more second characteristics different from the first characteristics. The generator neural network can include multiple network layers including, for example, one or more fully connected layers, convolutional layers, parametric rectified linear unit (PReLU) layers, and / or batch normalization layers. In some implementations, the generator neural network can include one or more residual blocks including a skip connection layer.

[0082] The generator neural network includes a set of network parameters, including weight and bias parameters for the network layers, which are conveniently referred to as the first set of network parameters.

[0083] The discriminator neural network is configured to process discriminator input data that combines the discriminator input signal with at least a portion of the generator input data to generate a prediction of whether the discriminator input signal is a real signal provided in one of a plurality of training examples or a synthetic signal output by the generator neural network.

[0084] The discriminator neural network may include multiple network layers, including, for example, one or more convolutional layers, leaky rectified linear unit (ReLU) layers, dense layers, and / or batch normalization layers. The discriminator neural network also includes a set of network parameters, including weight and bias parameters of the network layers. The set of network parameters of the discriminator neural network is conveniently referred to as a second set of network parameters.

[0085] Similar to the GAN model described with reference to FIG. 3, the input signal for the generator neural network model has one or more first characteristics. For example, the input signal may be a low-resolution input image having a first spatial resolution. The synthetic signal output by the generator neural network has one or more second characteristics different from the one or more first characteristics. For example, the synthetic signal may be a high-resolution synthetic image having a second spatial resolution higher than the first spatial resolution. Similar to the synthetic signal, the discriminator input signal for the discriminator neural network also has one or more second characteristics. For example, the discriminator input signal may be a high-resolution image having a second spatial resolution.

[0086] During the training process, a first set of network parameters for the generator neural network can be iteratively and alternately updated with a second set of network parameters for the discriminator neural network, as detailed in the following operations.

[0087] In operation 410, the system acquires a plurality of training examples. In some implementations, the system can acquire data including the plurality of training examples from various sources, such as a computing device, a data acquisition device, or a storage device, via a data transmission interface or a network. Each training example includes at least a first training signal having one or more first characteristics and a second training signal having one or more second characteristics. In some other implementations, the system can generate the training examples at least in part based on a computation, for example, by performing data synthesis and / or processing.

[0088] The first training signal and the second training signal may be audio, video, image, or other types of signals. For example, the first training signal is a low-resolution training image having a first spatial resolution. The second training signal is a high-resolution image having a second spatial resolution higher than the first spatial resolution.

[0089] In one exemplary implementation, the first training signal is a low-resolution training image that maps the spatial distribution of fires in a fire burn area at a first spatial resolution (e.g., a resolution of about 400 m / pixel or less), and the second training signal is a high-resolution image that maps the spatial distribution of fires in the same area at an increased spatial resolution (e.g., a resolution higher than 20 m / pixel).

[0090] An example of a low-resolution training image includes a low-resolution satellite infrared image in one or more bands. Another example of a low-resolution training image includes a fire distribution map derived from satellite infrared measurements. In some cases, both high-resolution and low-resolution satellite measurements are available for the same area at the same time during an active fire. These measurements can be collected as a high-resolution training image (second training signal) and a low-resolution training image (first training signal), respectively. In some other cases, only high-resolution satellite measurements are available for an area under an active fire. In these cases, to create additional training examples, the system can generate low-resolution training images by downsampling the corresponding high-resolution training images.

[0091] In some implementations, the generator neural network is configured to process inputs including the input signal and the reference signal to generate a composite signal. Thus, in addition to the first training signal and the second training signal, each training example further includes a reference training signal having one or more third characteristics. In the above-mentioned exemplary implementation, the reference training signal can be a reference training image having a third spatial resolution higher than the first spatial resolution. The reference training image can include information indicative of characteristics of the region. For example, the reference input image can be a satellite image in the visible band, an aerial photograph (e.g., collected by a drone), a labeled survey map, and a vegetation index map calculated from the visible and near-IR images. The reference training image can be collected before a fire and can provide information on fire susceptibility with respect to features such as topographical features (e.g., elevation, slope, rivers, coastlines, etc.), man-made structures (roads, buildings, land, etc.), vegetation index, and / or soil moisture of the same region at a higher resolution compared to the low-resolution input image. The reference input image can also be a post-fire image showing burn scars of the region, which also provides information indicative of fire susceptibility.

[0092] Based on each of the multiple training examples, the system repeatedly performs operations 420-450.

[0093] At operation 420, the system generates training input data for the generator neural network. The training input data for the generator neural network is input data to the generator neural network based on the training examples and includes at least the first training signal in the training examples. In some implementations, when the training examples also include a reference training signal, the training input data for the generator neural network further includes the reference training signal. The system can generate the training input data for the generator neural network by combining, e.g., concatenating, the first training input signal with the reference training signal.

[0094] In an example implementation, the training examples include low-resolution training images (first training signal) having a first spatial resolution, high-resolution training images (second training signal) having a second spatial resolution higher than the first spatial resolution, and reference training images (reference training signal) having a third spatial resolution higher than the first spatial resolution. The system can generate training input data for the generator neural network by combining the low-resolution training images with the reference training images.

[0095] At operation 430, the system processes the training input data using a generator neural network to generate a synthetic signal, the synthetic signal having one or more second characteristics. For example, the system can process training input data for a generator neural network that combines the low-resolution training images and the reference training images to generate a high-resolution synthetic image having a second spatial resolution.

[0096] In operation 440, the system generates a first instance of discriminator input data by combining the first instance of the discriminator input signal with at least a portion of the training input data for the generator neural network, and processes the first instance of the discriminator input data using the discriminator neural network to generate a first prediction. The first instance of the discriminator input signal is a composite signal output by the generator neural network.

[0097] In an exemplary implementation, the training examples include a low-resolution training image (first training signal) having a first spatial resolution, a high-resolution training image (second training signal) having a second spatial resolution higher than the first spatial resolution, and a reference training image (reference training signal) having a third spatial resolution higher than the first spatial resolution. By using a generator neural network to process training input data for the generator neural network that combines the low-resolution training images and the reference training images, the system can generate a high-resolution synthetic image (synthetic signal) having the second spatial resolution. The system then generates a first instance of discriminator input data by combining the high-resolution synthetic image output by the generator neural network with at least a portion of the training input data for the generator neural network, for example, including the low-resolution training images and / or the reference training images. For example, in some cases, a portion of the training input data for the generator neural network may include only the low-resolution training images. In some other cases, a portion of the training input data for the generator neural network may include both the low-resolution training images and the reference training images.

[0098] In operation 450, the system generates a second instance of the discriminator input data by combining the second instance of the discriminator input signal with a portion of the training input data for the generator neural network, and processes the second instance of the discriminator input data using the discriminator neural network to generate a second prediction. The second instance of the discriminator input signal is a second training signal in the training examples.

[0099] In the same example implementation described in operation 440, the first instance of the discriminator input data includes high-resolution synthetic data output by the generator neural network, and a portion of the training input data for the generator neural network includes, for example, low-resolution training images and / or reference training images. The system can generate a second instance of the discriminator input data by combining the high-resolution training images in the training examples with the same portion of the training input data for the generator neural network included in the first instance of the discriminator input data. That is, the second instance of the discriminator input data includes high-resolution training images combined with a portion of the training input data for the generator neural network including, for example, low-resolution training images and / or reference training images.

[0100] In operation 460, the system updates a first set of network parameters of the discriminator neural network based on a first loss function. The first loss function may measure a comparison difference between a prediction output by the discriminator neural network and whether the discriminator input signal in the discriminator input data is the second training signal or a synthetic signal.

[0101] The discriminator neural network is configured to generate a prediction of whether a discriminator input signal in the discriminator input data is a "real" signal provided in the training examples (i.e., the second training signal) or a synthetic signal output by the generator neural network. For example, the discriminator neural network can output a prediction score ranging from 0 to 1 that predicts the probability that the discriminator input signal is a "real" signal (e.g., a prediction score of "1" for predicting that the discriminator input signal is a real signal and a prediction score of "0" for predicting that the discriminator input signal is a synthetic signal). To update the network parameters (e.g., weight and bias parameters) of the discriminator neural network, the system can calculate a prediction error of the discriminator neural network as a first loss function and update the network parameters of the discriminator neural network to minimize the first loss function. The system can update the network parameters of the discriminator neural network based on the first loss function using any suitable backpropagation-based machine learning technique, for example, using an Adam or AdaGrad optimizer.

[0102] At operation 470, the system updates a second set of network parameters of the generator neural network based on a second loss function. The second loss function may include an adversarial loss that measures a decreasing function of the prediction output by the discriminator neural network while the discriminator input signal included in the discriminator input data is a synthetic signal output by the generator neural network. A larger value of the adversarial loss indicates that the discriminator neural network predicts a decreasing probability that the discriminator input signal in the discriminator input data is a "real" signal.

[0103] In some implementations, after the network parameters of the discriminator neural network are updated (as in operation 460), the system can use the updated discriminator neural network to generate predictions again based on the synthetic signal output by the generator neural network as the discriminator input signal in the discriminator input data. The system can then update the network parameters of the generator neural network to minimize a second loss function that includes a measure of a decreasing function of the predictions output by the generator neural network. The system can update the network parameters of the generator neural network based on the second loss function using any suitable backpropagation-based machine learning technique, for example, using an Adam or AdaGrad optimizer.

[0104] Step 460 and operation 470 for updating the network parameters of the discriminator neural network and the generator neural network can be repeated alternately until a stopping criterion is reached, e.g., when the difference between the synthetic signal and the "real" signal falls below a threshold. Both the network parameters of the generator neural network and the discriminator neural network improve over time during the repeated alternating training process.

[0105] 5 is a block diagram of an exemplary computer system 500 that can be used to perform the operations described above. The system 500 includes a processor 510, a memory 520, a storage device 530, and an input / output device 540. Each of the components 510, 520, 530, and 540 can be interconnected using, for example, a system bus 550. The processor 510 can process instructions for execution within the system 500. In one implementation, the processor 510 is a single-threaded processor. In another implementation, the processor 510 is a multi-threaded processor. The processor 510 can process instructions stored in the memory 520 or the storage device 530.

[0106] The memory 520 stores information within the system 500. In one implementation, the memory 520 is a computer-readable medium. In one implementation, the memory 520 is a volatile memory unit. In another implementation, the memory 520 is a non-volatile memory unit.

[0107] The storage device 530 can provide mass storage for the system 500. In one implementation, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 can include, for example, a hard disk device, an optical disk device, a storage device shared over a network by multiple computing devices (e.g., a cloud storage device), or some other large capacity storage device.

[0108] The input / output device(s) 540 provide input / output operations for the system 500. In one implementation, the input / output device(s) 540 may include one or more network interface devices, such as Ethernet cards, serial communication devices, such as RS-232 ports, and / or wireless interface devices, such as 502.11 cards. In another implementation, the input / output device(s) may include driver devices configured to receive input data and send output data to other input / output devices, such as keyboards, printers, and display devices 560. However, other implementations, such as mobile computing devices, mobile communication devices, set-top boxes, television client devices, etc., may also be used.

[0109] Although an exemplary processing system is illustrated in FIG. 5, implementations of the subject matter and functional operations described herein can be implemented in other types of digital electronic circuitry, or computer software, firmware, or hardware, or in combinations of one or more of them, including the structures disclosed herein and structural equivalents thereof.

[0110] The term "configured" is used herein in connection with systems and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when in operation, causes the system to perform the operation or action. A system of one or more computer programs configured to perform a particular operation or action means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0111] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, or a combination of one or more of them, including the structures disclosed herein and their structural equivalents. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or to control the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode information to be transmitted to a receiving device suitable for execution by a data processing apparatus.

[0112] The term "data processing apparatus" refers to data processing hardware and encompasses any kind of apparatus, device, and machine for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or further include special purpose logic circuitry, such as, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). An apparatus may optionally include, in addition to hardware, code that creates an execution environment for a computer program, such as code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0113] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including standalone programs or modules, components, subroutines, or other units suitable for use in a computing environment. A program may correspond to a file in a file system, but need not. A program may be stored in one or more scripts stored in a portion of a file that holds other programs or data, for example, in a markup language document, in a single file dedicated to the program in question, or in multiple cooperating files, for example, files that store one or more modules, subprograms, or portions of code. A computer program may be deployed to be executed on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communication network.

[0114] As used herein, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components and installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine. In other cases, multiple engines may be installed and run on the same computer(s).

[0115] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data to generate output. The processes and logic flows may also be performed by special purpose logic circuitry, such as, for example, an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0116] A computer suitable for executing a computer program can be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. In general, the central processing unit receives instructions and data from a read-only memory, or a random access memory, or both. The essential elements of a computer are a central processing unit for implementing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special purpose logic circuitry. In general, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from, or transfer data to, or both. However, a computer need not have such devices. Additionally, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few.

[0117] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory including semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0118] To provide for interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can be used to provide interaction with the user as well, for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, for example, sending a web page to a web browser on the user's device in response to a request received from the web browser. The computer can also interact with the user by sending text messages or other types of messages to a personal device, such as a smartphone running a messaging application, and receiving a response message from the user in return.

[0119] A data processing device for implementing machine learning models may also include, for example, dedicated hardware accelerator units for handling the typical, computationally intensive parts of machine learning training or production, i.e., inference, workloads.

[0120] The machine learning model can be implemented and deployed using a machine learning framework, for example, the TensorFlow framework.

[0121] An embodiment of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network ("LAN") and a wide area network ("WAN"), e.g., the Internet.

[0122] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data, e.g., HTML pages, to a user device for the purpose of, e.g., displaying data to and receiving user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., a result of user interaction, can be received at the server from the device.

[0123] Although the present specification contains details of many individual implementations, these are not to be construed as limiting the scope of any feature or the scope of what may be claimed, but as descriptions of features specific to particular embodiments. Certain features described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in a certain combination, and may even be initially claimed as such, one or more features from the claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0124] Similarly, although operations are depicted in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown, or in a sequential order, or that all of the illustrated operations be performed, to achieve desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.

[0125] Thus, certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some instances, the actions recited in the claims can be performed in a different order and still achieve desirable results. Moreover, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

[0126] References Ledig et al., "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network," CoRR, September 2016, https: / / arxiv.org / abs / 1609.04802

[0127] Rakotonirina et al., "ESRGAN+: Further Improving Enhanced Super-Resolution Generative Adversarial Network," CoRR, January 2020, https: / / arxiv.org / abs / 2001.08073

[0128] Wang et al., "ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks," CoRR, September 2018, https: / / arxiv.org / abs / 1809.00219

Claims

1. 1. A computer-implemented method comprising: obtaining generator input data including at least one input signal having one or more first characteristics; processing the generator input data using a trained generator neural network based on a plurality of training examples with a discriminator neural network to generate output data including a synthetic signal having one or more second characteristics, the discriminator neural network being configured to process the discriminator input data to combine the discriminator input signal having the one or more second characteristics with at least a portion of the generator input data to generate a prediction of whether the discriminator input signal is a real signal provided in one of a plurality of training examples or a synthetic signal output by the generator neural network; outputting the composite signal to a device; the input signal is a low resolution input image having a first spatial resolution; the composite signal is a high-resolution composite image having a second spatial resolution greater than the first spatial resolution; the discriminator input signal is a high resolution image having the second spatial resolution; The method further comprising: obtaining the plurality of training examples, each of the training examples including at least a low-resolution training image having the first spatial resolution and a high-resolution training image having the second spatial resolution; Based on each of the plurality of training examples, generating training input data for the generator neural network by including at least the low-resolution training images in the training examples, and processing the training input data using the generator neural network to generate the output data comprising the high-resolution synthetic image; generating a first instance of the discriminator input data by combining the high resolution synthetic image with at least a portion of the training input data for the generator neural network, and processing the first instance of the discriminator input data using the discriminator neural network to generate a first prediction; generating a second instance of the discriminator input data by combining the high resolution training images in the training examples with the portion of the training input data for the generator neural network, and processing the second instance of the discriminator input data using the discriminator neural network to generate a second prediction; updating a first set of network parameters of the discriminator neural network based on a first loss function that measures a difference between the predictions output by the discriminator neural network and whether the discriminator input data includes the high-resolution training images in the training examples or the high-resolution synthetic images output by the generator neural network; updating a second set of network parameters of the generator neural network based on a second loss function including an adversarial loss that measures a decreasing function of the prediction output by the discriminator neural network while the discriminator input signal included in the discriminator input data is the high resolution synthetic image output by the generator neural network; and the generator input data further comprises a reference input image having a third spatial resolution higher than the first spatial resolution; each of the training examples further includes a reference training image having the third spatial resolution; generating the first instance of the discriminator input data includes combining the high-resolution synthetic image with the low-resolution training images and the reference training images in the training examples; The method of claim 1, wherein generating the second instance of the discriminator input data includes combining the high resolution training images with the low resolution training images and the reference training images in the training examples.

2. Obtaining the plurality of training examples 2. The method of claim 1 , comprising, for each of one or more of the plurality of training examples, generating the low-resolution training images from the high-resolution training images by downsampling the high-resolution training images from the second spatial resolution to the first spatial resolution.

3. The method of claim 1 , wherein the reference input image is associated with a different image modality than the low-resolution input image.

4. The low-resolution input image is a low-resolution distribution map showing a fire distribution of a fire burning area, the reference input image is a distribution map indicating characteristics of the region; The method of claim 1 , wherein the high resolution synthetic image is a high resolution synthetic distribution map showing fire distribution in the area.

5. The method of claim 4 , wherein the reference input image depicts a pre-fire fuel distribution of the region.

6. The method of claim 4 , wherein the reference input image depicts a ground topography of the area.

7. 1. A computer-implemented method comprising: training a neural network based on a plurality of training examples, the neural network comprising: a generator neural network configured to process generator input data including at least one input signal having one or more first characteristics to generate a synthetic signal having one or more second characteristics different from the one or more first characteristics; and a discriminator neural network configured to process discriminator input data combining a discriminator input signal having the one or more second characteristics with at least a portion of the generator input data to generate a prediction of whether the discriminator input signal is a real signal provided in one of a plurality of training examples or a synthetic signal output by the generator neural network; the input signal has a first spatial resolution, and the composite signal is a high resolution composite image having a second spatial resolution greater than the first spatial resolution; The training, acquiring the plurality of training examples, each of the training examples including at least a first training signal having the one or more first characteristics, a second training signal having the one or more second characteristics, and a reference training image, the first training signal being a low resolution training image having the first spatial resolution, the second training signal being a high resolution training image having the second spatial resolution, and the reference training image having a third spatial resolution; Based on each of the plurality of training examples, generating training input data for the generator neural network by including at least the first training signal in the training examples, and processing the training input data using the generator neural network to generate the synthetic signal; generating a first instance of the discriminator input data by combining a first instance of a discriminator input signal with at least a portion of the training input data for the generator neural network, the first instance of the discriminator input signal being the synthetic signal; and processing the first instance of the discriminator input data using the discriminator neural network to generate a first prediction, the generating the first instance of the discriminator input data comprising combining the high resolution synthetic image with the low resolution training images and the reference training images in the training examples; generating a second instance of the discriminator input data by combining a second instance of the discriminator input signal with the portion of the training input data for the generator neural network, the second instance of the discriminator input signal being the second training signal in the training examples; and processing the second instance of the discriminator input data using the discriminator neural network to generate a second prediction, the generating the second instance of the discriminator input data comprising combining the high resolution training image with the low resolution training image and the reference training image in the training examples. updating a first set of network parameters of the discriminator neural network based on a first loss function that measures a difference between the prediction output by the discriminator neural network and whether the discriminator input signal in the discriminator input data is the second training signal or the synthetic signal; updating a second set of network parameters of the generator neural network based on a second loss function that includes at least the first prediction; and and performing the above.

8. Obtaining the plurality of training examples 8. The method of claim 7, comprising, for each of one or more of the plurality of training examples, generating the low-resolution training images from the high-resolution training images by downsampling the high-resolution training images from the second spatial resolution to the first spatial resolution.

9. The method of claim 7 , wherein the reference training images are associated with a different image modality than the low-resolution training images.

10. The low-resolution training image is a low-resolution distribution map showing a fire distribution of a fire burning area, the reference training image is a distribution map of features of the region; The method of claim 7 , wherein the high resolution synthetic image is a high resolution synthetic distribution map showing fire distribution in the area.

11. The method of claim 10 , wherein the reference training images show a pre-fire fuel distribution of the region.

12. The method of claim 10 , wherein the reference training images show a ground topography of the area.

13. 1. A system comprising: one or more computers; and one or more storage devices that store instructions, which when executed by the one or more computers, cause the one or more computers to perform operations including the method of any one of claims 1 to 12.

14. One or more computer readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations including the method of any one of claims 1 to 12.

Images