Super-Resolution Satellite Images via Neural Networks with Physical Constraints

Abstract

A method, computer program product, and system are provided for generating a higher-resolution geospatial image. A processor receives a time series of spatial data images at a first resolution. The processor determines physical laws applicable to the spatial data images from a plurality of spatial data images. The processor subdivides each of the plurality of spatial data images into a plurality of small spatial region images. The processor solves each of the physical laws in each of the small spatial region images. The processor trains a neural network to apply each of the physical laws to each of the small spatial region images by applying a local physical law loss function. The processor determines the most applicable local physical law based on the difference between a small spatial region image and an image predicted for that region by the physical law. The processor generates a second image at a higher resolution than the first resolution.

Description

Technical Field

[0001] The present invention generally relates to the field of image processing, and more specifically to generating higher resolution satellite images via a neural network with physical constraints.

[0002] Satellite images are images captured at high altitudes and are usually captured by orbiting satellites or suborbital satellites (e.g., high-altitude aircraft) that can capture large-area visualizations at once. Satellite images can include images in the visible spectrum using built-in cameras or other sensors such as infrared image sensors to provide up-to-date images related to global phenomena such as weather. A neural network is a directed graph model that represents the complex relationship between inputs and outputs and can predict results even when given partial or incomplete inputs.

Summary of the Invention

[0003] Embodiments of the present invention provide a method, computer program product, and system for generating higher resolution geospatial images. A processor receives a time series of spatial data images at a first resolution. The processor determines physical laws applicable to the spatial data images from a plurality of spatial data images. The processor subdivides each of the plurality of spatial data images into a plurality of small spatial region images. The processor solves each physical law in each of the small spatial region images. The processor trains a neural network to apply each physical law to each of the small spatial region images by applying a local physical law loss function. The processor determines the most applicable local physical law based on the difference between the small spatial region image and the image predicted for that region by the physical law. The processor generates a second image at a higher resolution than the first resolution.

[0004] According to an aspect of the present invention, a method executed by a computer includes receiving a first plurality of time-series spatial data images at a first resolution, determining one or more physical laws applicable to one or more of the plurality of spatial data images from among the plurality of spatial data images, subdividing each of the one or more spatial data images into a plurality of small spatial region images, solving each of the one or more physical laws in each of the small spatial region images to determine physical law coefficients for the small spatial region images, training a neural network to apply each of the physical laws to each of the small spatial region images by applying a local physical law loss function, determining the most applicable local physical law based on the difference between the small spatial region image and the image predicted for the region by the physical law, and generating a second image with a higher resolution than the first resolution by applying the neural network for the most applicable local physical law to the first plurality of time-series images.

[0005] According to another aspect of the present invention, there is provided a computer program product comprising one or more computer-readable storage media and program instructions stored on the one or more computer-readable storage media, the program instructions comprising: program instructions for receiving a first plurality of spatio-temporal data images at a first resolution; program instructions for determining one or more physical laws applicable to one or more of the plurality of spatio-temporal data images from one or more of the plurality of spatio-temporal data images; program instructions for subdividing each of the one or more spatio-temporal data images into a plurality of small spatio-region images; program instructions for solving each of the one or more physical laws in each of the small spatio-region images to determine physical law coefficients for the small spatio-region images; program instructions for training a neural network to apply each of the physical laws to each of the small spatio-region images by applying a local physical law loss function; program instructions for determining the most applicable local physical law based on the difference between a small spatio-region image and an image predicted for that region by the physical law; and program instructions for generating a second image at a higher resolution than the first resolution by applying the neural network for the most applicable local physical law to the first plurality of time-series images. A computer program product is provided.

[0006] According to another aspect of the present invention, there is provided a computer system, comprising: one or more computer processors; one or more computer-readable storage media; and program instructions stored on a computer-readable storage media for execution by at least one of the one or more processors, the program instructions including: program instructions for receiving a first plurality of spatio-temporal data images at a first resolution; program instructions for determining one or more physical laws applicable to one or more of the plurality of spatio-temporal data images; program instructions for subdividing each of the one or more spatio-temporal data images into a plurality of small spatio-region images; program instructions for solving each of the one or more physical laws in each of the small spatio-region images to determine physical law coefficients for the small spatio-region images; program instructions for training a neural network to apply each of the physical laws to each of the small spatio-region images by applying a local physical law loss function; program instructions for determining the most applicable local physical law based on the difference between the small spatio-region image and the image predicted for that region by the physical law; and program instructions for generating a second image at a higher resolution than the first resolution by applying the neural network for the most applicable local physical law to the first plurality of time-series images.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

[0008] The present invention can be a system, method, or computer program product, or any combination thereof, at any possible technical detail level of integration. The computer program product may include a computer-readable storage medium (or multiple computer-readable storage media) having computer-readable program instructions for causing a processor to implement aspects of the present invention.

[0009] The computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. The computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or structures engraved in grooves with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed to be a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through an optical fiber cable), or an electrical signal transmitted through an electrical wire.

[0010] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to respective computing / processing devices, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within each computing / processing device.

[0011] Computer-readable program instructions for performing the operations of the present invention may be in any combination of source code or object code written in one or more programming languages, including assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuits, or object-oriented programming languages such as Smalltalk(R), C++, and procedural programming languages such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer as a stand-alone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can utilize the state information of the computer-readable program instructions to execute the computer-readable program instructions to customize the electronic circuit in order to implement aspects of the present invention.

[0012] Aspects of the present invention will be described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0013] These computer-readable program instructions, when executed via the processor of a computer or other programmable data processing apparatus, may create means for implementing the functions / acts specified in one or more blocks of a flowchart, a block diagram, or both, thereby causing the machine to perform the functions / acts specified in one or more blocks of a flowchart, a block diagram, or both. These computer-readable program instructions may also be stored in a computer-readable storage medium having instructions for implementing the functions / acts specified in one or more blocks of a flowchart, a block diagram, or both, the computer-readable storage medium being included in a manufactured article that causes a computer, a programmable data processing apparatus, or other device or combination thereof to function in a particular manner.

[0014] These computer-readable program instructions may also be loaded onto a computer, other programmable apparatus, or other device to create a computer-implemented process such that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / acts specified in one or more blocks of a flowchart, a block diagram, or both, thereby causing a series of operational steps to be performed on the computer, other programmable apparatus, or other device.

[0015] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram can represent a module, segment, or portion of instructions that include one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions represented by the blocks may occur in a different order than shown in the drawings. For example, two blocks shown in succession may actually be performed as one step, or may be executed simultaneously, substantially simultaneously, in a partially or wholly temporally overlapping manner, or the blocks may sometimes be executed in the reverse order depending on the functionality involved. It should also be noted that each block of the block diagram or flowchart diagram, or combinations of blocks of the block diagram or flowchart diagram, or both, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or that executes a combination of dedicated hardware and computer instructions.

[0016] Here, the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing a computing environment, generally designated 100, according to one embodiment of the present invention. The computing environment 100 includes a computing device connected to a network 120. The computing device 110 includes an image enhancement program 112, a physical module 113, a neural network 114, input image data 116, and enhanced image data 118.

[0017] In various embodiments of the present invention, computing device 110 is a computing device that can be a stand-alone device, a server, a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), or a desktop computer. In another embodiment, computing device 110 represents a computing system that utilizes clustered computers and components in order to function as a single pool of seamless resources. Generally, computing device 110 can be any computing device or combination of devices having access to physical module 113, neural network 114, input image data 116, and enhanced image data 118, and can execute image enhancement program 112. Computing device 110 may include internal and external hardware components, as shown and described in further detail with respect to FIG. 4.

[0018] In this exemplary embodiment, the image enhancement program 112, the physical module 113, the neural network 114, the input image data 116, and the enhanced image data 118 are stored in the computing device 110. However, in other embodiments, the image enhancement program 112, the physical module 113, the neural network 114, the input image data 116, and the enhanced image data 118 may be stored externally and accessed through a communication network such as the network 120. The network 120 can be, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of both, and may include any wired, wireless, fiber optic or any other connection known in the art. Generally, the network 120 can be any combination of connections and protocols that support communication between the computing device 110 and any other device (not shown) connected to the network 120, according to a preferred embodiment of the present invention.

[0019] In various embodiments, the image enhancement (IE) program 112 receives input image data 116 that includes various satellite images or simulations for that area. The images are stored in chronological order from when the image was captured. In some embodiments, the input image data 116 may be arranged in any order. Further, the images may be captured optically via an image sensor or via various other sensors such as RADAR, gravitational field, magnetic field strength, or thermal images. The input image data 116 also includes a layer or other embedding in the image that conveys any physical / meteorological or other geographical location information data that occurred in the area at the time the sequence was captured, such as air temperature or humidity. The input image data 116 may also include geospatial non-weather data such as wildfire smoke density. Those skilled in the art will recognize that the input image data 116 can be an array of any geospatial data that is faithful to one or more physical laws, as described herein.

[0020] In various embodiments, the IE program 112 includes a physics module 113 that includes various models and mathematical formulas that can model various physical properties / weather and other phenomena captured in the input image data 116. For example, an area where a forest fire that generates a smoke plume is occurring is captured in the input image data 116 along with geospatial data (e.g., smoke concentration per square mile) indicating the smoke concentration in various sections of the input image data 116. The physics module 113 includes the following models for convection-diffusion: ∂ t C+∇*Cu=∇(K*∇C)+S [E.1]

[0021] In Equation E.1, C is the concentration of the plume in a given section, S is any source or sink in the section, K is the diffusion coefficient for the section, and u is the velocity component of the medium (e.g., wind speed and direction).

[0022] In various embodiments, the IE program 112 separates the input image data 116 into a series of primary sections corresponding to portions of the captured satellite image. In one scenario, each section corresponds to each pixel of the geospatial data, where the image can clearly resolve the physical characteristics within that particular image. The image can be at a certain resolution, but the actual physical characteristics governing the system behavior may be captured at a different resolution. Based on the resolution of various weather or other geospatial data points in the input image data 116, each measurement or pixel is separated into primary sections. For example, if a satellite image includes temperature readings every square kilometer in the image, the primary sections are set to a corresponding square kilometer size to match the resolution of the geospatial data. If a large number of geospatial data points exist in the input image data 116, the IE program 112 sets the primary section size to the lowest resolution of the data points (e.g., the minimum area per measurement). In some scenarios, the IE program 112 sets the primary sections to a fixed size according to a grid such as latitude and longitude coordinates, or can have any shape such as a polygon. The boundaries of the regions are calculated based on the dominant physical characteristics in that area and thresholds defined by the user. Such an example can be the identification of hurricane or strong wind conditions, where the user can define the thresholds that define the wind conditions. In another embodiment, the computer determines the boundaries based on the intensity of the physical characteristics. One such example is the separation of strong wind and high precipitation areas during a hurricane, where the physical laws determining the intensity can be different and the two areas can be generated based on observations and thresholds. The two areas can be separated, and the boundary layer can be a combination of two physical laws operating simultaneously (strong wind and high precipitation acting simultaneously), and the boundary can change in the next image when one physical law becomes dominant over the other.

[0023] In various embodiments, the physical module 113 includes various constraint-based models for phenomena that can be captured by the input image data 116 and other observable events. For example, the physical module 113 may include one or more of the following physical property models from the table below:

[0024]

Table 1

[0025] In the above table, "Physical property type" indicates the natural laws that can exist or affect the information represented in the input image data 116. "Variable" indicates the measured variable in each subsection. For example, when solving "Aerodynamics", ρ is the density of the gas for the pixels in the input image data 116, and p is the pressure of the gas for the pixels in the input image data 116. The PDE equation is a partial differential equation that must be solved for the model to be satisfied. When each equation is solved, the resulting "variable" that gives the solution is the predicted value of the physical model for the pixels or portions of the super-resolution image. For example, when the IE program 112 solves for the variables of the aerodynamics model for the pixels in the super-resolution image, the IE program 112 identifies the predicted "density" or "pressure" measurements for the pixels of the subsection. The "Equation of state / Constitutive equation" section indicates the constants and other constraints that must be satisfied for the PDE equation to be correct when solved.

[0026] In a scenario where the order of the images in the input image data 116 is not sequential, the IE program 112 determines the rate of change between all the images in the input image data 116. The amount of change from each comparison image is used to sort the images in the correct order. If multiple physical laws apply to a section or sub-section, the IE program 112 ranks the sections / sub-sections of the images based on the rate of change. In some scenarios, the ranking of the sections / sub-sections may be provided as input to the neural network 114.

[0027] In some scenarios, the original image data set from the satellite may not be of sufficient size and quality to train the neural network, and simulations based on the dominant physical laws can be generated using the physical laws, and the training images can be generated at any spatial and temporal resolution. In such scenarios, the IE program 112 generates simulated images based on physical simulations using the physical module 113, using the simulated images as training data for the neural network 114.

[0028] In various embodiments for each primary section, the IE program 112 generates two or more subsections along with predicted values for each subsection. As described herein, based on the training and application of neural networks to geospatial data, the IE program 112 generates a higher-resolution image, enhanced image data 118, based on the geospatial data of the input image data 116. When the original geospatial imaging in the input image data 116 is captured at a certain resolution, the IE program 112 sometimes generates a higher-resolution geospatial image of the input image data 116, sometimes referred to as a super-resolution image of the input image data 116. In previous solutions, super-resolution was achieved through pixel interpolation that infers higher-resolution pixels based on the average values of adjacent pixels. Embodiments of the present invention provide new techniques and systems for applying neural networks to simulate subsection pixels. By comparing the output of the neural network with the constraints of various physical models determined by the physical module 113, the IE program 112 generates a higher-resolution geospatial image of the input image data 116.

[0029] In various embodiments, the network learns to reconcile physical laws at different spatial and temporal scales, such as identifying and implementing seasonal changes that may exist in an image, for example, greening or defoliation of trees. In another embodiment, the network can detect changes in an image, such as damage due to strong winds, and this damage can be quantified based on the area of the change.

[0030] In various embodiments, neural network 114 can be any general adversarial neural network or GAN. A GAN includes two neural networks that are "adversarial" or antagonistic to each other, and the output layers of both are compared in a loss function. In this arrangement, neural network 114 includes a trained neural network for enhancing the resolution in input image data 116. In this setup, neural network 114 includes two neural networks that form the adversarial aspect of the GAN, namely a generator network and a discriminator network. The generator network is trained to generate higher-resolution image data. Based on a set of known satellite data and images, the generator is supervised and trained to predict super-resolution images. The discriminator network generates a loss measurement of the prediction. The discriminator network is trained to distinguish incorrect or inconsistent predictions of super-resolution images. When the generator produces implausible or incorrect results, the discriminator penalizes the generator by increasing the loss value for the prediction. When both the generator network and the discriminator network generate a super-resolution image that minimizes the loss function, IE program 112 evaluates the super-resolution image for consistency using the physical laws identified by physical module 113 applicable to a sub-section of the image.

[0031] In various embodiments, the neural network 114 includes a loss function that minimizes both typical losses seen in GANs and physical inconsistency measurement criteria derived from the physical module 113. Based on applicable physical laws for the subsection, the IE program 112 determines a physical inconsistency measurement criterion that compares the predicted pixels of the neural network to the applicable physical laws identified for the section or subsection. The physical inconsistency measurement criterion is determined by the physical module 113 for each pixel by comparing the pixel physical prediction to other pixel values for the same type of prediction surrounding the candidate pixel. For example, if the super-resolution image is increasing the pixel density of the smoke concentration, the physical module 113 compares the predicted density, ρ, for a pixel to other pixels in adjacent subsections.

[0032] For each connection section, the physical module 113 enforces energy, flux, and mass conservation between the sections. For any adjacent predictions that do not maintain energy, flux, and mass conservation between the sections, the physical module 113 determines a physical inconsistency measurement criterion for each pixel based on the lack of conservation when comparing the subsection to all adjacent subsections for the overall section including the subpixel and all larger overall sections. In various embodiments, the IE program 112 ensures that adjacent subsections respect and maintain any applicable physical laws to adjacent subsections.

[0033] Referring to FIG. 3, FIG. 3 shows an exemplary visualization 300 of an input satellite image 310 and a higher resolution image 320 generated by the IE program 112 as enhanced image data 118. The input satellite image 310 shows exemplary input image data 116 including six pixels of the observed conditions in the satellite image. For example, each pixel may represent the cloud density or wind speed in a given area for each pixel. As described herein, the IE program 112 generates a higher resolution image 320 by separating each pixel in the input satellite image 310 into a number of subsections. In this example, the IE program 112 performs pixel doubling of the original input satellite image 310, and four pixels or subsections are generated in the higher resolution image 320 for each pixel in the input satellite image 310. For each subsection or subpixel of the higher resolution image 320 relative to the input satellite image 310, the IE program 112 determines the pixel value for the subsection. A neural network 114 as described herein generates predictions for each subpixel based on a trained GAN network based on minimizing the loss function between the generative neural network and the discriminative neural network. Subsequently, the physical module 113 solves one or more constraint-based physical models to predict each subpixel. If the predictions contain losses in the conservation of energy, flux, and mass when compared to adjacent pixels, the physical module 113 generates a physical inconsistency measurement criterion for the subpixel, which is included in the loss function of the neural network 114. Next, the IE program 112 selects the predictions from the neural network 114 that minimize the loss function of the GAN, along with the included physical inconsistency measurement criteria derived from the physical module 113.

[0034] Returning to FIG. 1, the IE program 112 generates sub-pixels in the improved image data 118 based on sub-pixels that minimize the loss function of the neural network 114, along with the additional physical mismatch measurement criteria derived from the physical module 113. Once the improved image data 118 is first generated, the IE program 112 evaluates a subsection of the improved image data 118 to determine any dominant or influential physical laws that may affect the input image data 116. Based on the physical mismatch measurement criteria that exceed the threshold, the IE program 112 identifies each physical law applied to the adjacent subsections. If the law is not applied to the adjacent subsections, the IE program 112 applies the law to the adjacent subsections. Next, the IE program 112 repeats the above process and reapplies the new law for the subsections by the physical module 113. When the physical mismatch measurement criteria are reduced by expanding the effect of the physical model, the IE program 112 expands the application of the physical law to the adjacent subsections. The IE program 112 repeats these substitutions until the optimal physical mismatch measurement criteria are achieved.

[0035] Figure 2 shows the operation process, generally indicated at 200, of the Image Enhancement (IE) program 112. In process 202, the IE program 112 receives an image sequence of spatial data. The image sequence captures spatial data for a given area over a period of time. In process 204, the IE program 112 separates each image into sections. In some scenarios, the IE program 112 generates a section for each data element or pixel of the spatial data included in the image. In other scenarios, the IE program 112 separates the image into larger sections based on the coordinates or positions of specific spatial data clusters. For example, an area may have cloud cover spatial data due to cloud systems passing overhead, while another area of the image may have little or no cloud cover spatial data because weather systems do not affect that area. Therefore, the IE program 112 may generate larger sections for the unaffected areas rather than for each pixel in order to conserve computing resources.

[0036] In process 206, the IE program 112 identifies one or more physical laws applicable to each section of the image sequence generated in process 204. As described herein, the physical module 113 of the IE program 112 includes various PDE models for different physical laws. By evaluating the various sections and the predicted subsections of the input image data 116 and the enhanced image data 118, the IE program 112 determines which PDE model correlates with the data points seen in the section of the input image. If the physical model does not produce an output or a statistically significant output (i.e., the impact or output of the model is below a threshold), the IE program 112 determines that the model is not applicable to the section or subsection. However, if the physical model produces a larger result with the same data, the IE program 112 identifies that the model for the physical law is applicable.

[0037] In process 208, the IE program 112 subdivides each section into two or more subsections. For example, for each pixel or data element in an image, the IE program 112 may double the pixel count and generate 2×2 pixel blocks for each 1×1 pixel. As previously explained, since the IE program 112 does not utilize any pixel interpolation as used by conventional image enhancement solutions, the IE program 112 can select any magnification factor. In process 210, the IE program 112 trains the neural network 114 based on the identified applicable physical laws from process 206. In some scenarios, the IE program 112 is monitored by the user while training the neural network 114 with a training data set. In other scenarios, the IE program 112 performs unsupervised training in the neural network 114. In such scenarios where adversarial neural networks are deployed, the IE program 112 may automatically retrain the neural network 114 if the loss function exceeds a threshold for a predetermined number of pixels (e.g., if more than half of the sub-pixels have a minimized loss function higher than a certain error rate).

[0038] In process 212, the IE program 112 determines the most applicable physical law or model for each section. Based on the determined loss function and the adversarial neural network 114 for each subsection, the physical law and corresponding network model that generate the minimum loss function are determined to be the most applicable. Once the most applicable law or model is determined, the IE program 112 generates pixels for the subsections based on the selected physical law. Thus, in process 214, the IE program 112 generates a higher resolution image for each subsection and increases the pixel count of the input image sequence.

[0039] Figure 4 shows a block diagram 400 of components of a computing device 110, according to an exemplary embodiment of the present invention. It should be understood that Figure 4 provides merely an illustration of one implementation form and does not imply any limitation regarding the environment in which different embodiments may be implemented. Many modifications may be made to the depicted environment.

[0040] The computing device 110 includes a communication fabric 402 that provides communication between a computer processor 404, a memory 406, a persistent storage 408, a communication unit 410, and an input / output (I / O) interface 412. The communication fabric 402 can be implemented in any architecture designed to pass data or control information or both between a processor (such as a microprocessor, communication and network processor, etc.), system memory, peripheral devices, and any other hardware components within the system. For example, the communication fabric 402 can be implemented with one or more buses.

[0041] The memory 406 and the persistent storage 408 are computer-readable storage media. In this embodiment, the memory 406 includes a random access memory (RAM) 414 and a cache memory 416. Generally, the memory 406 can include any suitable volatile or non-volatile computer-readable storage media.

[0042] The image enhancement program 112, physical module 113, neural network 114, input image data 116, and enhanced image data 118 are stored in the persistent storage 408 for execution or access or both by one or more of the respective computer processors 404 via one or more memories of the memory 406. In this embodiment, the persistent storage 408 includes a magnetic hard disk drive. As an alternative to or in addition to the magnetic hard disk drive, the persistent storage 408 can include a solid state hard drive, semiconductor memory device, read only memory (ROM), erasable programmable read only memory (EPROM), flash memory, or any other computer readable storage medium capable of storing program instructions or digital information.

[0043] Also, the medium used by the persistent storage 408 may be removable. For example, a removable hard drive may be used for the persistent storage 408. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer to another computer readable storage medium that is also part of the persistent storage 408.

[0044] The communication unit 410 provides communication with other data processing systems or devices, including the resources of the network 120 in these examples. In these examples, the communication unit 410 includes one or more network interface cards. The communication unit 410 can communicate through the use of either or both physical and wireless communication links. The image enhancement program 112, physical module 113, neural network 114, input image data 116, and enhanced image data 118 may be downloaded to the persistent storage 408 via the communication unit 410.

[0045] The I / O interface 412 enables input and output of data with other devices that may be connected to the computing device 110. For example, the I / O interface 412 may provide a connection to an external device 418 such as a keyboard, keypad, touch screen, or other suitable input device, or a combination thereof. The external device 418 may also include a portable computer-readable storage medium, such as a thumb drive, portable optical or magnetic disk, and memory card. Software and data used to implement embodiments of the present invention, such as the image enhancement program 112, physical module 113, neural network 114, input image data 116, and enhanced image data 118, may be stored on such a portable computer-readable storage medium and loaded into the persistent storage 408 via the I / O interface 412. The I / O interface 412 is also connected to a display 420.

[0046] The display 420 provides a mechanism for displaying data to the user and may be, for example, a computer monitor or a television screen.

[0047] The programs described herein are identified based on the uses implemented in particular embodiments of the present invention. Nevertheless, any particular program terminology herein is used for convenience only, and thus, it should be understood that the present invention should not be limited to use in any particular use that is identified or suggested or both by such terminology.

Claims

1. A method executed by a computer, comprising: Receiving a plurality of first time-series spatial data images at a first resolution; Determining, from one or more of the plurality of spatial data images, one or more physical laws applicable to the one or more spatial data images; Subdividing each of the one or more spatial data images into a plurality of small spatial region images; Solving each of the one or more physical laws in each of the small spatial region images to determine physical law coefficients for the small spatial region images; Training a neural network to apply each of the physical laws to each of the small spatial region images by applying a local physical law loss function; Determining the most applicable local physical law based on the difference between the small spatial region image and the image predicted for the region by the physical law; Generating a second image with a higher resolution than the first resolution by applying the neural network for the most applicable local physical law to the plurality of first time-series images.

2. The method executed by the computer further comprises: Determining at least one pixel of the second higher-resolution image based on the neural network; Determining a physical inconsistency measurement criterion for the at least one pixel of the second higher resolution based on the neural network; Applying the physical inconsistency measurement criterion to a loss function of the neural network.

3. The method executed by the computer according to claim 2, wherein the neural network is an adversarial neural network.

4. The method executed by the computer according to claim 2, wherein the subdivision in the second higher-resolution image is compared with adjacent subdivisions for preservation of the applicable local physical law.

5. The method executed by the computer according to claim 4, wherein the value for the adjacent subdivision of the applicable local physical law ensures preservation of the applicable local physical law between the adjacent subdivisions.

6. The method executed by a computer according to claim 5, wherein the neural network is penalized for a decision that does not guarantee preservation of the applicable local physical laws between the adjacent subdivisions.

7. The method executed by a computer according to claim 6, wherein energy, mass, or flux is conserved between the adjacent subdivisions.

8. A computer program product, comprising one or more computer-readable storage media and program instructions stored on the one or more computer-readable storage media, the program instructions being program instructions for receiving a first plurality of spatio-temporal data images at a first resolution, program instructions for determining one or more physical laws applicable to the one or more spatio-temporal data images from one or more of the plurality of spatio-temporal data images, program instructions for subdividing each of the one or more spatio-temporal data images into a plurality of small spatio-regional images, program instructions for solving each of the one or more physical laws in each of the small spatio-regional images to determine physical law coefficients for the small spatio-regional images, program instructions for training a neural network to apply each of the physical laws to each of the small spatio-regional images by applying a local physical law loss function, program instructions for determining the most applicable local physical law based on the difference between the small spatio-regional image and the image predicted for the region by the physical law, program instructions for generating a second image with a higher resolution than the first resolution by applying the neural network for the most applicable local physical law to the first plurality of spatio-temporal images.

9. The program instructions further include program instructions for determining at least one pixel of the second higher-resolution image based on the neural network, and program instructions for determining a physical inconsistency measurement criterion for the at least one pixel of the second higher resolution based on the neural network. Program instructions for applying the physical inconsistency measurement criteria to the loss function of the neural network, and, the computer program product according to claim 8. **Claim 10** The computer program product according to claim 9, wherein the neural network is an adversarial neural network. **Claim 11** The computer program product according to claim 9, wherein the subdivision in the second, higher-resolution image is compared with adjacent subdivisions for preservation of the applicable local physical laws. **Claim 12** The computer program product according to claim 11, wherein values for the adjacent subdivisions of the applicable local physical laws guarantee preservation of the applicable local physical laws between the adjacent subdivisions. **Claim 13** The computer program product according to claim 12, wherein the neural network is penalized for a determination that it does not guarantee preservation of the applicable local physical laws between the adjacent subdivisions. **Claim 14** The computer program product according to claim 13, wherein energy, mass, or flux is conserved between the adjacent subdivisions. **Claim 15** A computer system, One or more computer processors, One or more computer-readable storage media, and Program instructions stored in the computer-readable storage media for execution by at least one of the one or more processors, the stored program instructions including: Program instructions for receiving a first plurality of spatio-temporal data images at a first resolution, Program instructions for determining one or more physical laws applicable to the one or more spatio-temporal data images from one or more of the plurality of spatio-temporal data images, Program instructions for subdividing each of the one or more spatio-temporal data images into a plurality of small spatio-regional images, Program instructions for solving each of the one or more physical laws in each of the small spatio-regional images to determine physical law coefficients for the small spatio-regional images, and Program instructions for training a neural network to apply each of the physical laws to each of the small spatio-regional images by applying a local physical law loss function. Program instructions for determining the most applicable local physical law based on the difference between the small spatial region image and the image predicted for the region by the physical law, Program instructions for generating a second image with a higher resolution than the first resolution by applying the neural network for the most applicable local physical law to the first plurality of time-series images, **Claim 16** The program instructions further Program instructions for determining at least one pixel of the second higher-resolution image based on the neural network, Program instructions for determining a physical inconsistency measurement criterion for the at least one pixel of the second higher resolution based on the neural network, Program instructions for applying the physical inconsistency measurement criterion to the loss function of the neural network, the computer system according to claim 15. **Claim 17** The neural network is an adversarial neural network, the computer system according to claim 16. **Claim 18** The subdivision in the second higher-resolution image is compared with adjacent subdivisions for the preservation of the applicable local physical law, the computer system according to claim 16. **Claim 19** The value for the adjacent subdivision of the applicable local physical law guarantees the preservation of the applicable local physical law between the adjacent subdivisions, the computer system according to claim 18. **Claim 20** The neural network is penalized for the determination that it does not guarantee the preservation of the applicable local physical law between the adjacent subdivisions, the computer system according to claim 19.

Images