High resolution system, high resolution method, model generation device, inference device, model generation method, inference method, model generation program and inference program

Abstract

To enable a model for making an image high resolution to be generated even in the case that there are not many usable high resolution images.SOLUTION: An image generation device 200 uses a reference image having resolution higher than in a target image to generate a plurality of simulation images having resolution higher than in the target image, and deteriorates each of the plurality of simulation images to generate a plurality of deteriorated images. A learning device 300 learns a plurality of pairs of the deteriorated images and the simulation images to generate learned models. An inference device 400 inputs the target image and acquires an inference image by using the learned models.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to increasing the resolution of remote sensing images. [Background technology]

[0002] Ground sensors have been used to assess damage caused by disasters such as floods, heavy rain, and earthquakes. Methods using ground sensors are useful for assessing the damage caused to specific areas such as houses and power plants. However, when damage occurs over a wide area, it is difficult to obtain surface information using ground sensors, such as which areas have suffered the most damage.

[0003] In recent years, analytical methods using remote sensing data have been proposed as a way to grasp the extent of disaster damage over a wide area. Remote sensing uses aircraft, UAVs (drones), optical satellites, synthetic aperture radar satellites, etc. Remote sensing images obtained by onboard thermal infrared sensors are used as a means of monitoring various objects on the Earth's surface.

[0004] In addition, in order to improve the surface resolution of images, there is a technology that increases the resolution of images using a high-resolution model (super-resolution model) trained through machine learning. Patent Document 1 discloses the following technology for generating a high-resolution model. In this technology, a high-resolution image that serves as a teacher image and a low-resolution image that is generated from the high-resolution image as an input image are prepared, and a high-resolution model is generated by training these as a data pair. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2023-150145 Summary of the Invention [Problem to be solved by the invention]

[0006] The technology of Patent Document 1 uses high-resolution images acquired by a remote sensing technique that can acquire images with higher resolution than images from a target remote sensing technique to which inference processing is applied as follows: A degradation process that simulates a satellite image is applied to the high-resolution image using target remote sensing technique information. This generates a simulated degraded image. Then, the high-resolution image and the simulated degraded image are paired as data for learning. However, when using thermal infrared remote sensing images with the technology of Patent Document 1, there are issues such as the fact that the number of available learning images and their source images is smaller than that of optical images, and that the image degradation process is unique compared to that of optical images.

[0007] The present disclosure aims to enable the generation of a model for increasing the resolution of an image even when there are few high-resolution images available. [Means for solving the problem]

[0008] The resolution enhancement system of the present disclosure comprises: a resolution enhancement system that receives a target image obtained by a first remote sensing technique as an input and obtains an inference image having a higher resolution than the target image, an image enhancement unit that uses a reference image obtained by a second remote sensing technique and having a higher resolution than the target image to generate a plurality of simulated images having a higher resolution than the target image; an image degradation unit that degrades each of the plurality of simulated images to generate a plurality of degraded images; a model generation unit that generates a trained model by learning a plurality of pairs of the degraded image and the simulated image; an inference unit that receives the target image as an input and obtains the inferred image using the trained model; Equipped with. [Effects of the Invention]

[0009] According to the present disclosure, even when there are few high-resolution images (reference images) available, it is possible to generate a model (trained model) that increases the resolution of images. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a configuration diagram of a resolution increasing system 100 according to a first embodiment. [Figure 2] FIG. 1 is a configuration diagram of an image generating device 200 according to a first embodiment. [Figure 3] FIG. 2 is a functional configuration diagram of an image generating device 200 according to the first embodiment. [Figure 4] FIG. 2 is a configuration diagram of a learning device 300 according to the first embodiment. [Figure 5] FIG. 2 is a functional configuration diagram of a learning device 300 according to the first embodiment. [Figure 6] FIG. 1 is a configuration diagram of an inference device 400 according to the first embodiment. [Figure 7] FIG. 2 is a functional configuration diagram of an inference device 400 according to the first embodiment. [Figure 8] 3 is a flowchart of an image generation process according to the first embodiment. [Figure 9] FIG. 2 is a diagram showing the processing configuration of an image extension unit 210 according to the first embodiment. [Figure 10] 4 is a graph showing an example of a curve representing a temperature change in one day in the first embodiment. [Figure 11] FIG. 3 is a diagram showing the processing configuration of an image degradation unit 220 according to the first embodiment. [Figure 12] 4 is a flowchart of a learning process according to the first embodiment. [Figure 13] FIG. 1 is a conceptual diagram of a neural network according to the first embodiment. [Figure 14] 4 is a flowchart of an inference process according to the first embodiment. [Figure 15] FIG. 2 is a diagram showing an example of the functional configuration of a learning device 300 according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] In the embodiments and drawings, the same or corresponding elements are denoted by the same reference numerals. The description of elements denoted by the same reference numerals as those already described will be omitted or simplified as appropriate. Arrows in the drawings primarily indicate the flow of data or the flow of processing.

[0012] Embodiment 1 The resolution increasing system 100 will be described with reference to FIGS.

[0013] ***Configuration Description*** The configuration of a resolution increasing system 100 will be described with reference to FIG. The resolution increasing system 100 includes an image generating device 200, a learning device 300, and an inference device 400. The image generation device 200 and the learning device 300 are collectively referred to as a model generation device 101.

[0014] The configuration of the image generating device 200 will be described with reference to FIG. The image generating device 200 is a computer that includes hardware such as a processor 201, a memory 202, an auxiliary storage device 203, a communication device 204, and an input / output interface 205. These pieces of hardware are connected to one another via signal lines.

[0015] The processor 201 is a processor of the image generating device 200 . A processor is an integrated circuit (IC) that performs computational processing and controls other hardware. For example, a processor can be a CPU, a DSP, or a GPU. IC is an abbreviation for Integrated Circuit. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor. GPU is an abbreviation for Graphics Processing Unit.

[0016] The memory 202 is the memory of the image generating device 200 . The memory is a volatile or non-volatile storage device. The memory is also called a primary storage device or a main memory. For example, the memory is a RAM. The data stored in the memory 202 is saved in the secondary storage device 203 as needed. RAM is an abbreviation for Random Access Memory.

[0017] The auxiliary storage device 203 is an auxiliary storage device for the image generating device 200 . The auxiliary storage device is a non-volatile storage device. For example, the auxiliary storage device is a ROM, a HDD, a flash memory, or a combination thereof. Data stored in the auxiliary storage device 203 is loaded into the memory 202 as needed. ROM is an abbreviation for Read Only Memory. HDD is an abbreviation for Hard Disk Drive.

[0018] The communication device 204 is a communication device of the image generation device 200 . The communication device is a receiver and a transmitter. For example, the communication device is a communication chip or a NIC. Communication for the image production device 200 is performed using the communication device 204. NIC is an abbreviation for Network Interface Card.

[0019] The input / output interface 205 is the input / output interface 205 of the image generating device 200 . The input / output interface is a port to which an input device and an output device are connected. For example, the input / output interface is a USB terminal, the input device is a keyboard and a mouse, and the output device is a display. Input and output of the image generation device 200 is performed using the input / output interface 205. USB is an abbreviation for Universal Serial Bus.

[0020] The image generation device 200 comprises elements such as an image enhancement unit 210 and an image degradation unit 220. These elements are realized in software.

[0021] The auxiliary storage device 203 stores an image generation program for causing the computer to function as the image enhancement unit 210 and the image degradation unit 220. The image generation program is loaded into the memory 202 and executed by the processor 201. The auxiliary storage device 203 also stores an OS. At least a part of the OS is loaded into the memory 202 and executed by the processor 201. The processor 201 executes an image generation program while running the OS. OS is an abbreviation for Operating System.

[0022] The data of the image generation program (input data, output data, etc.) is stored in the storage unit 290. The memory 202 functions as the storage unit 290. However, a storage device such as the auxiliary storage device 203, a register in the processor 201, or a cache memory in the processor 201 may function as the storage unit 290 instead of or together with the memory 202.

[0023] The image generating program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or a flash memory.

[0024] FIG. 3 shows the functional configuration of the image generating device 200. Data such as the reference image 281 , the reference information 282 , the extended information 283 , the first remote sensing information 284 , the simulated image 291 , and the degraded image 292 are stored in a storage unit 290 . Each data will be described later.

[0025] The configuration of the learning device 300 will be described with reference to FIG. The learning device 300 is a computer that includes hardware such as a processor 301, a memory 302, an auxiliary storage device 303, a communication device 304, and an input / output interface 305. These pieces of hardware are connected to each other via signal lines.

[0026] The processor 301 is the processor of the learning device 300 . The memory 302 is a memory of the learning device 300. The data stored in the memory 302 is saved in the auxiliary storage device 303 as needed. Auxiliary storage device 303 is an auxiliary storage device of learning device 300. Data stored in auxiliary storage device 303 is loaded into memory 302 as needed. The communication device 304 is a communication device for the learning device 300. The communication device 304 is used for communication of the learning device 300. The input / output interface 305 is an input / output interface for the learning device 300. The input / output of the learning device 300 is performed using the input / output interface 305.

[0027] The learning device 300 includes elements such as a data acquisition unit 310 and a model generation unit 320. These elements are realized by software.

[0028] The auxiliary storage device 303 stores a learning program for causing the computer to function as the data acquisition unit 310 and the model generation unit 320. The learning program is loaded into the memory 302 and executed by the processor 301. The auxiliary storage device 303 also stores an OS. At least a part of the OS is loaded into the memory 302 and executed by the processor 301. The processor 301 executes the learning program while running the OS.

[0029] The learning program data is stored in the memory unit 390. The memory 302 functions as the storage unit 390. However, a storage device such as the auxiliary storage device 303, a register in the processor 301, or a cache memory in the processor 301 may function as the storage unit 390 instead of the memory 302 or together with the memory 302.

[0030] The learning program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or flash memory.

[0031] FIG. 5 shows the functional configuration of the learning device 300. Data such as the simulated image 291, the degraded image 292, and the trained model 391 are stored in a storage unit 390. Each data will be described later.

[0032] The configuration of the inference device 400 will be described with reference to FIG. The inference device 400 is a computer that includes hardware such as a processor 401, a memory 402, an auxiliary storage device 403, a communication device 404, and an input / output interface 405. These pieces of hardware are connected to one another via signal lines.

[0033] Processor 401 is the processor of inference device 400 . Memory 402 is the memory of inference device 400. Data stored in memory 402 is saved in auxiliary storage device 403 as needed. Auxiliary storage device 403 is an auxiliary storage device for reasoning device 400. Data stored in auxiliary storage device 403 is loaded into memory 402 as needed. Communication device 404 is a communication device for inference device 400. Communication for inference device 400 is performed using communication device 404. Input / output interface 405 is an input / output interface for inference device 400. Input / output to / from inference device 400 is performed using input / output interface 405.

[0034] The inference device 400 comprises elements such as a data acquisition unit 410 and an inference unit 420. These elements are realized by software.

[0035] The auxiliary storage device 403 stores an inference program for causing the computer to function as a data acquisition unit 410 and an inference unit 420. The inference program is loaded into the memory 402 and executed by the processor 401. The auxiliary storage device 403 also stores an OS. At least a part of the OS is loaded into the memory 402 and executed by the processor 401. The processor 401 executes an inference program while running the OS.

[0036] The data of the inference program is stored in the memory unit 490. The memory 402 functions as the storage unit 490. However, a storage unit such as the auxiliary storage unit 403, a register in the processor 401, or a cache memory in the processor 401 may function as the storage unit 490 instead of the memory 402 or together with the memory 402.

[0037] The inference program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or flash memory.

[0038] FIG. 7 shows the functional configuration of the inference device 400. Data such as the trained model 391, the target image 481, and the inferred image 491 are stored in a storage unit 490. Each data will be described later.

[0039] ***Explanation of Operation*** The procedure of operation of the resolution enhancement system 100 corresponds to a resolution enhancement method. The operation procedure of the resolution enhancement system 100 corresponds to the processing procedure of the resolution enhancement program. The resolution enhancement program includes an image generation program, a learning program, and an inference program. The image generation program and the learning program are collectively referred to as a model generation program.

[0040] The high-resolution method is a method in which a target image 481 is input to obtain an inferred image 491. The target image 481 is an image obtained by the first remote sensing technique. For example, the target image 481 is an image captured by exposing to light with a wavelength in the infrared band. The inference image 491 is an image with a higher resolution than the target image 481 .

[0041] Examples of remote sensing techniques are imaging from flying objects (aerial photography) and imaging from artificial satellites. Examples of air vehicles include helicopters, UAVs, and aircraft. UAV is an abbreviation for Unmanned Aerial Vehicle. Examples of satellites include optical satellites and SAR satellites. Optical satellites are satellites equipped with optical sensors. SAR satellites are satellites equipped with SAR. SAR is an abbreviation for Synthetic Aperture Radar.

[0042] The image generation process of the high resolution method will be described with reference to FIG. In step S210, the image extension unit 210 generates a plurality of simulated images 291 using the reference image 281.

[0043] The reference image 281 is an image obtained by a second remote sensing technique. The resolution of the reference image 281 is higher than the resolution of the target image 481.

[0044] The second remote sensing technique is a remote sensing technique that can obtain images with higher resolution than the images obtained by the first remote sensing technique.

[0045] Simulated image 291 corresponds to an image obtained by a second remote sensing technique.

[0046] The plurality of simulated images 291 are generated as follows. First, the image extension unit 210 acquires data such as a reference image 281, reference information 282, and multiple pieces of extension information 283. For example, a user inputs this information into the image generating device 200, and the image extension unit 210 receives the input data. The reference information 282 indicates the meteorological environment at the reference date and time. For example, the reference information 282 indicates at least one of the atmospheric characteristics and the temperature of features at the reference date and time. The reference date and time is the date and time when the second remote sensing method for obtaining the reference image 281 was performed. The extended information 283 indicates the weather environment at the extended date and time. For example, the extended information 283 indicates at least one of the atmospheric characteristics and the temperature of features at the extended date and time. The extended date and time is a date and time that is different from the reference date and time. Then, the image extension unit 210 generates an image corresponding to the reference image at the extension date and time using the reference image 281, the reference information 282, and the extension information 283 for each extension information 283. The generated image is the simulated image 291. Specifically, the image extension unit 210 generates the simulated image 291 by processing the reference image 281 based on the difference between the weather environment at the reference date and time and the weather environment at the extension date and time.

[0047] Step S210 will be described in detail with reference to FIG. One or more simulated images 291 are generated by dilation processing on the reference image 281 . Examples of the extended processes are the atmospheric characteristics simulation process 211 and the temperature simulation process 212. Either the atmospheric characteristics simulation process 211 or the temperature simulation process 212 may be executed, or both the atmospheric characteristics simulation process 211 and the temperature simulation process 212 may be executed.

[0048] The atmospheric characteristics simulation process 211 will now be described. First, the image extension unit 210 acquires a reference image 281, reference information 282, and extension information 283. For example, the reference information 282 and extension information 283 are acquired from external data. The reference information 282 indicates atmospheric characteristics at the reference date and time. The extended information 283 indicates atmospheric characteristics at the extended date and time. For example, the reference information 282 and the extended information 283 indicate atmospheric characteristics such as the radiance and atmospheric transmittance of the atmosphere itself in the thermal infrared wavelength range. The reference information 282 and the extended information 283 are obtained from external meteorological data. Alternatively, the reference information 282 and the extended information 283 are calculated based on meteorological information obtained from meteorological data. The image extension unit 210 then subtracts the atmospheric characteristics at the reference date and time and adds the atmospheric characteristics at the extended date and time to the reference image 281 to simulate the atmospheric characteristics at the extended date and time, thereby generating a simulated image 291.

[0049] The temperature simulation process 212 will now be described. First, the image extension unit 210 acquires the reference image 281, the reference information 282, and the extension information 283 as described above. The reference information 282 indicates the temperature at the reference date and time. The extended information 283 indicates the temperature at the extended date and time. Specifically, the reference information 282 indicates the temperature of the feature at the reference date and time, and the extended information 283 indicates the temperature of the feature at the extended date and time. For example, the reference information 282 is obtained from external data. The external data indicates a global temperature map for each season, temperature changes of the feature at each time zone, etc. For example, the extended information 283 is calculated by simulating the temperature of the feature at the extended date and time using the reference information 282.

[0050] A curve (graph) showing the temperature change over one day is shown in Figure 10. If the reference date and the extended date are the same, the curve of the temperature change over one day can be estimated, and the temperature at the reference time (1) and the temperature at the extended time (2) can be read from the estimated curve. Similarly, by using the temperature change curve for the same time of day throughout the year, it is also possible to simulate the temperature at the same time on another day.

[0051] It is possible to achieve even higher accuracy by using highly accurate land cover classification as external data. For example, it is possible to estimate the emissivity of land objects using highly accurate land cover classification and reflect this in temperature simulations. For example, it is possible to determine that areas that have been identified as man-made using highly accurate land cover classification and have a temperature above a certain level are self-heating, and to simulate the temperature while still self-heating.

[0052] Returning to FIG. 8, step S220 will be described. In step S220, the image degradation unit 220 degrades each of the plurality of simulated images 291 to generate a plurality of degraded images 292.

[0053] The resolution of the degraded image 292 is lower than the resolution of the simulated image 291. For example, the resolution of the degraded image 292 is approximately the same as the resolution of the target image 481.

[0054] The multiple degraded images 292 are generated as follows. First, the image degradation unit 220 acquires the simulated image 291 and the first remote sensing information 284. For example, a user inputs the first remote sensing information 284 into the image generation device 200, and the image degradation unit 220 receives the input first remote sensing information 284. The first remote sensing information 284 is information about the first remote sensing technique. Then, for each simulated image 291 , the image degrading unit 220 generates a degraded image 292 using the simulated image 291 and the first remote sensing information 284 . Specifically, the image degrading unit 220 processes the simulated image 291 based on the first remote sensing information 284 to generate a degraded image 292 .

[0055] Step S220 will be described in detail with reference to FIG. The degraded image 292 is generated by performing a degradation process on the simulated image 291 . Examples of degradation processes are resolution degradation process 221, noise degradation process 222, and compression degradation process 223. Resolution degradation process 221 is a process for lowering resolution. Noise degradation process 222 is a process for adding noise. Compression degradation process 223 is a process for compressing an image. Any one or two of the resolution degradation process 221, noise degradation process 222, and compression degradation process 223 may be executed, or all of the resolution degradation process 221, noise degradation process 222, and compression degradation process 223 may be executed. Furthermore, degradation processing may be performed based on at least one of PSF characteristics, GSD characteristics, sensitivity performance, noise characteristics, data compression, and quantization. PSF is an abbreviation for Point Spread Function. GSD is an abbreviation for Ground Sampling Distance.

[0056] An example of degradation processing using the PSF of the first remote sensing technique will be described. In remote sensing using optical sensors, the PSF is determined as an index of resolution by combining multiple degradation models for the optical performance of the sensor, the aperture shape, the effects of turbulence and image streaming, etc. Similarly, the PSF is determined as an index of resolution for the reference image 281, taking into account the optical performance of the sensor, the influence of disturbances, and the like. By applying the first remote sensing PSF to the simulated image 291 using the first remote sensing technique PSF and the reference image PSF, it is possible to perform degradation processing to convert the resolution of the simulated image 291 to a resolution equivalent to that of the target image 481. The PSF degradation is realized by a method of multiplying the PSF in frequency space or by a method of convolving the PSF with the simulated image 291, for example.

[0057] When checking the PSF for each band, the PSF may differ for each band. For example, aliasing may occur in the multispectral image because the MTF of the multispectral band is higher than the PSF of the panchromatic band. By incorporating such a difference in PSF into the degradation process, it is possible to reproduce aliasing in the simulated image 291.

[0058] An example of degradation processing using the GSD of the first remote sensing technique will be described. The degradation process compares the GSD of the reference information 282 with the GSD of the first remote sensing technique, and simulates the GSD using a downsampling technique such as bilinear or bicubic interpolation.

[0059] An example of degradation processing using the sensitivity performance of the first remote sensing technique will now be described. Degradation processing is performed on thermal infrared images, and adds the following information to the thermal infrared image. The added information is, for example, the NETD, which is calculated by taking into account the temperature of the observed feature, the total temperature integration time, sensor performance, and noise characteristics due to internal radiation. Examples of noise characteristics include shot noise, 1 / f noise, and shading. NETD is an abbreviation for Noise Equivalent Temperature Difference.

[0060] An example of a deterioration process simulating the atmosphere will be described. If the second remote sensing technique is imaging from within the atmosphere, the reference image 281 does not contain noise caused by the atmosphere. On the other hand, if the first remote sensing technique is observation from a high altitude (for example, satellite observation), the target image 481 contains noise such as a reduction in image contrast caused by the atmosphere, and occlusion and disturbance. The degradation process generates information on image degradation caused by the atmosphere based on a publicly available atmospheric model, and adds the image degradation information to the image, thereby generating a degraded image 292 that simulates a satellite image.

[0061] The learning process for the high resolution method will be described with reference to FIG. In step S310, the data acquisition unit 310 acquires a plurality of pairs of the degraded image 292 and the simulated image 291.

[0062] For example, the data acquisition unit 310 receives a plurality of pairs of the degraded image 292 and the simulated image 291 from the image generation device 200 .

[0063] A plurality of pairs of the degraded image 292 and the simulated image 291 serve as learning data. In supervised learning, the simulated image 291 serves as teacher data (correct answer data) for the degraded image 292.

[0064] In step S320, the model generation unit 320 generates a trained model 391 by training a plurality of pairs of the degraded image 292 and the simulated image 291.

[0065] That is, the model generation unit 320 learns the combination of the degraded image 292 and the simulation image 291 .

[0066] The trained model 391 is a trained model for inferring an inferred image 491 corresponding to the target image 481.

[0067] A known algorithm can be used as the learning algorithm. Examples of learning algorithms are supervised learning, unsupervised learning, reinforcement learning, etc. Also, neural networks can be applied to learning algorithms.

[0068] In supervised learning, the same subject must appear in the degraded image 292 and the simulated image 291.

[0069] In unsupervised learning, the degraded image 292 and the simulated image 291 do not need to contain the same subject.

[0070] For example, the model generation unit 320 learns a simulated image 291 corresponding to the degraded image 292 through supervised learning using a neural network model. Supervised learning is a learning method in which a set of input data and result data is used as learning data, the characteristics of the learning data are learned, and the result data is inferred from the input data. The neural network uses a combination of the degraded image 292 and the simulated image 291 as learning data and learns a trained model 391 for inferring the simulated image 291 corresponding to the degraded image 292 through supervised learning.

[0071] A neural network consists of an input layer consisting of multiple neurons, an intermediate layer (hidden layer) consisting of multiple neurons, and an output layer consisting of multiple neurons. The intermediate layer may be one layer or two or more layers.

[0072] An example of a three-layer neural network will be described with reference to FIG. A plurality of input values ​​are input to the input layer (X1 to X3). In the input layer (X1 to X3), each input value is multiplied by a first weight (w11 to w16) to calculate a plurality of calculated values. The multiple calculated values ​​from the input layer are input to the hidden layer (Y1, Y2). In the intermediate layers (Y1, Y2), each calculated value in the input layer is multiplied by a second weight (w21 to w26) to calculate a plurality of output values. A plurality of output values ​​are output from the output layers (Z1 to Z3). The plurality of output values ​​change when either the first weights (w11 to w16) or the second weights (w21 to w26) change.

[0073] The three-layer neural network inputs a simulated image 291 to the input layer, and adjusts the first weights (w11 to w16) and second weights (w21 to w26) so that the result output from the output layer approaches the degraded image 292. In this way, a trained model 391 is trained.

[0074] Returning to FIG. 12, step S330 will be described. In step S330, the model generation unit 320 stores the trained model 391.

[0075] For example, the model generation unit 320 stores the trained model 391 in the storage unit 390. The model generation unit 320 also transmits the trained model 391 to the inference device 400 and stores the trained model 391 in the storage unit 490.

[0076] The inference process of the high resolution method will be described with reference to FIG. In step S410, the data acquisition unit 410 acquires the target image 481.

[0077] For example, a user inputs a target image 481 into the inference device 400 , and the data acquisition unit 410 receives the input target image 481 .

[0078] In step S420, the inference unit 420 receives the target image 481 as input and uses the trained model 391 to obtain an inference image 491.

[0079] In step S430, the inference unit 420 outputs the inference image 491.

[0080] For example, the inference unit 420 displays an inference image 491 on a display.

[0081] ***Effects of the First Embodiment*** The first embodiment relates to a method for learning a high-resolution model for remote sensing images equipped with a thermal infrared sensor. For example, the first embodiment relates to a method for learning a high-resolution model for images obtained by remote sensing using a thermal infrared sensor. The first embodiment simulates image quality degradation specific to thermal infrared images and expands learning data, and applies high-precision high-resolution processing to a small amount of original thermal infrared images.

[0082] The first embodiment has the following features: In the following description, elements corresponding to elements described in the embodiment are placed in parentheses, and the reference numerals of the elements described in the embodiment are written in parentheses. In the first embodiment, the learning device (200, 300) includes a reference high-resolution image extension unit (220) and a model generation unit (320). The reference high-resolution image extension unit (220) uses a reference high-resolution image (281) acquired by a remote sensing method capable of acquiring an image with a higher resolution than the target remote sensing image (481) to which the inference process is applied. The reference high-resolution image extension unit (220) also uses acquisition time information (282) of the reference high-resolution image. Furthermore, the reference high-resolution image extension unit (220) uses external data such as background temperature changes for each season and solar altitude, and atmospheric models. The reference high-resolution image extension unit (220) then extends the reference high-resolution image by simulating various imaging dates and times. The model generation unit (320) uses learning data consisting of an extended reference high-resolution image (291) and a simulated degraded image (292) to train a trained model (391) for inferring an estimated high-resolution image (491) from the simulated degraded image. The simulated degraded image (292) is generated by applying a degradation process that simulates the target remote sensing image (481) to the extended reference high-resolution image using target remote sensing method information (284). In the first embodiment, the utilization device (400) includes a data acquisition unit (410) and an inference unit (420). The data acquisition unit (410) acquires a target remote sensing image (481) of the resolution enhancement system (100). The inference unit (420) uses the trained model (391) to infer an estimated high-resolution image (491) from the target remote sensing image, and then infers and outputs the estimated high-resolution image from the target remote sensing image input from the data acquisition unit.

[0083] According to the first embodiment, when a remote sensing image is input to a learning model, highly accurate super-resolution can be achieved.

[0084] ***Summary of the first embodiment*** There is a high-resolution technology that uses machine learning to improve the resolution of thermal infrared images. High-resolution techniques using machine learning include a method for generating simulated degraded images by applying degradation processing to high-resolution images. High-resolution images are acquired using a remote sensing method that can obtain images with a higher resolution than the images acquired by the target remote sensing method to which inference processing is applied. When targeting thermal infrared remote sensing images, the challenges are that there are fewer available training images and source images compared to optical images, and the image degradation process is unique compared to optical images. The first embodiment relates to a method for training a high-resolution model for thermal infrared remote sensing images. The first embodiment provides a training method for applying high-precision high-resolution modeling to a small amount of original thermal infrared images by simulating image quality degradation specific to thermal infrared images and expanding training data.

[0085] ***Features of the First Embodiment*** The model generating device 101 includes an image extension unit 210 and a model generating unit 320 . The image extension unit 210 uses external data to simulate various imaging conditions for the reference high-resolution image and extends the reference high-resolution image. The reference high-resolution image is acquired by a remote sensing method that can acquire an image with a higher resolution than the image of the target remote sensing method to which the inference process is applied. The model generation unit 320 uses training data consisting of the extended reference high-resolution image and the simulated degraded image to train a trained model for inferring an estimated high-resolution image from the simulated degraded image. The simulated degraded image is generated by applying a degradation process to the extended reference high-resolution image using target remote sensing method information to simulate the target remote sensing image.

[0086] The inference device 400 includes a memory unit 490 and an inference unit 420 . The storage unit 490 stores the trained model. a data acquisition unit for acquiring a target remote sensing image; The inference unit 420 uses the trained model to infer an estimated high-resolution image from the target remote sensing image.

[0087] Reference high-resolution images are captured using aerial photography methods such as helicopters, UAVs, and aircraft, as well as satellites that can acquire higher-resolution images than target remote sensing methods.

[0088] The image extension unit 210 performs either or a combination of extension processing based on atmospheric characteristics and extension processing based on temperature information of features, taking into consideration information on the time of acquisition of the reference high-resolution image, the time period to be extended, and external data.

[0089] The image degradation unit 220 performs one or a combination of degradation processes based on the PSF characteristics, GSD, sensitivity performance, noise characteristics, data compression, and quantization.

[0090] In addition to the training data consisting of the simulated degraded image and the extended reference high-resolution image, the training data consists of the target remote sensing image and the target remote sensing technique high-resolution image, which is an image captured in an observation mode with a higher resolution than the target remote sensing image.

[0091] A target remote sensing image is an image captured by exposing it to light rays with wavelengths in the infrared light band.

[0092] ***Supplement to the first embodiment*** The first embodiment relates to a method for training a high resolution model for infrared remote sensing images. The high resolution system 100 can achieve high-precision high resolution when an actual target remote sensing image is input to the training model.

[0093] The learning algorithm is not limited to supervised learning, and reinforcement learning, unsupervised learning, semi-supervised learning, etc. may be used as the learning algorithm.

[0094] There may be multiple resolution enhancement systems 100. The learning device 300 may learn learning data created by another high resolution system 100. The learning device 300 may acquire learning data from another high resolution system 100 used in the same area, or may acquire learning data from another high resolution system 100 used in a different area.

[0095] When multiple high resolution systems 100 are present, one or more high resolution systems 100 may be added or removed.

[0096] The learning device 300 may update a trained model 391 generated in another resolution enhancement system 100 to generate a trained model 391 for its own resolution enhancement system 100. In this case, the learning device 300 updates the trained model 391 by re-training the trained model 391 using training data created in one of the resolution enhancement systems 100.

[0097] Deep learning may be used as the learning algorithm. Deep learning is a method for learning to extract features themselves. Other known methods may be used as the learning algorithm, such as genetic programming, functional logic programming, or support vector machines.

[0098] FIG. 15 shows an example of the functional configuration of the learning device 300. When a high-resolution image 482 is obtained separately from the target image 481 by the first remote sensing technique, a pair of the target image 481 and the high-resolution image 482 may be added to the learning data. The high-resolution image 482 is obtained by the first remote sensing technique and has a higher resolution than the target image 481. For example, the high-resolution image 482 is obtained by changing the observation mode of the first remote sensing technique to the highest resolution mode.

[0099] In the first embodiment, "higher resolution" and "higher resolution" mean not only an improvement in ground resolution but also an improvement in the signal-to-noise ratio.

[0100] "Temperature of land features" can be calculated by simulating the temperature at an extended time / day from a daily / annual temperature change curve, or by using highly accurate land cover classification as external data to estimate the emissivity of land features and simulate the temperature.

[0101] The "atmospheric characteristics" can be calculated by using a method such as subtracting the atmospheric characteristics at the reference date and time from the external atmospheric data and adding the atmospheric characteristics at the extended date and time.

[0102] The first embodiment is an example of a preferred embodiment and is not intended to limit the technical scope of the present disclosure. The first embodiment may be implemented in part or in combination with other embodiments. The procedures described using flowcharts and the like may be modified as appropriate.

[0103] Each element of each device of the resolution increasing system 100 may be realized by software, hardware, firmware, or a combination of these. The "unit" of each element of each device in the resolution increasing system 100 may be read as "processing," "step," "circuit," or "circuitry."

[0104] Various aspects of the present disclosure are described below as appendices. (Appendix 1) a resolution enhancement system that receives a target image obtained by a first remote sensing technique as an input and obtains an inference image having a higher resolution than the target image, an image enhancement unit that uses a reference image obtained by a second remote sensing technique and having a higher resolution than the target image to generate a plurality of simulated images having a higher resolution than the target image; an image degradation unit that degrades each of the plurality of simulated images to generate a plurality of degraded images; a model generation unit that generates a trained model by learning a plurality of pairs of the degraded image and the simulated image; an inference unit that receives the target image as an input and obtains the inferred image using the trained model; A high-resolution system comprising:

[0105] (Appendix 2) The image extension unit generates, for each of a plurality of extended dates and times that are different from the reference date and time at which the second remote sensing technique was performed, an image equivalent to the reference image at the extended date and time using the reference image, reference information indicating the meteorological environment at the reference date and time, and extended information indicating the meteorological environment at the extended date and time as the simulated image. 2. The high-resolution system according to claim 1.

[0106] (Appendix 3) The reference information and the extended information indicate at least one of atmospheric characteristics and object temperature as the meteorological environment. 3. The high-resolution system according to claim 2.

[0107] (Appendix 4) The temperature of the ground object is calculated by simulation based on a temperature change curve that shows the temperature change in one day or one year, or by simulation based on the emissivity of the ground object estimated using land cover classification data. 4. The high-resolution system according to claim 3.

[0108] (Appendix 5) The atmospheric characteristics are calculated using atmospheric data by subtracting atmospheric characteristics at the reference date and time and adding atmospheric characteristics at the extended date and time. 4. The high-resolution system according to claim 3.

[0109] (Appendix 6) The image degradation unit generates the plurality of degraded images by applying at least one of degradation based on PSF characteristics, degradation based on GSD, degradation based on sensitivity characteristics, degradation based on noise characteristics, degradation based on data compression, and degradation based on quantization to each of the plurality of simulated images. 6. A resolution enhancement system according to any one of claims 1 to 5.

[0110] (Appendix 7) The model generation unit generates the trained model by learning a plurality of sets of the degraded image and the simulated image, as well as a set of the target image and a high-resolution image obtained by the first remote sensing technique and having a higher resolution than the target image. 7. A resolution enhancement system according to any one of claims 1 to 6.

[0111] (Appendix 8) The target image is an image captured by exposing it to light having a wavelength in the infrared band. 8. A resolution enhancement system according to any one of claims 1 to 7.

[0112] (Appendix 9) each of the first remote sensing technique and the second remote sensing technique is imaging from an aerial vehicle or an artificial satellite; The second remote sensing technique is a remote sensing technique with a higher resolution than the first remote sensing technique. 9. A resolution enhancement system according to any one of appendices 1 to 8.

[0113] (Appendix 10) A resolution enhancement method for inputting a target image obtained by a first remote sensing technique and obtaining an inference image having a higher resolution than the target image, generating a plurality of simulated images having a higher resolution than the target image using a reference image obtained by a second remote sensing technique and having a higher resolution than the target image; degrading each of the plurality of simulated images to generate a plurality of degraded images; generating a trained model by training a plurality of pairs of the degraded image and the simulated image; The target image is input and the inferred image is obtained using the trained model. High resolution method.

[0114] (Appendix 11) a model generation device that receives a target image obtained by a first remote sensing technique as input and generates a trained model for obtaining an inference image having a higher resolution than the target image; an image enhancement unit that uses a reference image obtained by a second remote sensing technique and having a higher resolution than the target image to generate a plurality of simulated images having a higher resolution than the target image; an image degradation unit that degrades each of the plurality of simulated images to generate a plurality of degraded images; a model generation unit that generates the trained model by learning a plurality of pairs of the degraded image and the simulated image; A model generation device comprising:

[0115] (Appendix 12) An inference device that uses a trained model generated by the model generation device according to Supplementary Note 11, an inference unit that receives a target image obtained by a first remote sensing technique as an input and uses the trained model to obtain an inference image with a higher resolution than the target image; An inference device comprising:

[0116] (Appendix 13) A model generation method for generating a trained model for obtaining an inference image having a higher resolution than the target image by inputting a target image obtained by a first remote sensing technique, generating a plurality of simulated images having a higher resolution than the target image using a reference image obtained by a second remote sensing technique and having a higher resolution than the target image; degrading each of the one or more simulated images to generate one or more degraded images; A plurality of pairs of the degraded image and the simulated image are trained to generate the trained model. Model generation method.

[0117] (Appendix 14) An inference method that uses a learned model generated by the model generation method described in Supplementary Note 13, using the target image obtained by the first remote sensing method as input, and using the learned model to obtain an inference image with a higher resolution than the target image Inference method.

[0118] (Supplementary Note 15) A model generation program that generates a learned model for obtaining an inference image with a higher resolution than the target image by using the target image obtained by the first remote sensing method as input, an image enhancement process for generating a plurality of simulated images with a higher resolution than the target image by using a reference image obtained by the second remote sensing method and having a higher resolution than the target image, an image degradation process for degrading each of the plurality of simulated images to generate a plurality of degraded images, a model generation process for generating the learned model by learning a plurality of pairs of the degraded images and the simulated images, A model generation program for causing a computer to execute.

[0119] (Supplementary Note 16) An inference program that uses a learned model generated by the model generation program described in Supplementary Note 15, an inference process for obtaining an inference image with a higher resolution than the target image by using the target image obtained by the first remote sensing method as input and using the learned model An inference program for causing a computer to execute.

Explanation of Signs

[0120] 100 High-resolution system, 101 Model generation device, 200 Image generation device, 201 Processor, 202 Memory, 203 Auxiliary storage device, 204 Communication device, 205 Input / output interface, 210 Image extension unit, 211 Atmospheric characteristic simulation processing, 212 Temperature simulation processing, 220 Image degradation unit, 221 Resolution degradation processing, 222 Noise degradation processing, 223 Compression degradation processing, 281 Reference image, 282 Reference information, 283 Extended information, 284 First remote sensing information, 290 Memory unit, 291 Simulated image, 292 Degraded image, 300 Learning device, 301 Processor, 302 Memory, 303 Auxiliary storage device, 304 Communication device, 305 Input / output interface, 310 Data acquisition unit, 320 Model generation unit, 390 Memory unit, 391 Trained model, 400 Inference device, 401 processor, 402 memory, 403 auxiliary storage device, 404 communication device, 405 input / output interface, 410 data acquisition unit, 420 inference unit, 481 target image, 482 high-resolution image, 490 memory unit, 491 inference image.

Claims

1. a resolution enhancement system that receives a target image obtained by a first remote sensing technique as an input and obtains an inference image having a higher resolution than the target image, an image enhancement unit that uses a reference image obtained by a second remote sensing technique and that has a higher resolution than the target image to generate a plurality of simulated images that have a higher resolution than the target image; an image degradation unit that degrades each of the plurality of simulated images to generate a plurality of degraded images; a model generation unit that generates a trained model by learning a plurality of pairs of the degraded image and the simulated image; an inference unit that receives the target image as an input and obtains the inferred image using the trained model; A high-resolution system comprising:

2. The image extension unit generates, for each of a plurality of extended dates and times that are different from the reference date and time at which the second remote sensing technique was performed, an image equivalent to the reference image at the extended date and time using the reference image, reference information indicating the meteorological environment at the reference date and time, and extended information indicating the meteorological environment at the extended date and time as the simulated image. The resolution enhancement system according to claim 1 .

3. The reference information and the extended information indicate at least one of atmospheric characteristics and object temperature as the meteorological environment. The resolution enhancement system according to claim 2 .

4. The temperature of the feature is calculated by simulation based on a temperature change curve that shows the temperature change in one day or one year, or by simulation based on the emissivity of the feature estimated using land cover classification data. The resolution enhancement system according to claim 3 .

5. The atmospheric characteristics are calculated using atmospheric data by subtracting atmospheric characteristics at the reference date and time and adding atmospheric characteristics at the extended date and time. The resolution enhancement system according to claim 3 .

6. The image degradation unit generates the plurality of degraded images by applying at least one of degradation based on PSF characteristics, degradation based on GSD, degradation based on sensitivity characteristics, degradation based on noise characteristics, degradation based on data compression, and degradation based on quantization to each of the plurality of simulated images. The resolution enhancement system according to claim 1 .

7. The model generation unit generates the trained model by training a plurality of sets of the degraded image and the simulated image, as well as a set of the target image and a high-resolution image obtained by the first remote sensing technique and having a higher resolution than the target image. The resolution enhancement system according to claim 1 .

8. The target image is an image captured by exposing it to light having a wavelength in the infrared band. The resolution enhancement system according to claim 1 .

9. each of the first remote sensing technique and the second remote sensing technique is imaging from an airborne object or an artificial satellite; The second remote sensing technique is a remote sensing technique with a higher resolution than the first remote sensing technique. The resolution increasing system according to any one of claims 1 to 8.

10. a resolution enhancement method for inputting a target image obtained by a first remote sensing technique and obtaining an inference image having a higher resolution than the target image, generating a plurality of simulated images having a higher resolution than the target image using a reference image obtained by a second remote sensing technique and having a higher resolution than the target image; degrading each of the plurality of simulated images to generate a plurality of degraded images; generating a trained model by training a plurality of pairs of the degraded image and the simulated image; The target image is input and the inferred image is obtained using the trained model. High resolution method.

11. a model generation device that receives a target image obtained by a first remote sensing technique as input and generates a trained model for obtaining an inference image having a higher resolution than the target image; an image enhancement unit that uses a reference image obtained by a second remote sensing technique and that has a higher resolution than the target image to generate a plurality of simulated images that have a higher resolution than the target image; an image degradation unit that degrades each of the plurality of simulated images to generate a plurality of degraded images; a model generation unit that generates the trained model by learning a plurality of pairs of the degraded image and the simulated image; A model generation device comprising:

12. An inference device that uses a trained model generated by the model generation device according to claim 11, an inference unit that receives a target image obtained by a first remote sensing technique as an input and uses the trained model to obtain an inference image with a higher resolution than the target image; An inference device comprising:

13. A model generation method for generating a trained model for obtaining an inference image having a higher resolution than the target image by using a target image obtained by a first remote sensing technique as input, generating a plurality of simulated images having a higher resolution than the target image using a reference image obtained by a second remote sensing technique and having a higher resolution than the target image; degrading each of the one or more simulated images to generate one or more degraded images; A plurality of pairs of the degraded image and the simulated image are trained to generate the trained model. Model generation method.

14. An inference method using a trained model generated by the model generation method according to claim 13, A target image obtained by a first remote sensing technique is input, and an inferred image having a higher resolution than the target image is obtained using the trained model. Reasoning method.

15. a model generation program that receives a target image obtained by a first remote sensing technique as input and generates a trained model for obtaining an inference image having a higher resolution than the target image; an image enhancement process using a reference image obtained by a second remote sensing technique and having a higher resolution than the target image to generate a plurality of simulated images having a higher resolution than the target image; an image degradation process for degrading each of the plurality of simulated images to generate a plurality of degraded images; a model generation process for generating the trained model by learning a plurality of pairs of the degraded image and the simulated image; A model generation program for running the above on a computer.

16. An inference program that uses a trained model generated by the model generation program according to claim 15, An inference process in which a target image obtained by a first remote sensing technique is input and an inference image having a higher resolution than the target image is obtained using the trained model. An inference program that allows a computer to execute the above.

Images