Image quality improvement system, learning device, learning method, and inference method

Abstract

To generate a high-resolution model by setting an image having resolution exceeding the maximum resolution of a target satellite as a teacher image, and to infer an image equivalent to a target image in which image quality is improved.SOLUTION: A deterioration processing device 200 receives target satellite information showing information on remote sensing executed by a target satellite and a reference image having higher image quality than a target image obtained by the remote sensing, and generates a deterioration image that is the reference image in which image quality is reduced by performing deterioration processing for performing down-sampling to the reference image by using a different offset for each band on the basis of the target satellite information. A learning device 300 generates a learned model by learning a set of the deterioration image and the reference image. An inferring device 400 receives the target image, and infers an estimated image equivalent to the target image in which image quality is improved from the target image by using the learned model.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to improving the image quality of images.

Background Art

[0002] Conventionally, ground sensors have been used to grasp the damage when disasters such as floods, heavy rains, and earthquakes occur. The method using ground sensors is useful for grasping the damage situation at spots such as houses and power plants. On the other hand, in the method using ground sensors, it is difficult to obtain surface information for grasping areas with greater damage in a situation where damage occurs over a wide area.

[0003] In recent years, in order to grasp the damage situation of wide-area disasters, analysis methods utilizing remote sensing data have been proposed. Examples of remote sensing include sensing from flying objects such as airplanes, UAVs, optical satellites, and SAR satellites. UAV is an abbreviation for Unmanned Aerial Vehicle and is also called a drone. SAR is an abbreviation for synthetic aperture radar.

[0004] Images obtained by optical satellites are used as a monitoring means for various objects on the earth's surface. In addition, there is a technique for improving the resolution of an image using a high-resolution model in order to improve the ground resolution of the image. The high-resolution model, also called a super-resolution model, is generated by machine learning.

[0005] Patent Document 1 discloses a technique for improving the resolution of an image using a high-resolution model. The high-resolution model is generated as follows. First, a high-resolution image is prepared. Next, a low-resolution image is generated based on the high-resolution image. Then, a pair of the high-resolution image and the low-resolution image is learned as a pair of a teacher image and an input image, and a high-resolution model is generated.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0007]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0008] In conventional high-resolution techniques (including the technique of Patent Document 1), a high-resolution model is generated by machine learning in which a pair of a high-resolution image and a low-resolution image is used as learning data. The low-resolution image is generated by subjecting the high-resolution image to degradation processing such as downsampling. When an image obtained by a target satellite is used as learning data, an image having a resolution exceeding the maximum resolution of the target satellite cannot be used as a teacher image. Therefore, even when using a high-resolution model, it is not possible to generate an image having a resolution exceeding the maximum resolution of the target satellite. Note that since there are differences in the image domains between the target satellite and other remote sensing, even if the images of other remote sensing are used as training data, the images of the target satellite cannot be upscaled to high resolution with high accuracy.

[0009] The present disclosure aims to generate an upscaling model using an image having a resolution exceeding the maximum resolution of the target satellite as a teacher image, so that an image corresponding to the target image with improved image quality can be inferred.

Means for Solving the Problems

[0010] The high-image-quality system of the present disclosure an information reception unit that receives target satellite information indicating information of remote sensing performed by a target satellite and a reference image having higher image quality than the target image obtained by the remote sensing; a degradation processing unit that generates a degraded image, which is the reference image with reduced image quality, by subjecting the reference image to downsampling using different offsets for each band based on the target satellite information; a model generation unit that generates a learned model by learning a pair of the degraded image and the reference image; an image reception unit that receives the target image; an inference unit that infers an estimated image corresponding to the target image with improved image quality from the target image using the learned model; is provided.

Advantages of the Invention

[0011] According to the present disclosure, it is possible to generate an upscaling model using an image having a resolution exceeding the maximum resolution of the target satellite as a teacher image, and infer an image corresponding to the target image with improved image quality.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Modes for Carrying Out the Invention

[0013] In the embodiments and the drawings, the same elements or corresponding elements are denoted by the same reference numerals. The description of the elements denoted by the same reference numerals as the described elements is omitted or simplified as appropriate. The arrows in the figures mainly indicate the flow of data or the flow of processing.

[0014] Embodiment 1. The high - image - quality system 100 will be described based on FIGS. 1 to 14.

[0015] ***Description of the Configuration*** Based on FIG. 1, the configuration of the high - quality image system 100 will be described. The high - quality image system 100 includes a degradation processing device 200, a learning device 300, and an inference device 400.

[0016] Based on FIG. 2, the configuration of the degradation processing device 200 will be described. The degradation processing device 200 is a computer including hardware such as a processor 201, a memory 202, an auxiliary storage device 203, a communication device 204, and an input / output interface 205. These hardware components are connected to each other via signal lines.

[0017] The processor 201 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 201 is a CPU. IC is an abbreviation for Integrated Circuit. CPU is an abbreviation for Central Processing Unit.

[0018] The memory 202 is a volatile or non - volatile storage device. The memory 202 is also called the main storage device or main memory. For example, the memory 202 is a RAM. The data stored in the memory 202 is saved in the auxiliary storage device 203 as needed. RAM is an abbreviation for Random Access Memory.

[0019] The auxiliary storage device 203 is a non - volatile storage device. For example, the auxiliary storage device 203 is a ROM, HDD, flash memory, or a combination thereof. The data stored in the auxiliary storage device 203 is loaded into the memory 202 as needed. ROM is an abbreviation for Read Only Memory. HDD is an abbreviation for Hard Disk Drive.

[0020] The communication device 204 is a receiver and a transmitter. For example, the communication device 204 is a communication chip or a NIC. The communication of the deterioration processing device 200 is performed using the communication device 204. NIC is an abbreviation for Network Interface Card.

[0021] The input / output interface 205 is a port to which an input device and an output device are connected. For example, the input / output interface 205 is a USB terminal, the input device is a keyboard and a mouse, and the output device is a display. The input / output of the deterioration processing device 200 is performed using the input / output interface 205. USB is an abbreviation for Universal Serial Bus.

[0022] The deterioration processing device 200 includes elements such as an information reception unit 210, a deterioration processing unit 220, and an image output unit 230. These elements are realized by software.

[0023] The auxiliary storage device 203 stores a deterioration processing program for causing a computer to function as the information reception unit 210, the deterioration processing unit 220, and the image output unit 230. The deterioration processing program is loaded into the memory 202 and executed by the processor 201. The auxiliary storage device 203 further stores an OS. At least a part of the OS is loaded into the memory 202 and executed by the processor 201. The processor 201 executes the deterioration processing program while executing the OS. OS is an abbreviation for Operating System.

[0024] The input / output data of the deterioration processing program is stored in the storage unit 290. The memory 202 functions as the storage unit 290. However, storage devices such as the auxiliary storage device 203, the registers in the processor 201, and the cache memory in the processor 201 may function as the storage unit 290 instead of, or together with, the memory 202.

[0025] The deterioration processing device 200 may include a plurality of processors that replace the processor 201.

[0026] The deterioration processing program can be recorded (stored) in a non-volatile recording medium such as an optical disk or a flash memory in a computer-readable manner.

[0027] Based on FIG. 3, the configuration of the learning device 300 will be described. The learning device 300 is a computer including hardware such as a processor 301, a memory 302, an auxiliary storage device 303, a communication device 304, and an input / output interface 305. These hardware components are connected to each other via signal lines.

[0028] The processor 301 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 301 is a CPU. The memory 302 is a volatile or non-volatile storage device. The memory 302 is also called a main storage device or main memory. For example, the memory 302 is a RAM. The data stored in the memory 302 is saved in the auxiliary storage device 303 as needed. The auxiliary storage device 303 is a non-volatile storage device. For example, the auxiliary storage device 303 is a ROM, an HDD, a flash memory, or a combination thereof. The data stored in the auxiliary storage device 303 is loaded into the memory 302 as needed. The communication device 304 is a receiver and a transmitter. For example, the communication device 304 is a communication chip or a NIC. The communication of the learning device 300 is performed using the communication device 304. The input / output interface 305 is a port to which an input device and an output device are connected. For example, the input / output interface 305 is a USB terminal, the input device is a keyboard and a mouse, and the output device is a display. The input / output of the learning device 300 is performed using the input / output interface 305.

[0029] The learning device 300 includes elements such as a data reception unit 310, a model generation unit 320, and a model output unit 330. These elements are realized by software.

[0030] The auxiliary storage device 303 stores a learning program for causing a computer to function as the data reception unit 310, the model generation unit 320, and the model output unit 330. The learning program is loaded into the memory 302 and executed by the processor 301. The auxiliary storage device 303 further stores an OS. At least a part of the OS is loaded into the memory 302 and executed by the processor 301. The processor 301 executes the learning program while executing the OS.

[0031] The input / output data of the learning program is stored in the storage unit 390. The memory 302 functions as the storage unit 390. However, storage devices such as the auxiliary storage device 303, the registers in the processor 301, and the cache memory in the processor 301 may function as the storage unit 390 instead of or together with the memory 302.

[0032] The learning device 300 may include a plurality of processors that replace the processor 301.

[0033] The learning program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or a flash memory.

[0034] Based on FIG. 4, the configuration of the inference device 400 will be described. The inference device 400 is a computer including hardware such as a processor 401, a memory 402, an auxiliary storage device 403, a communication device 404, and an input / output interface 405. These hardware components are connected to each other via signal lines.

[0035] The processor 401 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 401 is a CPU. The memory 402 is a volatile or non-volatile storage device. The memory 402 is also called the main storage device or main memory. For example, the memory 402 is a RAM. The data stored in the memory 402 is saved to the auxiliary storage device 403 as needed. The auxiliary storage device 403 is a non-volatile storage device. For example, the auxiliary storage device 403 is a ROM, HDD, flash memory, or a combination thereof. The data stored in the auxiliary storage device 403 is loaded into the memory 402 as needed. The communication device 404 is a receiver and a transmitter. For example, the communication device 404 is a communication chip or a NIC. The communication of the inference device 400 is performed using the communication device 404. The input / output interface 405 is a port to which an input device and an output device are connected. For example, the input / output interface 405 is a USB terminal, the input device is a keyboard and a mouse, and the output device is a display. The input / output of the inference device 400 is performed using the input / output interface 405.

[0036] The inference device 400 includes elements such as an image reception unit 410, an inference unit 420, and an image output unit 430. These elements are realized by software.

[0037] The auxiliary storage device 403 stores an inference program for causing a computer to function as the image reception unit 410, the inference unit 420, and the image output unit 430. The inference program is loaded into the memory 402 and executed by the processor 401. The auxiliary storage device 403 further stores an OS. At least a part of the OS is loaded into the memory 402 and executed by the processor 401. The processor 401 executes the inference program while executing the OS.

[0038] The input / output data of the inference program is stored in the storage unit 490. The memory 402 functions as a storage unit 490. However, storage devices such as the auxiliary storage device 403, registers in the processor 401, and cache memory in the processor 401 may function as the storage unit 490 instead of or together with the memory 402.

[0039] The inference device 400 may include a plurality of processors that replace the processor 401.

[0040] The inference program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or a flash memory.

[0041] ***Explanation of operations*** The operation procedure of the high-image-quality system 100 corresponds to the high-image-quality method.

[0042] Based on FIGS. 5 to 8, the outline of the high-image-quality method will be described. In step S110, the degradation processing device 200 degrades the reference image 112 based on the target satellite information 111.

[0043] FIG. 6 shows the outline of step S110. The target satellite information 111 is information on remote sensing performed by the target satellite. The target satellite is an artificial satellite that obtains an image (target image 131 described later) that is the target of high-image-quality through remote sensing. The reference image 112 is an image with higher image quality than the target image 131. For example, the reference image 112 is an image with higher resolution than the target image 131 and is obtained by high-resolution remote sensing. An example of remote sensing is shooting from an aircraft. Specific examples of the aircraft are a helicopter, a UAV, an airplane, and an artificial satellite. UAV is an abbreviation for Unmanned Aerial Vehicle. Also, remote sensing may be performed by a helisat. A helisat is a helicopter-satellite communication system. The degraded image 113 is the degraded reference image 112. In other words, the degraded image 113 is the reference image 112 whose image quality has been lowered to the image quality level of the target image 131.

[0044] Returning to FIG. 5, the description will be continued. Step S110 is executed for each of the plurality of reference images 112. By step S110, a plurality of pairs of the degraded image 113 and the reference image 112 are generated.

[0045] In step S120, the learning device 300 learns a plurality of pairs of the degraded image 113 and the reference image 112 to generate a learned model 122.

[0046] FIG. 7 shows an overview of step S120. The learning data 121 is a pair of the degraded image 113 and the reference image 112.

[0047] Returning to FIG. 5, the description will be continued. In step S130, the inference device 400 enhances the image quality of the target image 131 using the learned model 122.

[0048] FIG. 8 shows an overview of step S130. The target image 131 is an image to be the target of image quality enhancement, and is obtained by remote sensing of the target satellite. The type of the target image 131 is an optical image. The estimated image 132 corresponds to the target image 131 with enhanced image quality. That is, the estimated image 132 corresponds to the target image 131 whose image quality has been enhanced.

[0049] Based on FIG. 9, the procedure of step S110 will be described. Steps S111 to S113 are executed for each reference image 112.

[0050] In step S111, the information reception unit 210 receives the target satellite information 111 and the reference image 112. For example, the user inputs the target satellite information 111 and the reference image 112 into the degradation processing device 200 respectively. Then, the information receiving unit 210 receives the input target satellite information 111 and the input reference image 112.

[0051] In step S112, the degradation processing unit 220 performs degradation processing on the reference image 112 based on the target satellite information 111. As a result, a degraded image 113 is generated. The pair of the degraded image 113 and the reference image 112 is stored in the storage unit 290. The degradation processing is a process for degrading the image quality. The degradation processing will be described later.

[0052] In step S113, the image output unit 230 outputs the pair of the degraded image 113 and the reference image 112. For example, the image output unit 230 stores the pair of the degraded image 113 and the reference image 112 in the storage medium designated by the user. Alternatively, the image output unit 230 transmits the pair of the degraded image 113 and the reference image 112 to the learning device 300.

[0053] Based on FIG. 10, the degradation processing in step S112 will be described. Specific examples of the degradation processing are color shift processing, resolution processing, GSD processing, noise processing, compression processing, or a combination thereof.

[0054] The color shift processing is a process for reproducing color shift. A plurality of optical sensors are mounted on an optical satellite. The plurality of optical sensors have different light-receiving wavelength bands (bands) from each other and obtain a multi-band image. Then, a synthetic image is generated by synthesizing the multi-band images. At this time, color shift may occur in the synthetic image due to the influence of the physical position shift between the optical sensors. Also, the color shift may occur at the sub-pixel level. The target satellite information 111 indicates the arrangement of a plurality of optical sensors mounted on the target satellite. The color shift process performs downsampling of the reference image 112 by shifting the offset of downsampling for each band based on the arrangement of a plurality of optical sensors. The offset of downsampling means the binning start position. As a result, sub-pixel color shift is reproduced for the reference image 112, and the degraded image 113 is generated.

[0055] The resolution process is a process of degrading the resolution. For an optical image obtained by an optical satellite, a PSF which is an index of resolution is determined. PSF is an abbreviation of Point Spread Function. The resolution is obtained by combining a plurality of degradation models considering the optical performance of the optical sensor, the aperture shape of the optical sensor, and the effects such as disturbance and image flow. The target satellite information 111 indicates the PSF for the target image 131. The resolution process applies the PSF to the reference image 112. For example, the PSF is multiplied in the frequency domain. Or, the PSF is convolved and integrated with the reference image 112. As a result, the resolution of the reference image 112 is converted to be equivalent to the resolution of the target image 131, and the degraded image 113 is generated. In the resolution process, the PSF of the reference image 112 may be used. The reference information indicating the PSF of the reference image 112 is received together with the reference image 112. Furthermore, the difference in PSF between bands may be incorporated into the resolution process. This makes it possible to reproduce aliasing for the reference image 112. The difference in PSF will be explained. The PSF may be different for each band. For example, the MTF of the multi-spectral band is higher than the PSF of the panchromatic band. Therefore, aliasing may occur in the multi-spectral image. MTF is an abbreviation of Modulation Transfer Function.

[0056] The GSD process is a degradation process using GSD. GSD is an abbreviation of Ground Sampling Distance. The target satellite information 111 indicates the GSD of the target image 131. The GSD process compares the GSD of the target image 131 with the GSD of the reference image 112, and downsamples the reference image 112 based on the comparison result. Specific examples of downsampling include bilinear interpolation or bicubic interpolation. The reference information indicating the GSD of the reference image 112 is received together with the reference image 112. Thereby, the GSD of the target image 131 is simulated for the reference image 112, and the degraded image 113 is generated. In the GSD process, it is desirable that the ratio of the GSD of the reference image 112 to the GSD of the target image 131 is equal to or greater than the magnification of the high-resolution by the high-image-quality system 100.

[0057] The noise process is a degradation process that adds noise. The target satellite information 111 indicates the noise characteristics of the remote sensing of the target satellite. Specifically, the target satellite information 111 indicates the sensitivity performance of the remote sensing of the target satellite and the amount of light from the assumed subject. The noise process estimates the SNR of the target image 131 based on the sensitivity performance of the target satellite and the amount of light from the assumed subject, and adds noise corresponding to the SNR to the reference image 112. When it is assumed that the noise is additive white Gaussian noise, the SNR is related to the standard deviation of the Gaussian noise. Therefore, it is possible to perform a degradation process that applies noise equivalent to the noise of the target image 131 based on the SNR. SN is an abbreviation for Signal to Noise Ratio. In the noise process, noise other than additive white Gaussian noise may be added to the reference image 112. In this case, the target satellite information 111 indicates the characteristics of noise other than additive white Gaussian noise. Specific examples of noise other than additive white Gaussian noise are stripe noise and FPN. FPN is an abbreviation for Fixed Pattern Noise. Also, when the target image 131 is a thermal infrared image, shot noise, 1 / f noise, or shading due to internal radiation may be added to the reference image 112. Noise caused by the atmosphere may also be added to the reference image 112. When the reference image 112 is obtained by high-resolution remote sensing performed within the atmosphere, the reference image 112 does not contain noise caused by the atmosphere. On the other hand, the target image 131 contains noise caused by the atmosphere. For example, the target image 131 contains noise such as reduced image contrast, occlusion, and interference. Therefore, in noise processing, noise caused by the atmosphere is added to the reference image 112 using a published atmospheric model. Thereby, it is possible to generate an image simulating a satellite image.

[0058] The compression process is a degradation process by data compression. When the target image 131 is downlinked to the ground station, data compression is performed. If the data compression is irreversible compression, image quality degradation due to compression occurs. The target satellite information 111 indicates a data compression method for the target image 131. The compression process applies the data compression method for the target image 131 to the reference image 112. Thereby, it is possible to reproduce the same degradation as the target image 131 in the reference image 112.

[0059] Based on FIG. 11, the procedure of step S120 will be described. In step S121, the data reception unit 310 receives a plurality of sets of the degraded image 113 and the reference image 112. For example, a user inputs a plurality of sets into the learning device 300, and the data reception unit 310 receives the input plurality of sets. Alternatively, the degradation processing device 200 transmits a plurality of sets, and the data reception unit 310 receives the plurality of sets. Then, the data reception unit 310 stores a plurality of sets of the degraded image 113 and the reference image 112 in the storage unit 390 as learning data 121.

[0060] In step S122, the model generation unit 320 learns a plurality of pairs of the degraded image 113 and the reference image 112. As a result, the learned model 122 is generated. The learned model 122 is stored in the storage unit 390. The learned model 122 is a learned model for inferring a high-quality image corresponding to the input image. The high-quality image is an image corresponding to the input image with enhanced quality. The learning in step S122 will be described later.

[0061] In step S123, the model output unit 330 outputs the learned model 122. For example, the model output unit 330 stores the learned model 122 in a storage medium specified by the user. Alternatively, the model output unit 330 transmits the learned model 122 to the inference device 400.

[0062] The learning in step S122 will be explained. In the learning data 121, the reference image 112 is the correct answer to be inferred from the degraded image 113. That is, the reference image 112 is the teacher data corresponding to the degraded image 113. The model generation unit 320 learns the reference image 112 corresponding to the degraded image 113. That is, the model generation unit 320 learns the combination of the degraded image 113 and the reference image 112. As a result, the learned model 122 is generated.

[0063] The model generation unit 320 can use known learning algorithms such as supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. When supervised learning is applied, the subject shown in the reference image 112 is the same as the subject shown in the target image 131. When unsupervised learning is applied, it is not necessary for the subject shown in the reference image 112 to be the same as the subject shown in the target image 131.

[0064] For example, the model generation unit 320 performs supervised learning using a neural network model as follows. Supervised learning is a method for inferring result data from input data, and it learns features in the learning data using pairs of input data and result data as the learning data. A neural network is composed of an input layer consisting of a plurality of neurons, one or more intermediate layers each consisting of a plurality of neurons, and an output layer consisting of a plurality of neurons. The intermediate layer is also called a hidden layer.

[0065] Figure 12 shows an example of a neural network model. "X1", "X2", and "X3" constitute the input layer. "Y1" and "Y2" constitute one intermediate layer. "Z1", "Z2", and "Z3" constitute the output layer. A plurality of input values are input to the input layer (X1~X3). When a plurality of input values are input to the input layer (X1~X3), the first weights (w11~w16) are multiplied by the plurality of input values. The plurality of values obtained thereby are referred to as a plurality of calculated values. The plurality of calculated values are input to the intermediate layer (Y1, Y2). When the plurality of calculated values are input to the intermediate layer (Y1, Y2), the second weights (w21~w26) are multiplied by the plurality of calculated values. The plurality of values obtained thereby are referred to as a plurality of output values. The plurality of output values are input to the output layer (Z1~Z3) and output from the output layer (Z1~Z3). The plurality of output values change depending on the first weights (w11~w16) and the second weights (w21~W26).

[0066] In supervised learning, the model generation unit 320 inputs the degraded image 113 as a plurality of input values to the input layer, and adjusts the first weights and the second weights so that the plurality of output values output from the output layer approach the reference image 112.

[0067] Based on Figure 13, the procedure of step S130 will be described. In step S130, the learned model 122 is stored in the storage unit 490. For example, a user inputs the learned model 122 into the learning device 300, and the inference unit 420 stores the input learned model 122 in the storage unit 490. Alternatively, the learning device 300 transmits the learned model 122, and the inference unit 420 receives the learned model 122 and stores it in the storage unit 490.

[0068] In step S131, the image reception unit 410 receives the target image 131. For example, a user inputs the target image 131 into the inference device 400, and the image reception unit 410 receives the input target image 131.

[0069] In step S132, the inference unit 420 infers the estimated image 132 from the target image 131 using the learned model 122. Specifically, the inference unit 420 calculates the learned model 122 with the target image 131 as the input. As a result, the estimated image 132 is output from the learned model 122.

[0070] In step S133, the image output unit 430 outputs the estimated image 132. For example, the image output unit 430 displays the estimated image 132 on a display.

[0071] ***Effects of Embodiment 1*** According to Embodiment 1, it becomes possible to obtain a high-quality model (learned model 122) for optical satellite images. And it becomes possible to obtain a high-quality image (estimated image 132) with high accuracy by using an actual satellite image (target image 131) as the input of the high-quality model.

[0072] ***Supplementary Explanation of Embodiment 1*** High quality means high resolution, improvement of ground resolution, improvement of SNR, improvement of color misregistration between bands, etc.

[0073] The type of the target image 131 is an optical image. An optical image is an image obtained by exposing light rays. The light rays to be exposed may be light rays having wavelengths in any band of visible light, near-infrared light, and far-infrared light.

[0074] The learning device 300 may perform learning using learning data generated by other high-image-quality systems. When multiple high-image-quality systems are used in multiple areas, the learning device 300 may acquire learning data from a high-image-quality system used in the same area as the high-image-quality system 100, or may acquire learning data from a high-image-quality system used in an area different from the high-image-quality system 100. The learning device 300 may add each high-image-quality system to the system group that is the source of the learning data at any time, or may remove each high-image-quality system from the system group that is the source of the learning data at any time. The learning device 300 may use a learned model generated by another high-image-quality system. For example, the learning device 300 performs learning using the learning data 121 to update the learned model. The updated learned model becomes the learned model 122.

[0075] The learning algorithm of the learning device 300 may be deep learning that learns the extraction of the feature amounts themselves. Or, the learning algorithm may be machine learning according to a genetic program, a functional logic program, a support vector machine, or the like.

[0076] FIG. 14 shows an embodiment of the learning device 300. When a corresponding image 123 can be obtained in addition to the target image 131 by remote sensing of the target satellite, a pair of the target image 131 and the corresponding image 123 may be added to the learning data 121. The corresponding image 123 is an image corresponding to the target image 131. The corresponding image 123 is obtained in an observation mode different from the observation mode in which the target image 131 is obtained, and has a higher image quality than the target image 131. This embodiment is possible when the target image 131 is obtained in an observation mode other than the highest resolution observation mode.

[0077] The degradation processing device 200 functions as an input image generation device that generates input images for learning. The learning device 300 functions as a model generation device that generates a learned model. The inference device 400 functions as a utilization device that utilizes the learned model.

[0078] Embodiment 1 is an example of a preferred form and is not intended to limit the technical scope of the present disclosure. Embodiment 1 may be partially implemented or may be implemented in combination with other forms. The procedures described using flowcharts and the like may be changed as appropriate.

[0079] Each element of each device of the high - quality image system 100 may be realized by any of software, hardware, firmware, or a combination thereof. The "section" of each element of each device of the high - quality image system 100 may be read as "processing", "step", "circuit", or "circuitry".

[0080] Hereinafter, aspects of the present disclosure will be described as appendices.

[0081] (Appendix 1) An information reception unit that receives target satellite information indicating information on remote sensing performed by a target satellite and a reference image having higher image quality than the target image obtained by the remote sensing; A degradation processing unit that generates a degraded image, which is the reference image with reduced image quality, by performing degradation processing on the reference image based on the target satellite information; A model generation unit that generates a learned model by learning a pair of the degraded image and the reference image; An image reception unit that receives the target image; An inference unit that uses the learned model to infer, from the target image, an estimated image corresponding to the target image with enhanced image quality. An image quality improvement system comprising the same.

[0082] (Appendix 2) The reference image is an image obtained by photographing from a flying object. The flying object is any one of a helicopter, an unmanned aerial vehicle, and an aircraft, or a satellite that can obtain an image with higher image quality than the target image. The image quality improvement system according to Appendix 1.

[0083] (Appendix 3) The degradation process is any one of a color shift process, a resolution process, a ground sampling distance process, a noise process, and a compression process, or any combination of the color shift process, the resolution process, the ground sampling distance process, the noise process, and the previous compression process. The target satellite information indicates the arrangement of a plurality of optical sensors mounted on the target satellite, and the color shift process reproduces the color shift based on the arrangement of the plurality of optical sensors in the reference image. The target satellite information indicates a point spread function for the target image, and the resolution process applies the point spread function to the reference image. The target satellite information indicates the ground sampling distance of the target image, and the ground sampling distance process simulates the ground sampling distance for the reference image. The target satellite information indicates the noise characteristics of the remote sensing, and the noise process adds noise based on the noise characteristics to the reference image. The target satellite information indicates a data compression method for the target image, and the compression process applies the data compression method to the reference image. The image quality improvement system according to Appendix 1 or Appendix 2.

[0084] (Supplementary Note 4) The model generation unit generates the learned model by learning the pair of the degraded image and the reference image and the pair of the target image and the corresponding image, wherein the corresponding image is obtained in an observation mode different from the observation mode in which the target image is obtained in the remote sensing, and has higher image quality than the target image The high-quality image system according to any one of Supplementary Notes 1 to 3.

[0085] (Supplementary Note 5) The target satellite information indicates the arrangement of a plurality of optical sensors mounted on the target satellite, The degradation process performs downsampling of the reference image using different offsets for each band based on the arrangement of the plurality of optical sensors. The high-quality image system according to Supplementary Note 1, Supplementary Note 2 or Supplementary Note 4.

[0086] The target image is an image obtained by exposing light rays having wavelengths in the visible light band. The high-quality image system according to any one of Supplementary Notes 1 to 5.

[0087] (Supplementary Note 6) The target image is an image obtained by exposing light rays having wavelengths in the infrared light band. The high-quality image system according to any one of Supplementary Notes 1 to 5.

Explanation of Reference Numerals

[0088] 100 High-quality system, 111 Target satellite information, 112 Reference image, 113 Degraded image, 121 Learning data, 122 Trained model, 123 Corresponding image, 131 Target image, 132 Estimated image, 200 Degradation processing device, 201 Processor, 202 Memory, 203 Auxiliary storage device, 204 Communication device, 205 Input / output interface, 210 Information reception unit, 220 Degradation processing unit, 230 Image output unit, 290 Storage unit, 300 Learning device, 301 Processor, 302 Memory, 303 Auxiliary storage device, 304 Communication device, 305 Input / output interface, 310 Data reception unit, 320 Model generation unit, 330 Model output unit, 390 Storage unit, 400 Inference device, 401 Processor, 402 Memory, 403 Auxiliary storage device, 404 Communication device, 405 Input / output interface, 410 Image reception unit, 420 Inference unit, 430 Image output unit, 490 Storage unit.

Claims

1. An information reception unit that receives target satellite information indicating information on remote sensing performed by a target satellite, and a reference image having a higher image quality than a target image obtained by the remote sensing; A degradation processing unit that generates a degraded image, which is the reference image with reduced image quality, by subjecting the reference image to downsampling using different offsets for each band based on the target satellite information; A model generation unit that generates a learned model by learning a pair of the degraded image and the reference image; An image reception unit that receives the target image; An inference unit that infers an estimated image corresponding to the target image with enhanced image quality from the target image using the learned model; A high-image-quality system comprising the above.

2. An information reception unit that receives target satellite information indicating information on remote sensing performed by a target satellite, and a reference image having a higher image quality than a target image obtained by the remote sensing; A degradation processing unit that generates a degraded image, which is the reference image with reduced image quality, by subjecting the reference image to a degradation process that reproduces color shift based on the target satellite information; A model generation unit that generates a learned model by learning a pair of the degraded image and the reference image; An image reception unit that receives the target image; An inference unit that infers an estimated image corresponding to the target image with enhanced image quality from the target image using the learned model; A high-image-quality system comprising the above.

3. An information reception unit that receives target satellite information indicating information on remote sensing performed by a target satellite, and a reference image having a higher image quality than a target image obtained by the remote sensing; A degradation processing unit that generates a degraded image, which is the reference image with reduced image quality, by subjecting the reference image to a degradation process based on the target satellite information; A model generation unit that generates a learned model by learning a pair of the degraded image and the reference image; An image reception unit that receives the target image; An inference unit that infers an estimated image corresponding to the target image with enhanced image quality from the target image using the learned model; Comprising the above, The degradation process performs resolution processing taking into account the difference in the point spread function between bands based on the target satellite information on the reference image. High-definition system.

4. A learning device for generating a learned model for inferring an estimated image corresponding to the target image with enhanced image quality from a target image obtained by remote sensing of a target satellite, a data reception unit that receives, as learning data, a set of a reference image with higher image quality than the target image and a degraded image that is the reference image with degraded image quality; a model generation unit that generates the learned model by learning the learning data; comprising: The degraded image is generated by subjecting the reference image to a degradation process of performing downsampling using different offsets for each band based on target satellite information indicating the remote sensing information. Learning device.

5. A learning device for generating a learned model for inferring an estimated image corresponding to the target image with enhanced image quality from a target image obtained by remote sensing of a target satellite, a data reception unit that receives, as learning data, a set of a reference image with higher image quality than the target image and a degraded image that is the reference image with degraded image quality; a model generation unit that generates the learned model by learning the learning data; comprising: The degraded image is generated by subjecting the reference image to a degradation process of reproducing color shift based on target satellite information indicating the remote sensing information. Learning device.

6. A learning device for generating a learned model for inferring an estimated image corresponding to the target image with enhanced image quality from a target image obtained by remote sensing of a target satellite, a data reception unit that receives, as learning data, a set of a reference image with higher image quality than the target image and a degraded image that is the reference image with degraded image quality; a model generation unit that generates the learned model by learning the learning data; comprising: The degraded image is generated by subjecting the reference image to a resolution process taking into account the difference in the point spread function between bands based on target satellite information indicating the remote sensing information. Learning device.

7. A learning method for generating a learned model for inferring an estimated image corresponding to the target image with enhanced image quality from a target image obtained by remote sensing of a target satellite, comprising: receiving, as learning data, a set of a reference image having higher image quality than the target image and a degraded image that is the reference image with degraded image quality; generating the learned model by learning the learning data; wherein the degraded image is generated by subjecting the reference image to a degradation process of performing downsampling using different offsets for each band based on target satellite information indicating the remote sensing information; Learning method.

8. A learning method for generating a learned model for inferring an estimated image corresponding to the target image with enhanced image quality from a target image obtained by remote sensing of a target satellite, comprising: receiving, as learning data, a set of a reference image having higher image quality than the target image and a degraded image that is the reference image with degraded image quality; generating the learned model by learning the learning data; wherein the degraded image is generated by subjecting the reference image to a degradation process of reproducing color shift based on target satellite information indicating the remote sensing information; Learning method.

9. A learning method for generating a learned model for inferring an estimated image corresponding to the target image with enhanced image quality from a target image obtained by remote sensing of a target satellite, comprising: receiving, as learning data, a set of a reference image having higher image quality than the target image and a degraded image that is the reference image with degraded image quality; generating the learned model by learning the learning data; wherein the degraded image is generated by subjecting the reference image to a resolution process taking into account differences in point spread functions between bands based on target satellite information indicating the remote sensing information; Learning method.

10. receiving a target image obtained by remote sensing of a target satellite; An inference method for inferring, using a learned model, an estimated image corresponding to the target image with enhanced image quality from the target image, comprising: the learned model is generated by learning a set of a degraded image and a reference image; the reference image is an image having higher image quality than the target image; The degraded image is the reference image with reduced image quality, and is generated by subjecting the reference image to a degradation process of performing downsampling using different offsets for each band based on the target satellite information indicating the remote sensing information. Inference method.

11. Receiving a target image obtained by remote sensing of a target satellite, An inference method for inferring, from the target image, an estimated image corresponding to the target image with enhanced image quality using a learned model, The learned model is generated by learning a pair of a degraded image and a reference image, The reference image is an image with higher image quality than the target image, The degraded image is the reference image with reduced image quality, and is generated by subjecting the reference image to a degradation process of reproducing color shift based on the target satellite information indicating the remote sensing information. Inference method.

12. Receiving a target image obtained by remote sensing of a target satellite, An inference method for inferring, from the target image, an estimated image corresponding to the target image with enhanced image quality using a learned model, The learned model is generated by learning a pair of a degraded image and a reference image, The reference image is an image with higher image quality than the target image, The degraded image is the reference image with reduced image quality, and is generated by subjecting the reference image to a resolution process taking into account the difference in the point spread function between bands based on the target satellite information indicating the remote sensing information. Inference method.

Images