Learning device, learning method, learning program, inference device, inference method, inference program, high image quality system, and high image quality method

Abstract

To generate a high-resolution model using an image with a resolution exceeding the highest resolution of a target satellite as a teacher image.SOLUTION: A learning apparatus 300 receives as learning data a set of a reference image with higher image quality than a target image and a degraded image obtained by degrading the image quality of the reference image. The target image is obtained by remote sensing of a target satellite. The learning apparatus 300 then generates a learned model by learning the learning data. The degraded image is generated by subjecting the reference image to degradation processing based on target satellite information. The target satellite information indicates information about the remote sensing.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to improving the image quality of images.

Background Art

[0002] Conventionally, ground sensors have been used to grasp the damage when disasters such as floods, heavy rains, and earthquakes occur. The method using ground sensors is useful for grasping the damage status of spots such as houses and power plants. On the other hand, in the method using ground sensors, it is difficult to obtain comprehensive information for grasping areas with greater damage in a situation where damage occurs over a wide area.

[0003] In recent years, in order to grasp the damage status of wide-area disasters, analysis methods utilizing remote sensing data have been proposed. Examples of remote sensing include sensing from flying objects such as airplanes, UAVs, optical satellites, and SAR satellites. UAV is an abbreviation for Unmanned Aerial Vehicle and is also called a drone. SAR is an abbreviation for synthetic aperture radar.

[0004] Images obtained by optical satellites are used as a monitoring means for various objects on the earth's surface. In addition, there is a technology for improving the resolution of an image using a high-resolution model in order to improve the ground resolution of the image. The high-resolution model, also called a super-resolution model, is generated by machine learning.

[0005] Patent Document 1 discloses a technique for improving the resolution of an image using a high-resolution model. The high-resolution model is generated as follows. First, a high-resolution image is prepared. Next, a low-resolution image is generated based on the high-resolution image. Then, a pair of the high-resolution image and the low-resolution image is learned as a pair of a teacher image and an input image, and a high-resolution model is generated.

Prior Art Documents

Patent Document

[0006]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] In conventional high-resolution technologies (including the technology of Patent Document 1), a high-resolution model is generated by machine learning in which a pair of a high-resolution image and a low-resolution image is used as learning data. The low-resolution image is generated by subjecting the high-resolution image to degradation processing such as downsampling. When an image obtained by a target satellite is used as learning data, an image having a resolution exceeding the maximum resolution of the target satellite cannot be used as a teacher image. Therefore, even when using the high-resolution model, an image having a resolution exceeding the maximum resolution of the target satellite cannot be generated. Note that since there are differences in the image domains between the target satellite and other remote sensing, even if an image of other remote sensing is used as learning data, the image of the target satellite cannot be highly resolved with high accuracy.

[0008] An object of the present disclosure is to enable generation of a high-resolution model using an image having a resolution exceeding the maximum resolution of a target satellite as a teacher image.

Means for Solving the Problems

[0009] The learning device of the present disclosure enhances the image quality from a target image obtained by remote sensing of a target satellite and generates a learned model for inferring an estimated image corresponding to the target image. The learning device A reference image with higher image quality than the target image and degradation of the reference image with reduced image quality A data reception unit that receives a set of images as learning data, A model generation unit that generates the learned model by learning the learning data and, are provided. The degraded image is generated by performing degradation processing based on target satellite information indicating the remote sensing information on the reference image 、 The deterioration process includes either a color shift process or a compression process, The color shift process reproduces the color shift based on the arrangement of a plurality of optical sensors mounted on the target satellite indicated by the target satellite information in the reference image, The compression process applies the data compression method for the target image indicated by the target satellite information to the reference image.

Advantages of the Invention

[0010] According to the present disclosure, an image having a resolution exceeding the maximum resolution of a target satellite can be used as a teacher image to generate a high-resolution model.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

[0012] In the embodiments and the drawings, the same elements or corresponding elements are denoted by the same reference numerals. The description of elements denoted by the same reference numerals as those already described will be omitted or simplified as appropriate. The arrows in the figures mainly indicate the flow of data or the flow of processing.

[0013] Embodiment 1. The high - quality enhancement system 100 will be described based on FIGS. 1 to 14.

[0014] ***Description of the Configuration*** Based on FIG. 1, the configuration of the high - quality enhancement system 100 will be described. The high - quality enhancement system 100 includes a degradation processing device 200, a learning device 300, and an inference device 400.

[0015] Based on FIG. 2, the configuration of the degradation processing device 200 will be described. The degradation processing device 200 is a computer including hardware such as a processor 201, a memory 202, an auxiliary storage device 203, a communication device 204, and an input / output interface 205. These hardware components are connected to each other via signal lines.

[0016] The processor 201 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 201 is a CPU. IC is the abbreviation of Integrated Circuit. CPU is the abbreviation of Central Processing Unit.

[0017] The memory 202 is a volatile or non-volatile storage device. The memory 202 is also called the main storage device or main memory. For example, the memory 202 is RAM. The data stored in the memory 202 is saved to the auxiliary storage device 203 as needed. RAM is the abbreviation of Random Access Memory.

[0018] The auxiliary storage device 203 is a non-volatile storage device. For example, the auxiliary storage device 203 is ROM, HDD, flash memory, or a combination thereof. The data stored in the auxiliary storage device 203 is loaded into the memory 202 as needed. ROM is the abbreviation of Read Only Memory. HDD is the abbreviation of Hard Disk Drive.

[0019] The communication device 204 is a receiver and a transmitter. For example, the communication device 204 is a communication chip or NIC. The communication of the degradation processing device 200 is performed using the communication device 204. NIC is the abbreviation of Network Interface Card.

[0020] The input / output interface 205 is a port to which an input device and an output device are connected. For example, the input / output interface 205 is a USB terminal, the input device is a keyboard and a mouse, and the output device is a display. The input / output of the degradation processing device 200 is performed using the input / output interface 205. USB is the abbreviation of Universal Serial Bus.

[0021] The degradation processing device 200 includes elements such as an information reception unit 210, a degradation processing unit 220, and an image output unit 230. These elements are realized by software.

[0022] The auxiliary storage device 203 stores a degradation processing program for causing a computer to function as an information reception unit 210, a degradation processing unit 220, and an image output unit 230. The degradation processing program is loaded into the memory 202 and executed by the processor 201. The auxiliary storage device 203 further stores an OS. At least a part of the OS is loaded into the memory 202 and executed by the processor 201. The processor 201 executes the degradation processing program while executing the OS. OS is an abbreviation for Operating System.

[0023] The input / output data of the degradation processing program is stored in the storage unit 290. The memory 202 functions as the storage unit 290. However, storage devices such as the auxiliary storage device 203, registers in the processor 201, and cache memories in the processor 201 may function as the storage unit 290 instead of or together with the memory 202.

[0024] The degradation processing device 200 may include a plurality of processors that replace the processor 201.

[0025] The degradation processing program can be recorded (stored) in a non-volatile recording medium such as an optical disk or a flash memory in a computer-readable manner.

[0026] Based on FIG. 3, the configuration of the learning device 300 will be described. The learning device 300 is a computer including hardware such as a processor 301, a memory 302, an auxiliary storage device 303, a communication device 304, and an input / output interface 305. These hardware components are connected to each other via signal lines.

[0027] The processor 301 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 301 is a CPU. The memory 302 is a volatile or non-volatile storage device. The memory 302 is also referred to as the main storage device or main memory. For example, the memory 302 is RAM. The data stored in the memory 302 is saved in the auxiliary storage device 303 as needed. The auxiliary storage device 303 is a non-volatile storage device. For example, the auxiliary storage device 303 is ROM, HDD, flash memory, or a combination thereof. The data stored in the auxiliary storage device 303 is loaded into the memory 302 as needed. The communication device 304 is a receiver and a transmitter. For example, the communication device 304 is a communication chip or NIC. The communication of the learning device 300 is performed using the communication device 304. The input / output interface 305 is a port to which an input device and an output device are connected. For example, the input / output interface 305 is a USB terminal, the input device is a keyboard and a mouse, and the output device is a display. The input / output of the learning device 300 is performed using the input / output interface 305.

[0028] The learning device 300 includes elements such as a data reception unit 310, a model generation unit 320, and a model output unit 330. These elements are realized by software.

[0029] The auxiliary storage device 303 stores a learning program for causing a computer to function as the data reception unit 310, the model generation unit 320, and the model output unit 330. The learning program is loaded into the memory 302 and executed by the processor 301. The auxiliary storage device 303 further stores an OS. At least a part of the OS is loaded into the memory 302 and executed by the processor 301. The processor 301 executes the learning program while executing the OS.

[0030] The input / output data of the learning program is stored in the storage unit 390. The memory 302 functions as a storage unit 390. However, storage devices such as the auxiliary storage device 303, the registers in the processor 301, and the cache memory in the processor 301 may function as the storage unit 390 instead of, or together with, the memory 302.

[0031] The learning device 300 may include a plurality of processors that replace the processor 301.

[0032] The learning program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or a flash memory.

[0033] Based on FIG. 4, the configuration of the inference device 400 will be described. The inference device 400 is a computer including hardware such as a processor 401, a memory 402, an auxiliary storage device 403, a communication device 404, and an input / output interface 405. These hardware components are connected to each other via signal lines.

[0034] The processor 401 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 401 is a CPU. The memory 402 is a volatile or non-volatile storage device. The memory 402 is also called a main storage device or a main memory. For example, the memory 402 is a RAM. The data stored in the memory 402 is saved in the auxiliary storage device 403 as needed. The auxiliary storage device 403 is a non-volatile storage device. For example, the auxiliary storage device 403 is a ROM, an HDD, a flash memory, or a combination thereof. The data stored in the auxiliary storage device 403 is loaded into the memory 402 as needed. The communication device 404 is a receiver and a transmitter. For example, the communication device 404 is a communication chip or a NIC. The communication of the inference device 400 is performed using the communication device 404. The input / output interface 405 is a port to which an input device and an output device are connected. For example, the input / output interface 405 is a USB terminal, the input device is a keyboard and a mouse, and the output device is a display. Input and output of the inference device 400 are performed using the input / output interface 405.

[0035] The inference device 400 includes elements such as an image reception unit 410, an inference unit 420, and an image output unit 430. These elements are realized by software.

[0036] The auxiliary storage device 403 stores an inference program for causing a computer to function as the image reception unit 410, the inference unit 420, and the image output unit 430. The inference program is loaded into the memory 402 and executed by the processor 401. The auxiliary storage device 403 further stores an OS. At least a part of the OS is loaded into the memory 402 and executed by the processor 401. The processor 401 executes the inference program while executing the OS.

[0037] The input / output data of the inference program is stored in the storage unit 490. The memory 402 functions as the storage unit 490. However, storage devices such as the auxiliary storage device 403, registers in the processor 401, and cache memory in the processor 401 may function as the storage unit 490 instead of or together with the memory 402.

[0038] The inference device 400 may include a plurality of processors that replace the processor 401.

[0039] The inference program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or a flash memory.

[0040] ***Description of Operations*** The procedure of the operation of the high-image-quality system 100 corresponds to the high-image-quality method.

[0041] Based on FIGS. 5 to 8, the outline of the high-image-quality method will be described. In step S110, the degradation processing device 200 degrades the reference image 112 based on the target satellite information 111.

[0042] FIG. 6 shows the outline of step S110. The target satellite information 111 is information on remote sensing performed by the target satellite. The target satellite is an artificial satellite that obtains an image (target image 131 described later) that is the target of high-image-quality improvement by remote sensing. The reference image 112 is an image with higher image quality than the target image 131. For example, the reference image 112 is an image with higher resolution than the target image 131 and is obtained by high-resolution remote sensing. An example of remote sensing is shooting from an aircraft. Specific examples of the aircraft are a helicopter, a UAV, an airplane, and an artificial satellite. UAV is an abbreviation for Unmanned Aerial Vehicle. Also, remote sensing may be performed by a heli-sat. A heli-sat is a helicopter-satellite communication system. The degraded image 113 is the degraded reference image 112. In other words, the degraded image 113 is the reference image 112 whose image quality has been lowered to the image quality level of the target image 131.

[0043] Returning to FIG. 5, the description will continue. Step S110 is executed for each of the plurality of reference images 112. By step S110, a plurality of pairs of the degraded image 113 and the reference image 112 are generated.

[0044] In step S120, the learning device 300 learns a plurality of pairs of the degraded image 113 and the reference image 112 to generate a learned model 122.

[0045] FIG. 7 shows the outline of step S120. The learning data 121 is a pair of a degraded image 113 and a reference image 112.

[0046] Returning to FIG. 5, the description will be continued. In step S130, the inference device 400 enhances the quality of the target image 131 using the learned model 122.

[0047] FIG. 8 shows an overview of step S130. The target image 131 is an image to be enhanced in quality and is obtained by remote sensing of the target satellite. The type of the target image 131 is an optical image. The estimated image 132 corresponds to the target image 131 with enhanced quality. That is, the estimated image 132 corresponds to the target image 131 with improved image quality.

[0048] Based on FIG. 9, the procedure of step S110 will be described. Steps S111 to S113 are executed for each reference image 112.

[0049] In step S111, the information reception unit 210 receives the target satellite information 111 and the reference image 112. For example, the user inputs the target satellite information 111 and the reference image 112 respectively into the degradation processing device 200. Then, the information reception unit 210 receives the input target satellite information 111 and the input reference image 112.

[0050] In step S112, the degradation processing unit 220 performs a degradation process on the reference image 112 based on the target satellite information 111. Thereby, the degraded image 113 is generated. The pair of the degraded image 113 and the reference image 112 is stored in the storage unit 290. The degradation process is a process for degrading the image quality. The degradation process will be described later.

[0051] In step S113, the image output unit 230 outputs the pair of the degraded image 113 and the reference image 112. For example, the image output unit 230 stores the pair of the degraded image 113 and the reference image 112 in a storage medium designated by the user. Alternatively, the image output unit 230 transmits the pair of the degraded image 113 and the reference image 112 to the learning device 300.

[0052] Based on FIG. 10, the degradation process in step S112 will be described. Specific examples of the degradation process are color shift processing, resolution processing, GSD processing, noise processing, compression processing, or combinations thereof.

[0053] Color shift processing is a process for reproducing color shift. Multiple optical sensors are mounted on an optical satellite. The multiple optical sensors have different light-receiving wavelength bands (bands) from each other and obtain a multi-band image. Then, a composite image is generated by synthesizing the multi-band images. At this time, color shift may occur in the composite image due to the influence of the physical position shift between the optical sensors. Also, color shift may occur at the sub-pixel level. The target satellite information 111 indicates the arrangement of multiple optical sensors mounted on the target satellite. Color shift processing performs downsampling of the reference image 112 by shifting the offset of downsampling for each band based on the arrangement of the multiple optical sensors. The offset of downsampling means the binning start position. Thereby, sub-pixel color shift is reproduced for the reference image 112, and the degraded image 113 is generated.

[0054] Resolution processing is a process for degrading the resolution. For the optical image obtained by the optical satellite, a PSF which is an index of resolution is determined. PSF is an abbreviation for Point Spread Function. The resolution is obtained by combining multiple degradation models considering the optical performance of the optical sensor, the aperture shape of the optical sensor, disturbances, image flow, and the like. The target satellite information 111 indicates the PSF for the target image 131. The resolution process applies the PSF to the reference image 112. For example, the PSF is multiplied in the frequency domain. Alternatively, the PSF is convolved and integrated with the reference image 112. As a result, the resolution of the reference image 112 is converted to be equivalent to that of the target image 131, and the degraded image 113 is generated. In the resolution process, the PSF of the reference image 112 may be used. The reference information indicating the PSF of the reference image 112 is received together with the reference image 112. Furthermore, the difference in PSF between bands may be incorporated into the resolution process. This makes it possible to reproduce aliasing for the reference image 112. The difference in PSF will be explained. The PSF may be different for each band. For example, the MTF of the multispectral band is higher than that of the panchromatic band's PSF. Therefore, aliasing may occur in the multispectral image. MTF is the abbreviation of Modulation Transfer Function.

[0055] The GSD process is a degradation process that uses GSD. GSD is the abbreviation of Ground Sampling Distance. The target satellite information 111 indicates the GSD of the target image 131. The GSD process compares the GSD of the target image 131 with the GSD of the reference image 112 and performs downsampling of the reference image 112 based on the comparison result. Specific examples of downsampling include bilinear interpolation or bicubic interpolation. The reference information indicating the GSD of the reference image 112 is received together with the reference image 112. As a result, the GSD of the target image 131 is simulated for the reference image 112, and the degraded image 113 is generated. In the GSD process, it is desirable that the ratio of the GSD of the reference image 112 to the GSD of the target image 131 is equal to or greater than the magnification of the high-resolution conversion by the high-quality system 100.

[0056] The noise process is a degradation process that adds noise. Target satellite information 111 indicates the noise characteristics of remote sensing of the target satellite. Specifically, target satellite information 111 indicates the sensitivity performance of remote sensing of the target satellite and the amount of light from the assumed subject. The noise processing estimates the signal-to-noise ratio of the target image 131 based on the sensitivity performance of the target satellite and the amount of light from the assumed subject, and adds noise corresponding to the signal-to-noise ratio to the reference image 112. When it is assumed that the noise is additive white Gaussian noise, the signal-to-noise ratio is related to the standard deviation of the Gaussian noise. Therefore, it is possible to perform a degradation process that applies noise equivalent to the noise of the target image 131 based on the signal-to-noise ratio. SN is an abbreviation for Signal to Noise Ratio. In the noise processing, noise other than additive white Gaussian noise may be added to the reference image 112. In this case, target satellite information 111 indicates the characteristics of noise other than additive white Gaussian noise. Specific examples of noise other than additive white Gaussian noise are stripe noise and FPN. FPN is an abbreviation for Fixed Pattern Noise. Also, when the target image 131 is a thermal infrared image, shot noise, 1 / f noise, or shading due to internal radiation may be added to the reference image 112. Also, noise due to the influence of the atmosphere may be added to the reference image 112. When the reference image 112 is obtained by high-resolution remote sensing performed within the atmosphere, the reference image 112 does not contain noise caused by the atmosphere. On the other hand, the target image 131 contains noise caused by the atmosphere. For example, noise such as a decrease in image contrast, shielding, and disturbance is included in the target image 131. Therefore, the noise processing uses a published atmospheric model to add noise due to the influence of the atmosphere to the reference image 112. Thereby, it is possible to generate an image simulating a satellite image.

[0057] The compression processing is a degradation process by data compression. When the target image 131 is downlinked to a ground station, data compression is performed. When the data compression is irreversible compression, image quality degradation due to compression occurs. Target satellite information 111 indicates a data compression method for target image 131. The compression process applies the data compression method for target image 131 to reference image 112. As a result, the same degradation as target image 131 can be reproduced in reference image 112.

[0058] Based on FIG. 11, the procedure of step S120 will be described. In step S121, data reception unit 310 receives a plurality of sets of degraded image 113 and reference image 112. For example, a user inputs a plurality of sets into learning device 300, and data reception unit 310 receives the input plurality of sets. Alternatively, degradation processing device 200 transmits a plurality of sets, and data reception unit 310 receives the plurality of sets. Then, data reception unit 310 stores a plurality of sets of degraded image 113 and reference image 112 in storage unit 390 as learning data 121.

[0059] In step S122, model generation unit 320 learns a plurality of sets of degraded image 113 and reference image 112. As a result, a learned model 122 is generated. Learned model 122 is stored in storage unit 390. Learned model 122 is a learned model for inferring a high-quality image corresponding to an input image. The high-quality image is an image corresponding to the input image with enhanced quality. The learning in step S122 will be described later.

[0060] In step S123, model output unit 330 outputs learned model 122. For example, model output unit 330 stores learned model 122 in a storage medium designated by the user. Alternatively, model output unit 330 transmits learned model 122 to inference device 400.

[0061] The learning in step S122 will be described. In the learning data 121, the reference image 112 is the correct answer to be inferred from the degraded image 113. That is, the reference image 112 is the teacher data corresponding to the degraded image 113. The model generation unit 320 learns the reference image 112 corresponding to the degraded image 113. That is, the model generation unit 320 learns the combination of the degraded image 113 and the reference image 112. Thereby, the learned model 122 is generated.

[0062] The model generation unit 320 can use known learning algorithms such as supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. When supervised learning is applied, the subject shown in the reference image 112 is the same as the subject shown in the target image 131. When unsupervised learning is applied, it is not necessary for the subject shown in the reference image 112 to be the same as the subject shown in the target image 131.

[0063] For example, the model generation unit 320 performs supervised learning using a neural network model as follows. Supervised learning is a method of inferring result data from input data, and learns the features in the learning data using the pair of the input data and the result data as the learning data. A neural network is composed of an input layer consisting of a plurality of neurons, one or more intermediate layers consisting of a plurality of neurons, and an output layer consisting of a plurality of neurons. The intermediate layer is also called a hidden layer.

[0064] FIG. 12 shows an example of a neural network model. "X1", "X2", "X3" constitute the input layer. "Y1", "Y2" constitute one intermediate layer. "Z1", "Z2", "Z3" constitute the output layer. In the input layer (X1~X3), a plurality of input values are input. When a plurality of input values are input to the input layer (X1~X3), the first weights (w11~w16) are multiplied by the plurality of input values. The plurality of values obtained thereby are referred to as a plurality of calculated values. The plurality of calculated values are input to the intermediate layer (Y1, Y2). When the plurality of calculated values are input to the intermediate layer (Y1, Y2), the second weights (w21~w26) are multiplied by the plurality of calculated values. The plurality of values obtained thereby are referred to as a plurality of output values. The plurality of output values are input to the output layer (Z1~Z3) and output from the output layer (Z1~Z3). The plurality of output values vary depending on the first weights (w11~w16) and the second weights (w21~W26).

[0065] In supervised learning, the model generation unit 320 inputs the degraded image 113 as a plurality of input values to the input layer, and adjusts the first weights and the second weights so that the plurality of output values output from the output layer approach the reference image 112.

[0066] Based on FIG. 13, the procedure of step S130 will be described. In step S130, the learned model 122 is stored in the storage unit 490. For example, a user inputs the learned model 122 to the learning device 300, and the inference unit 420 stores the input learned model 122 in the storage unit 490. Alternatively, the learning device 300 transmits the learned model 122, and the inference unit 420 receives the learned model 122 and stores it in the storage unit 490.

[0067] In step S131, the image reception unit 410 receives the target image 131. For example, a user inputs the target image 131 to the inference device 400, and the image reception unit 410 receives the input target image 131.

[0068] In step S132, the inference unit 420 infers the estimated image 132 from the target image 131 using the learned model 122. Specifically, the inference unit 420 calculates the learned model 122 with the target image 131 as the input. As a result, the estimated image 132 is output from the learned model 122.

[0069] In step S133, the image output unit 430 outputs the estimated image 132. For example, the image output unit 430 displays the estimated image 132 on a display.

[0070] ***Effect of Embodiment 1*** According to Embodiment 1, it is possible to obtain a high-quality model (learned model 122) for optical satellite images. And it is possible to obtain a high-quality image (estimated image 132) with high accuracy by using an actual satellite image (target image 131) as the input of the high-quality model.

[0071] ***Supplement to Embodiment 1*** High-quality enhancement means improvement in resolution, improvement in ground resolution, improvement in signal-to-noise ratio, improvement in color shift between bands, and the like.

[0072] The type of the target image 131 is an optical image. The optical image is an image obtained by exposing light rays. The light rays to be exposed may be light rays having wavelengths in any band of visible light, near-infrared light, and far-infrared light.

[0073] The learning device 300 may perform learning using learning data generated by other high-quality enhancement systems. When a plurality of high-quality enhancement systems are used in a plurality of areas, the learning device 300 may acquire learning data from a high-quality enhancement system used in the same area as the high-quality enhancement system 100, or may acquire learning data from a high-quality enhancement system used in an area different from the high-quality enhancement system 100. The learning device 300 may add each high-quality enhancement system to the system group that is the acquisition source of the learning data at any time, or may remove each high-quality enhancement system from the system group that is the acquisition source of the learning data at any time. The learning device 300 may use a learned model generated by another high-image-quality system. For example, the learning device 300 performs learning using the learning data 121 to update the learned model. The updated learned model becomes the learned model 122.

[0074] The learning algorithm of the learning device 300 may be deep learning that learns the extraction of the feature amount itself. Alternatively, the learning algorithm may be machine learning according to a genetic program, a functional logic program, a support vector machine, or the like.

[0075] FIG. 14 shows an example of the learning device 300. When a corresponding image 123 can be obtained in addition to the target image 131 by remote sensing of the target satellite, a pair of the target image 131 and the corresponding image 123 may be added to the learning data 121. The corresponding image 123 is an image corresponding to the target image 131. The corresponding image 123 is obtained in an observation mode different from the observation mode in which the target image 131 is obtained, and has a higher image quality than the target image 131. This embodiment is possible when the target image 131 is obtained in an observation mode other than the highest resolution observation mode.

[0076] The degradation processing device 200 functions as an input image generation device that generates an input image for learning. The learning device 300 functions as a model generation device that generates a learned model. The inference device 400 functions as a utilization device that utilizes the learned model.

[0077] Embodiment 1 is an example of a preferred form and is not intended to limit the technical scope of the present disclosure. Embodiment 1 may be partially implemented or may be implemented in combination with other forms. The procedures described using flowcharts and the like may be changed as appropriate.

[0078] Each element of each device of the high - quality system 100 may be implemented by any of software, hardware, firmware, or a combination thereof. The "section" of each element of each device of the high - quality system 100 may be read as "process", "step", "circuit", or "circuitry".

[0079] The aspects of the present disclosure are described below as appendices.

[0080] (Appendix 1) An information reception unit that receives target satellite information indicating information on remote sensing performed by a target satellite and a reference image with higher image quality than the target image obtained by the remote sensing; A degradation processing unit that generates a degraded image, which is the reference image with reduced image quality, by performing degradation processing on the reference image based on the target satellite information; A model generation unit that generates a learned model by learning a pair of the degraded image and the reference image; An image reception unit that receives the target image; An inference unit that infers an estimated image corresponding to the target image with enhanced image quality from the target image using the learned model; A high - quality system comprising the above.

[0081] (Appendix 2) The reference image is an image obtained by photographing from an aircraft, The aircraft is any one of a helicopter, an unmanned aerial vehicle, and an airplane, or a satellite that can obtain an image with higher image quality than the target image. The high - quality system according to Appendix 1.

[0082] (Appendix 3) The degradation process is any one of color shift processing, resolution processing, ground sampling distance processing, noise processing, and compression processing, or any combination of the color shift processing, the resolution processing, the ground sampling distance processing, the noise processing, and the previous compression processing, The target satellite information indicates the arrangement of a plurality of optical sensors mounted on the target satellite, and the color shift processing reproduces the color shift based on the arrangement of the plurality of optical sensors in the reference image. The target satellite information indicates the point spread function for the target image, and the resolution processing applies the point spread function to the reference image. The target satellite information indicates the ground sampling distance of the target image, and the ground sampling distance processing simulates the ground sampling distance for the reference image. The target satellite information indicates the noise characteristics of the remote sensing, and the noise processing adds noise based on the noise characteristics to the reference image. The target satellite information indicates the data compression method for the target image, and the compression processing applies the data compression method to the reference image. The high-quality image system according to Appendix 1 or Appendix 2.

[0083] (Appendix 4) The model generation unit learns the pair of the degraded image and the reference image and the pair of the target image and the corresponding image to generate the learned model. The corresponding image is obtained in an observation mode different from the observation mode in which the target image is obtained in the remote sensing, and has a higher image quality than the target image. The high-quality image system according to any one of Appendices 1 to 3.

[0084] (Appendix 5) The target satellite information indicates the arrangement of a plurality of optical sensors mounted on the target satellite. The degradation process performs downsampling of the reference image using different offsets for each band based on the arrangement of the plurality of optical sensors. The high-image-quality system according to Supplementary Note 1, Supplementary Note 2, or Supplementary Note 4.

[0085] The target image is an image obtained by exposing light rays having wavelengths in the visible light band. The high-image-quality system according to any one of Supplementary Notes 1 to 5.

[0086] (Supplementary Note 6) The target image is an image obtained by exposing light rays having wavelengths in the infrared light band. The high-image-quality system according to any one of Supplementary Notes 1 to 5.

Explanation of Reference Signs

[0087] 100 High-image-quality system, 111 Target satellite information, 112 Reference image, 113 Degraded image, 121 Learning data, 122 Trained model, 123 Corresponding image, 131 Target image, 132 Estimated image, 200 Degradation processing device, 201 Processor, 202 Memory, 203 Auxiliary storage device, 204 Communication device, 205 Input / output interface, 210 Information reception unit, 220 Degradation processing unit, 230 Image output unit, 290 Storage unit, 300 Learning device, 301 Processor, 302 Memory, 303 Auxiliary storage device, 304 Communication device, 305 Input / output interface, 310 Data reception unit, 320 Model generation unit, 330 Model output unit, 390 Storage unit, 400 Inference device, 401 Processor, 402 Memory, 403 Auxiliary storage device, 404 Communication device, 405 Input / output interface, 410 Image reception unit, 420 Inference unit, 430 Image output unit, 490 Storage unit.

Claims

1. A learning device for generating a learned model for inferring an estimated image corresponding to the target image with enhanced image quality from the target image obtained by remote sensing of a target satellite, a data reception unit that receives, as learning data, a set of a reference image with higher image quality than the target image and a degraded image that is the reference image with degraded image quality, a model generation unit that generates the learned model by learning the learning data, comprising: the degraded image is generated by applying a degradation process based on target satellite information indicating the remote sensing information to the reference image, the degradation process includes either a color shift process or a compression process, the color shift process reproduces a color shift based on the arrangement of a plurality of optical sensors mounted on the target satellite indicated by the target satellite information in the reference image, the compression process applies a data compression method for the target image indicated by the target satellite information to the reference image learning device.

2. A learning method for generating a learned model for inferring an estimated image corresponding to the target image with enhanced image quality from the target image obtained by remote sensing of a target satellite, receiving, as learning data, a set of a reference image with higher image quality than the target image and a degraded image that is the reference image with degraded image quality, generating the learned model by learning the learning data, the degraded image is generated by applying a degradation process based on target satellite information indicating the remote sensing information to the reference image, the degradation process includes either a color shift process or a compression process, the color shift process reproduces a color shift based on the arrangement of a plurality of optical sensors mounted on the target satellite indicated by the target satellite information in the reference image, The compression process applies, to the reference image, a data compression method for the target image indicated by the target satellite information. Learning method.

3. A learning program for generating a learned model for inferring an estimated image corresponding to the target image with enhanced image quality from the target image obtained by remote sensing of a target satellite, The learning program includes: A data reception unit that receives, as learning data, a set of a reference image with higher image quality than the target image and a degraded image that is the reference image with degraded image quality; A model generation unit that generates the learned model by learning the learning data; causing a computer to function, The degraded image is generated by subjecting the reference image to a degradation process based on target satellite information indicating the remote sensing information, The degradation process includes either a color shift process or a compression process, The color shift process reproduces, on the reference image, a color shift based on the arrangement of a plurality of optical sensors mounted on the target satellite, indicated by the target satellite information, The compression process applies, to the reference image, a data compression method for the target image indicated by the target satellite information. Learning program.

4. An image reception unit that receives a target image obtained by remote sensing of a target satellite; An inference unit that uses the learned model to infer an estimated image corresponding to the target image with enhanced image quality from the target image; comprising The learned model is generated by learning a set of a degraded image and a reference image, The reference image is an image with higher image quality than the target image, The degraded image is the reference image with reduced image quality, and is generated by performing a degradation process based on target satellite information indicating the remote sensing information on the reference image. The degradation process includes either a color shift process or a compression process. The color shift process reproduces a color shift based on the arrangement of a plurality of optical sensors mounted on the target satellite indicated by the target satellite information in the reference image. The compression process applies a data compression method for the target image indicated by the target satellite information to the reference image. Inference device.

5. Receive a target image obtained by remote sensing of a target satellite, An inference method for inferring an estimated image corresponding to the target image with enhanced image quality from the target image using a learned model, The learned model is generated by learning a pair of a degraded image and a reference image, The reference image is an image with higher image quality than the target image, The degraded image is the reference image with reduced image quality, and is generated by performing a degradation process based on target satellite information indicating the remote sensing information on the reference image. The degradation process includes either a color shift process or a compression process. The color shift process reproduces a color shift based on the arrangement of a plurality of optical sensors mounted on the target satellite indicated by the target satellite information in the reference image. The compression process applies a data compression method for the target image indicated by the target satellite information to the reference image. Inference method.

6. An image reception unit that receives a target image obtained by remote sensing of a target satellite, As an inference unit that infers an estimated image corresponding to the target image with enhanced image quality from the target image using a learned model An inference program that causes a computer to function, The learned model is generated by learning a pair of a degraded image and a reference image, The reference image is an image with higher image quality than the target image, The degraded image is a reference image with reduced image quality, and is generated by subjecting the reference image to a degradation process based on target satellite information indicating information of remote sensing performed by a target satellite. The degradation process includes either a color shift process or a compression process. The color shift process reproduces a color shift based on the arrangement of a plurality of optical sensors mounted on the target satellite indicated by the target satellite information in the reference image. The compression process applies a data compression method for the target image indicated by the target satellite information to the reference image. Inference program.

7. An information reception unit that receives target satellite information indicating information of remote sensing performed by a target satellite and a reference image with higher image quality than the target image obtained by the remote sensing, A degradation processing unit that generates a degraded image, which is the reference image with reduced image quality, by subjecting the reference image to a degradation process based on the target satellite information. A model generation unit that generates a learned model by learning a pair of the degraded image and the reference image. An image reception unit that receives the target image, An inference unit that infers an estimated image corresponding to the target image with enhanced image quality from the target image using the learned model. comprising The degradation process includes either a color shift process or a compression process. The color shift process reproduces the color shift based on the arrangement of a plurality of optical sensors mounted on the target satellite indicated by the target satellite information in the reference image. The compression process applies the data compression method for the target image indicated by the target satellite information to the reference image. High-image-quality system.

8. The reference image is an image obtained by photographing from a flying object, and the flying object is any one of a helicopter, an unmanned aerial vehicle, and an aircraft, or a satellite that can obtain an image with higher image quality than the target image. The high-image-quality system according to claim 7.

9. The degradation process includes any one of resolution processing, ground sampling distance processing, and noise processing. The resolution processing reproduces aliasing for the reference image when there is a difference in the point spread function between bands based on the point spread function for the target image indicated by the target satellite information. The ground sampling distance processing simulates the ground sampling distance for the reference image based on the ground sampling distance of the target image indicated by the target satellite information. The noise processing adds noise to the reference image based on the noise characteristics of the remote sensing indicated by the target satellite information. The high-image-quality system according to claim 7.

10. An information reception unit that receives target satellite information indicating information on remote sensing performed by a target satellite and a reference image having higher image quality than the target image obtained by the remote sensing, A degradation processing unit that generates a degraded image, which is the reference image with reduced image quality, by subjecting the reference image to a degradation process based on the target satellite information. A model generation unit that generates a learned model by learning a set of the degraded image and the reference image and a set of the target image and the corresponding image; An image reception unit that receives the target image; An inference unit that infers an estimated image corresponding to the target image with enhanced image quality from the target image using the learned model; and comprising; wherein the corresponding image is obtained in an observation mode different from the observation mode in which the target image is obtained in the remote sensing, and has higher image quality than the target image An image quality enhancement system.

11. An information reception unit that receives target satellite information indicating information on remote sensing performed by a target satellite and a reference image having higher image quality than the target image obtained by the remote sensing; A degradation processing unit that generates a degraded image, which is the reference image with reduced image quality, by performing degradation processing based on the target satellite information on the reference image; A model generation unit that generates a learned model by learning a set of the degraded image and the reference image; An image reception unit that receives the target image; An inference unit that infers an estimated image corresponding to the target image with enhanced image quality from the target image using the learned model; and comprising; wherein the target satellite information indicates the arrangement of a plurality of optical sensors mounted on the target satellite, and the degradation processing performs downsampling of the reference image using different offsets for each band based on the arrangement of the plurality of optical sensors. An image quality enhancement system.

12. The target image is an image obtained by exposing light rays having wavelengths in the visible light band. The image quality enhancement system according to claim 7.

13. The target image is an image obtained by exposing a light beam having a wavelength in the infrared band. The high image quality system according to claim 7.

14. Receiving target satellite information indicating information of remote sensing performed by a target satellite and a reference image having a higher image quality than the target image obtained by the remote sensing, Generating a degraded image, which is the reference image with reduced image quality, by performing a degradation process based on the target satellite information on the reference image, Generating a learned model by learning a pair of the degraded image and the reference image, Receiving the target image, Inferring an estimated image corresponding to the target image with enhanced image quality from the target image using the learned model A high image quality method, The degradation process includes either a color shift process or a compression process, The color shift process reproduces a color shift based on the arrangement of a plurality of optical sensors mounted on the target satellite indicated by the target satellite information in the reference image, The compression process applies a data compression method for the target image indicated by the target satellite information to the reference image High image quality method.

15. A learning device for generating a learned model for inferring an estimated image corresponding to the target image with enhanced image quality from a target image obtained by remote sensing of a target satellite, A data reception unit that receives, as learning data, a pair of a reference image having a higher image quality than the target image and a degraded image that is the reference image with reduced image quality, and a pair of the target image and a corresponding image, A model generation unit that generates the learned model by learning the learning data, comprising, The degraded image is generated by performing a degradation process on the reference image based on target satellite information indicating the information of the remote sensing, The corresponding image is obtained in an observation mode different from the observation mode in which the target image is obtained in the remote sensing, and has higher image quality than the target image Learning device.

16. A learning device that generates a learned model for inferring an estimated image corresponding to the target image with enhanced image quality from a target image obtained by remote sensing of a target satellite, A data reception unit that receives, as learning data, a set of a reference image having higher image quality than the target image and a degraded image that is the reference image with degraded image quality, A model generation unit that generates the learned model by learning the learning data, comprising The degraded image is generated by performing a degradation process on the reference image based on target satellite information indicating the information of the remote sensing, The target satellite information indicates the arrangement of a plurality of optical sensors mounted on the target satellite, The degradation process performs downsampling of the reference image using different offsets for each band based on the arrangement of the plurality of optical sensors, Learning device.

17. A learning method for generating a learned model for inferring an estimated image corresponding to the target image with enhanced image quality from a target image obtained by remote sensing of a target satellite, receiving, as learning data, a set of a reference image having higher image quality than the target image and a degraded image that is the reference image with degraded image quality, and a set of the target image and a corresponding image, generating the learned model by learning the learning data, The degraded image is generated by subjecting the reference image to a degradation process based on target satellite information indicating the information of the remote sensing, The corresponding image is obtained in an observation mode different from the observation mode in which the target image is obtained in the remote sensing, and has higher image quality than the target image Learning method.

18. A learning method for generating a learned model for inferring an estimated image corresponding to the target image with enhanced image quality from the target image obtained by remote sensing of a target satellite, accepting, as learning data, a set of a reference image having higher image quality than the target image and a degraded image that is the reference image with degraded image quality, generating the learned model by learning the learning data, the degraded image is generated by subjecting the reference image to a degradation process based on target satellite information indicating the information of the remote sensing, the target satellite information indicates the arrangement of a plurality of optical sensors mounted on the target satellite, the degradation process performs downsampling of the reference image using different offsets for each band based on the arrangement of the plurality of optical sensors, Learning method.

19. An image reception unit that receives a target image obtained by remote sensing of a target satellite, an inference unit that uses a learned model to infer an estimated image corresponding to the target image with enhanced image quality from the target image, comprising the learned model is generated by learning a set of a degraded image and a reference image and a set of the target image and a corresponding image, the reference image is an image having higher image quality than the target image, The corresponding image is obtained in an observation mode different from the observation mode in which the target image is obtained in the remote sensing, and is an image with higher image quality than the target image. The degraded image is the reference image with reduced image quality, and is generated by performing a degradation process based on target satellite information indicating the information of the remote sensing on the reference image. Inference device.

20. An image reception unit that receives a target image obtained by remote sensing of a target satellite, An inference unit that uses a learned model to infer an estimated image corresponding to the target image with enhanced image quality from the target image. comprising The learned model is generated by learning a pair of a degraded image and a reference image. The reference image is an image with higher image quality than the target image. The degraded image is the reference image with reduced image quality, and is generated by performing a degradation process based on target satellite information indicating the information of the remote sensing on the reference image. The target satellite information indicates the arrangement of a plurality of optical sensors mounted on the target satellite. The degradation process performs downsampling of the reference image using different offsets for each band based on the arrangement of the plurality of optical sensors. Inference device.

21. A method of inferring an estimated image corresponding to the target image with enhanced image quality from the target image obtained by remote sensing of a target satellite, by receiving the target image and using a learned model, wherein the learned model is generated by learning a pair of a degraded image and a reference image and a pair of the target image and a corresponding image. The reference image is an image with higher image quality than the target image. ​The corresponding image is obtained in an observation mode different from the observation mode in which the target image is obtained in the remote sensing, and is an image with higher image quality than the target image. The degraded image is the reference image with reduced image quality, and is generated by subjecting the reference image to a degradation process based on target satellite information indicating the information of the remote sensing. Inference method.

22. Receiving a target image obtained by remote sensing of a target satellite. An inference method for inferring an estimated image corresponding to the target image with enhanced image quality from the target image using a learned model, The learned model is generated by learning a pair of a degraded image and a reference image. The reference image is an image with higher image quality than the target image. The degraded image is the reference image with reduced image quality, and is generated by subjecting the reference image to a degradation process based on target satellite information indicating the information of the remote sensing. The target satellite information indicates the arrangement of a plurality of optical sensors mounted on the target satellite. The degradation process performs downsampling of the reference image using different offsets for each band based on the arrangement of the plurality of optical sensors. Inference method.

Images