Learning device, learning method, learning program, inference device, inference method, inference program, and image quality improvement system

Abstract

To generate a high-resolution enhancement model using an image having a resolution exceeding the maximum resolution of a target satellite as a teacher image.SOLUTION: A learning device 300 accepts a pair of a reference image having higher image quality than a target image obtained by remote sensing of a target satellite and a degraded image, which is a version of a reference image with reduced image quality, as training data. The learning device 300 generates a trained model by learning the training data. The degraded image is generated by subjecting the reference image to color shift processing or compression processing.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to improving the image quality of images. [Background technology]

[0002] Conventionally, ground sensors have been used to grasp the damage caused by disasters such as floods, heavy rain, earthquakes, etc. Methods using ground sensors are useful for grasping the damage situation at specific spots such as houses and power plants. On the other hand, when damage occurs over a wide area, it is difficult to obtain surface information to identify areas with greater damage using methods that use ground sensors.

[0003] In recent years, analytical methods using remote sensing data have been proposed to grasp the extent of damage caused by wide-area disasters. Examples of remote sensing include sensing from flying objects such as aircraft, UAVs, optical satellites, and SAR satellites. UAV is an abbreviation for Unmanned Aerial Vehicle, also known as a drone. SAR is an abbreviation for Synthetic Aperture Radar.

[0004] Images obtained by optical satellites are used as a means of monitoring various objects on the Earth's surface. There is also a technique for increasing the resolution of images using a high-resolution model, also known as a super-resolution model, which is generated by machine learning.

[0005] Patent Document 1 discloses a technique for increasing the resolution of an image using a high-resolution model. The high-resolution model is generated as follows: First, a high-resolution image is prepared. Next, a low-resolution image is generated based on the high-resolution image. Then, the pair of high-resolution and low-resolution images is trained as a pair of teacher image and input image to generate the high-resolution model. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Application No. 2021-502251 [Patent Document 2] Japanese Patent Application Publication No. 2017-208813 [Non-patent literature]

[0007] [Non-Patent Document 1] Xiang Zhu et al., Super-Resolving Commercial Satellite Imagery Using Realistic Training Data, 2020 IEEE International Conference on Image Processing (ICIP), USA, IEEE, September 30, 2020, pp. 498-502, https: / / ieeexplore.ieee.org / document / 9190746 Summary of the Invention [Problem to be solved by the invention]

[0008] In conventional high-resolution techniques (including the technique of Patent Document 1), a high-resolution model is generated by machine learning using a set of a high-resolution image and a low-resolution image as learning data. A low-resolution image is generated by performing degradation processing such as downsampling on a high-resolution image. When images obtained by a target satellite are used as learning data, images having a resolution exceeding the maximum resolution of the target satellite cannot be used as training images. Therefore, even if a high-resolution model is used, it is not possible to generate an image having a resolution that exceeds the maximum resolution of the target satellite. Furthermore, since there are differences in the image domain between the target satellite and other remote sensing, even if images from other remote sensing are used as learning data, it is not possible to highly accurately increase the resolution of the target satellite's images.

[0009] The present disclosure aims to enable generation of a high-resolution model using an image having a resolution exceeding the maximum resolution of the target satellite as a training image. [Means for solving the problem]

[0010] The learning device of the present disclosure includes: A trained model is generated for inferring an estimated image corresponding to a target image with improved image quality from a target image obtained by remote sensing of the target satellite. The learning device a data receiving unit that receives, as learning data, a pair of a reference image having a higher image quality than the target image and a degraded image that is the reference image having a lower image quality; and a model generation unit that generates the trained model by learning the training data. The degraded image is generated by performing color shift processing or compression processing on the reference image. [Effects of the Invention]

[0011] According to the present disclosure, a high-resolution model can be generated using an image having a resolution exceeding the maximum resolution of the target satellite as a training image. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a configuration diagram of an image quality improvement system 100 according to a first embodiment. [Figure 2] FIG. 1 is a configuration diagram of a degradation processing device 200 according to the first embodiment. [Figure 3] FIG. 2 is a configuration diagram of a learning device 300 according to the first embodiment. [Figure 4] FIG. 1 is a configuration diagram of an inference device 400 according to the first embodiment. [Figure 5] 3 is a flowchart of an image quality improvement method according to the first embodiment. [Figure 6] FIG. 4 is a schematic diagram of step S110 in the first embodiment. [Figure 7] FIG. 4 is a schematic diagram of step S120 in the first embodiment. [Figure 8] FIG. 4 is a schematic diagram of step S130 according to the first embodiment. [Figure 9] 10 is a flowchart of step S110 in the first embodiment. [Figure 10] FIG. 4 is a schematic diagram of step S112 according to the first embodiment. [Figure 11] 10 is a flowchart of step S120 in the first embodiment. [Figure 12] FIG. 2 is a diagram showing an example of a neural network model according to the first embodiment. [Figure 13] 10 is a flowchart of step S130 in the first embodiment. [Figure 14] FIG. 2 is a diagram showing an example of the learning device 300 according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] In the embodiments and drawings, the same or corresponding elements are denoted by the same reference numerals. The description of elements denoted by the same reference numerals as those already described will be omitted or simplified as appropriate. Arrows in the drawings primarily indicate the flow of data or the flow of processing.

[0014] Embodiment 1 The image quality improvement system 100 will be described with reference to FIGS.

[0015] ***Configuration Description*** The configuration of the image quality improvement system 100 will be described with reference to FIG. The image quality improvement system 100 includes a degradation processing device 200, a learning device 300, and an inference device 400.

[0016] The configuration of the degradation processing device 200 will be described with reference to FIG. The degradation processing device 200 is a computer that includes hardware such as a processor 201, a memory 202, an auxiliary storage device 203, a communication device 204, and an input / output interface 205. These pieces of hardware are connected to one another via signal lines.

[0017] The processor 201 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 201 is a CPU. IC is an abbreviation for Integrated Circuit. CPU is an abbreviation for Central Processing Unit.

[0018] The memory 202 is a volatile or non-volatile storage device. The memory 202 is also called a primary storage device or a main memory. For example, the memory 202 is a RAM. Data stored in the memory 202 is saved in the secondary storage device 203 as needed. RAM is an abbreviation for Random Access Memory.

[0019] The auxiliary storage device 203 is a non-volatile storage device. For example, the auxiliary storage device 203 is a ROM, a HDD, a flash memory, or a combination thereof. Data stored in the auxiliary storage device 203 is loaded into the memory 202 as needed. ROM is an abbreviation for Read Only Memory. HDD is an abbreviation for Hard Disk Drive.

[0020] The communication device 204 is a receiver and a transmitter. For example, the communication device 204 is a communication chip or a NIC. The communication of the degradation processing device 200 is performed using the communication device 204. NIC is an abbreviation for Network Interface Card.

[0021] The input / output interface 205 is a port to which an input device and an output device are connected. For example, the input / output interface 205 is a USB terminal, the input devices are a keyboard and a mouse, and the output device is a display. Input and output to and from the degradation processing device 200 are performed using the input / output interface 205. USB is an abbreviation for Universal Serial Bus.

[0022] The degradation processing device 200 includes elements such as an information receiving unit 210, a degradation processing unit 220, and an image output unit 230. These elements are realized by software.

[0023] The auxiliary storage device 203 stores a degradation processing program for causing the computer to function as the information receiving unit 210, the degradation processing unit 220, and the image output unit 230. The degradation processing program is loaded into the memory 202 and executed by the processor 201. The auxiliary storage device 203 also stores an OS. At least a part of the OS is loaded into the memory 202 and executed by the processor 201. The processor 201 executes the deterioration processing program while running the OS. OS is an abbreviation for Operating System.

[0024] Input and output data of the degradation processing program are stored in the storage unit 290 . The memory 202 functions as the storage unit 290. However, a storage device such as the auxiliary storage device 203, a register in the processor 201, or a cache memory in the processor 201 may function as the storage unit 290 instead of or together with the memory 202.

[0025] The degradation processing device 200 may include a plurality of processors that replace the processor 201 .

[0026] The degradation processing program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or a flash memory.

[0027] The configuration of the learning device 300 will be described with reference to FIG. The learning device 300 is a computer that includes hardware such as a processor 301, a memory 302, an auxiliary storage device 303, a communication device 304, and an input / output interface 305. These pieces of hardware are connected to each other via signal lines.

[0028] The processor 301 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 301 is a CPU. The memory 302 is a volatile or non-volatile storage device. The memory 302 is also called a primary storage device or a main memory. For example, the memory 302 is a RAM. Data stored in the memory 302 is saved in the secondary storage device 303 as needed. The auxiliary storage device 303 is a non-volatile storage device. For example, the auxiliary storage device 303 is a ROM, a HDD, a flash memory, or a combination thereof. Data stored in the auxiliary storage device 303 is loaded into the memory 302 as needed. The communication device 304 is a receiver and a transmitter. For example, the communication device 304 is a communication chip or a NIC. Communication of the learning device 300 is performed using the communication device 304. The input / output interface 305 is a port to which an input device and an output device are connected. For example, the input / output interface 305 is a USB terminal, the input device is a keyboard and a mouse, and the output device is a display. Input and output of the learning device 300 is performed using the input / output interface 305.

[0029] The learning device 300 includes elements such as a data receiving unit 310, a model generating unit 320, and a model output unit 330. These elements are realized by software.

[0030] The auxiliary storage device 303 stores a learning program for causing the computer to function as a data receiving unit 310, a model generating unit 320, and a model output unit 330. The learning program is loaded into the memory 302 and executed by the processor 301. The auxiliary storage device 303 also stores an OS. At least a part of the OS is loaded into the memory 302 and executed by the processor 301. The processor 301 executes the learning program while running the OS.

[0031] The input and output data of the learning program are stored in the storage unit 390. The memory 302 functions as the storage unit 390. However, a storage device such as the auxiliary storage device 303, a register in the processor 301, or a cache memory in the processor 301 may function as the storage unit 390 instead of the memory 302 or together with the memory 302.

[0032] The learning device 300 may include multiple processors replacing the processor 301.

[0033] The learning program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or flash memory.

[0034] The configuration of the inference device 400 will be described with reference to FIG. The inference device 400 is a computer that includes hardware such as a processor 401, a memory 402, an auxiliary storage device 403, a communication device 404, and an input / output interface 405. These pieces of hardware are connected to one another via signal lines.

[0035] The processor 401 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 401 is a CPU. The memory 402 is a volatile or non-volatile storage device. The memory 402 is also called a primary storage device or a main memory. For example, the memory 402 is a RAM. Data stored in the memory 402 is saved in the secondary storage device 403 as needed. The auxiliary storage device 403 is a non-volatile storage device. For example, the auxiliary storage device 403 is a ROM, a HDD, a flash memory, or a combination thereof. Data stored in the auxiliary storage device 403 is loaded into the memory 402 as needed. The communication device 404 is a receiver and a transmitter. For example, the communication device 404 is a communication chip or a NIC. The communication of the reasoning device 400 is performed using the communication device 404. The input / output interface 405 is a port to which an input device and an output device are connected. For example, the input / output interface 405 is a USB terminal, the input devices are a keyboard and a mouse, and the output device is a display. Input and output of the inference device 400 is performed using the input / output interface 405.

[0036] The inference device 400 comprises elements such as an image receiving unit 410, an inference unit 420, and an image output unit 430. These elements are realized by software.

[0037] The auxiliary storage device 403 stores an inference program for causing the computer to function as an image receiving unit 410, an inference unit 420, and an image output unit 430. The inference program is loaded into the memory 402 and executed by the processor 401. The auxiliary storage device 403 also stores an OS. At least a part of the OS is loaded into the memory 402 and executed by the processor 401. The processor 401 executes an inference program while running the OS.

[0038] Input and output data of the inference program are stored in the storage unit 490. The memory 402 functions as the storage unit 490. However, a storage unit such as the auxiliary storage unit 403, a register in the processor 401, or a cache memory in the processor 401 may function as the storage unit 490 instead of or together with the memory 402.

[0039] Reasoning apparatus 400 may include multiple processors replacing processor 401 .

[0040] The inference program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or flash memory.

[0041] ***Explanation of Operation*** The procedure of operation of the image quality improvement system 100 corresponds to an image quality improvement method.

[0042] The image quality improvement method will be outlined with reference to FIGS. In step S110 , the degradation processor 200 degrades the reference image 112 based on the target satellite information 111 .

[0043] FIG. 6 shows an overview of step S110. The target satellite information 111 is information on the remote sensing performed by the target satellite. The target satellite is an artificial satellite that obtains, by remote sensing, an image (a target image 131, described later) that is a target for improving image quality. The reference image 112 is an image with higher image quality than the target image 131. For example, the reference image 112 is an image with higher resolution than the target image 131, and is obtained by high-resolution remote sensing. One example of remote sensing is photography from an airborne vehicle. Specific examples of airborne vehicles are helicopters, UAVs, aircraft, and artificial satellites. UAV is an abbreviation for Unmanned Aerial Vehicle. Remote sensing may also be performed by a HeliSat. A HeliSat is a helicopter satellite communication system. The degraded image 113 is the degraded reference image 112. In other words, the degraded image 113 is the reference image 112 whose image quality has been reduced to the image quality level of the target image 131.

[0044] Returning to FIG. 5, the explanation will continue. Step S110 is performed for each of the multiple reference images 112. In step S110, a plurality of pairs of the degraded image 113 and the reference image 112 are generated.

[0045] In step S120, the learning device 300 learns a plurality of pairs of the degraded image 113 and the reference image 112 to generate a learned model 122.

[0046] FIG. 7 shows an overview of step S120. The learning data 121 is a set of a degraded image 113 and a reference image 112 .

[0047] Returning to FIG. 5, the explanation will continue. In step S130, inference device 400 uses trained model 122 to improve the image quality of target image 131.

[0048] FIG. 8 shows an overview of step S130. The target image 131 is an image that is a target for image quality improvement and is obtained by remote sensing of a target satellite. The type of the target image 131 is an optical image. The estimated image 132 corresponds to the image quality of the target image 131. In other words, the estimated image 132 corresponds to the target image 131 with improved image quality.

[0049] The procedure of step S110 will be described with reference to FIG. Steps S111 to S113 are executed for each reference image 112.

[0050] In step S111, the information receiving unit 210 receives the target satellite information 111 and the reference image 112. For example, a user inputs the target satellite information 111 and the reference image 112 to the degradation processing device 200. Then, the information receiving unit 210 receives the input target satellite information 111 and the input reference image 112.

[0051] In step S112, the degradation processor 220 performs degradation processing based on the target satellite information 111 on the reference image 112. This generates a degraded image 113. The pair of the degraded image 113 and the reference image 112 is stored in the storage unit 290. The degradation process is a process for degrading the image quality of an image, and will be described later.

[0052] In step S113, the image output unit 230 outputs a pair of the degraded image 113 and the reference image 112. For example, the image output unit 230 stores the pair of the degraded image 113 and the reference image 112 in a storage medium designated by the user, or transmits the pair of the degraded image 113 and the reference image 112 to the learning device 300.

[0053] The degradation process in step S112 will be described with reference to FIG. Specific examples of degradation processing include color shift processing, resolution processing, GSD processing, noise processing, compression processing, or a combination of these.

[0054] The color shift processing is a process for reproducing color shift. Optical satellites are equipped with multiple optical sensors. Each optical sensor receives light in a different wavelength band, obtaining multi-band images. The multi-band images are then combined to generate a composite image. During this process, color shifts may occur in the composite image due to the influence of misalignment of the physical positions of the optical sensors. Color shifts may also occur at the sub-pixel level. The target satellite information 111 indicates the arrangement of a plurality of optical sensors mounted on the target satellite. The color shift processing shifts the downsampling offset for each band based on the arrangement of multiple optical sensors, and performs downsampling of the reference image 112. The downsampling offset means the binning start position. As a result, the sub-pixel color shift is reproduced with respect to the reference image 112, and a degraded image 113 is generated.

[0055] Resolution processing is a process that degrades the resolution. The PSF, an index of resolution, is determined for optical images obtained by optical satellites. PSF stands for Point Spread Function. Resolution is determined by combining multiple degradation models that take into account the optical performance of the optical sensor, the aperture shape of the optical sensor, and the effects of turbulence and image streaming. The target satellite information 111 indicates a PSF for the target image 131 . The resolution process applies the PSF to the reference image 112. For example, the PSF is multiplied in frequency space, or the PSF is convolved with the reference image 112. As a result, the resolution of the reference image 112 is converted to a resolution equivalent to that of the target image 131, and a degraded image 113 is generated. In the resolution processing, the PSF of the reference image 112 may be used. Reference information indicating the PSF of the reference image 112 is received together with the reference image 112. Furthermore, the difference in PSF between bands may be incorporated into the resolution processing. This makes it possible to reproduce aliasing with respect to the reference image 112. The difference in PSF will now be explained. The PSF may differ for each band. For example, the MTF of the multispectral band is higher than the PSF of the panchromatic band. This may result in aliasing in the multispectral image. MTF is an abbreviation for Modulation Transfer Function.

[0056] GSD processing is a degradation process that uses GSD. GSD is an abbreviation for Ground Sampling Distance. The target satellite information 111 indicates the GSD of the target image 131 . The GSD process compares the GSD of the target image 131 with the GSD of the reference image 112, and downsamples the reference image 112 based on the comparison result. Examples of downsampling include bilinear interpolation and bicubic interpolation. Reference information indicating the GSD of the reference image 112 is received along with the reference image 112. As a result, the GSD of the target image 131 is simulated relative to the reference image 112, and the degraded image 113 is generated. In the GSD processing, it is desirable that the ratio of the GSD of the reference image 112 to the GSD of the target image 131 be equal to or greater than the magnification of the resolution enhancement by the image quality enhancement system 100 .

[0057] Noise processing is a degradation process that adds noise. The target satellite information 111 indicates the noise characteristics of the remote sensing of the target satellite. Specifically, the target satellite information 111 indicates the sensitivity performance of the remote sensing of the target satellite and the amount of light from the assumed subject. Noise processing estimates the signal-to-noise ratio of the target image 131 based on the sensitivity performance of the target satellite and the amount of light from the assumed subject, and adds noise corresponding to the signal-to-noise ratio to the reference image 112. If the noise is assumed to be additive white Gaussian noise, the signal-to-noise ratio is related to the standard deviation of the Gaussian noise. Therefore, degradation processing is possible by applying noise equivalent to the noise in the target image 131 based on the signal-to-noise ratio. SN is an abbreviation for Signal to Noise Ratio. In the noise processing, noise other than additive white Gaussian noise may be added to the reference image 112. In this case, the target satellite information 111 exhibits the characteristics of noise other than additive white Gaussian noise. Specific examples of noise other than additive white Gaussian noise include stripe noise and FPN. FPN is an abbreviation for fixed pattern noise. Furthermore, if the target image 131 is a thermal infrared image, shot noise, 1 / f noise, or shading due to internal radiation may be added to the reference image 112 . Furthermore, noise due to atmospheric influences may be added to the reference image 112. If the reference image 112 is obtained by high-resolution remote sensing that takes images within the atmosphere, the reference image 112 does not contain noise due to the atmosphere. On the other hand, the target image 131 contains noise due to the atmosphere. For example, noise due to a decrease in image contrast, occlusion, turbulence, etc. is contained in the target image 131. Therefore, noise processing adds noise due to atmospheric influences to the reference image 112 using a publicly available atmospheric model. This makes it possible to generate an image that simulates a satellite image.

[0058] The compression process is a degradation process due to data compression. When the target image 131 is downlinked to the ground station, the data is compressed. If the data compression is lossy, image quality degradation occurs due to the compression. The target satellite information 111 indicates the data compression method for the target image 131 . The compression process applies a data compression technique to the target image 131 to the reference image 112 . This allows the same degradation as that of the target image 131 to be reproduced in the reference image 112.

[0059] The procedure of step S120 will be described with reference to FIG. In step S121, the data receiving unit 310 receives a plurality of pairs of the degraded image 113 and the reference image 112. For example, the user inputs multiple pairs to the learning device 300, and the data accepting unit 310 accepts the input multiple pairs. Alternatively, the degradation processing device 200 transmits multiple pairs, and the data accepting unit 310 receives the multiple pairs. Then, the data receiving unit 310 stores a plurality of pairs of the degraded image 113 and the reference image 112 as learning data 121 in the storage unit 390 .

[0060] In step S122, the model generation unit 320 learns a plurality of pairs of the degraded image 113 and the reference image 112. This generates a trained model 122. The trained model 122 is stored in the storage unit 390. The trained model 122 is a trained model for inferring a high-quality image corresponding to an input image. The high-quality image is an image equivalent to the input image whose quality has been improved. The learning in step S122 will be described later.

[0061] In step S123, the model output unit 330 outputs the trained model 122. For example, the model output unit 330 saves the trained model 122 in a storage medium designated by the user. Alternatively, the model output unit 330 transmits the trained model 122 to the inference device 400.

[0062] The learning in step S122 will now be described. In the learning data 121, the reference image 112 is a correct answer to be inferred from the degraded image 113. In other words, the reference image 112 is training data corresponding to the degraded image 113. The model generation unit 320 learns the reference image 112 corresponding to the degraded image 113. That is, the model generation unit 320 learns a combination of the degraded image 113 and the reference image 112. In this way, a trained model 122 is generated.

[0063] The model generator 320 can use known learning algorithms such as supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. When supervised learning is applied, the object captured in the reference image 112 is the same as the object captured in the target image 131 . When unsupervised learning is applied, the object in the reference image 112 does not have to be the same as the object in the target image 131 .

[0064] For example, the model generation unit 320 performs supervised learning using a neural network model as follows. Supervised learning is a method of inferring result data from input data, in which a set of input data and result data is used as training data to learn the features of the training data. A neural network consists of an input layer of neurons, one or more hidden layers of neurons, and an output layer of neurons. The hidden layers are also called input and output layers.

[0065] FIG. 12 shows an example of a neural network model. "X1", "X2", and "X3" make up the input layer. "Y1" and "Y2" form one middle layer. "Z1", "Z2", and "Z3" make up the output layer. A plurality of input values ​​are input to the input layers (X1 to X3). When the plurality of input values ​​are input to the input layers (X1 to X3), the plurality of input values ​​are multiplied by first weights (w11 to w16). The plurality of values ​​obtained as a result are referred to as a plurality of calculated values. The calculated values ​​are input to the intermediate layer (Y1, Y2). When the calculated values ​​are input to the intermediate layer (Y1, Y2), the calculated values ​​are multiplied by second weights (w21 to w26). The resulting values ​​are referred to as output values. The multiple output values ​​are input to the output layers (Z1 to Z3) and output from the output layers (Z1 to Z3). The multiple output values ​​vary depending on the first weights (w11 to w16) and second weights (w21 to W26).

[0066] In supervised learning, the model generation unit 320 inputs the degraded image 113 as multiple input values ​​to the input layer, and adjusts the first weight and the second weight so that the multiple output values ​​output from the output layer approach the reference image 112.

[0067] The procedure of step S130 will be described with reference to FIG. In step S130, the trained model 122 is stored in the memory unit 490. For example, the user inputs the trained model 122 to the learning device 300, and the inference unit 420 stores the input trained model 122 in the memory unit 490. Alternatively, the learning device 300 transmits the trained model 122, and the inference unit 420 receives the trained model 122 and stores it in the memory unit 490.

[0068] In step S131, the image receiving unit 410 receives the target image 131. For example, a user inputs a target image 131 into the inference device 400 , and the image receiving unit 410 receives the input target image 131 .

[0069] In step S132, the inference unit 420 infers the estimated image 132 from the target image 131 using the trained model 122. Specifically, the inference unit 420 receives the target image 131 as input and calculates the trained model 122. As a result, the trained model 122 outputs the estimated image 132.

[0070] In step S133, the image output unit 430 outputs the estimated image 132. For example, the image output unit 430 displays the estimated image 132 on a display.

[0071] ***Effects of the First Embodiment*** According to the first embodiment, it is possible to obtain a high-quality model (trained model 122) for optical satellite images. Then, it is possible to input an actual satellite image (target image 131) to the high-quality model and obtain a high-quality image (estimated image 132) with high accuracy.

[0072] ***Supplement to the first embodiment*** Higher image quality means higher resolution, improved ground resolution, improved signal-to-noise ratio, and improved color shift between bands.

[0073] The type of target image 131 is an optical image. An optical image is an image obtained by exposure to light, which may have a wavelength in any of the bands of visible light, near-infrared light, and far-infrared light.

[0074] The learning device 300 may perform learning using learning data generated by other image quality improvement systems. When multiple high definition systems are used in multiple areas, the learning device 300 may acquire learning data from a high definition system used in the same area as the high definition system 100, or may acquire learning data from a high definition system used in an area different from the high definition system 100. The learning device 300 may add each image quality improvement system to the group of systems from which learning data is obtained at any time, or may remove each image quality improvement system from the group of systems from which learning data is obtained at any time. The learning device 300 may use a trained model generated by another image quality improvement system. For example, the learning device 300 performs learning using the training data 121 to update the trained model. The updated trained model becomes the trained model 122.

[0075] The learning algorithm of the learning device 300 may be deep learning that learns to extract the feature amounts themselves. Alternatively, the learning algorithm may be a machine learning algorithm such as following a genetic program, a functional logic program, or a support vector machine.

[0076] FIG. 14 shows an embodiment of a learning device 300 . When the corresponding image 123 can be obtained in addition to the target image 131 by remote sensing of the target satellite, a pair of the target image 131 and the corresponding image 123 may be added to the training data 121 . The corresponding image 123 is an image that corresponds to the target image 131. The corresponding image 123 is obtained in an observation mode different from the observation mode in which the target image 131 is obtained, and has a higher image quality than the target image 131. This embodiment is possible if the target image 131 is acquired in an observation mode other than the highest resolution observation mode.

[0077] The degradation processing device 200 functions as an input image generation device that generates input images for learning. The learning device 300 functions as a model generation device that generates a trained model. The inference device 400 functions as an utilization device that utilizes the trained model.

[0078] The first embodiment is an example of a preferred embodiment and is not intended to limit the technical scope of the present disclosure. The first embodiment may be implemented in part or in combination with other embodiments. The procedures described using flowcharts and the like may be modified as appropriate.

[0079] Each element of each device in the image quality improvement system 100 may be realized by software, hardware, firmware, or a combination of these. The "unit" of each element of each device in the image quality improvement system 100 may be read as "processing," "step," "circuit," or "circuitry."

[0080] Various aspects of the present disclosure are described below as appendices.

[0081] (Appendix 1) an information receiving unit that receives target satellite information indicating information on remote sensing performed by a target satellite and a reference image having a higher image quality than the target image obtained by the remote sensing; a degradation processing unit that performs degradation processing based on the target satellite information on the reference image to generate a degraded image, which is the reference image with reduced image quality; a model generation unit that generates a trained model by learning a pair of the degraded image and the reference image; an image receiving unit that receives the target image; an inference unit that infers, from the target image using the trained model, an estimated image corresponding to the target image with improved image quality; A high-definition system equipped with:

[0082] (Appendix 2) the reference image is an image obtained by photographing from a flying object, The flying object is one of a helicopter, an unmanned aerial vehicle, and an aircraft, or an artificial satellite that can obtain images with higher image quality than the target image. 10. The image quality improvement system according to claim 1.

[0083] (Appendix 3) the degradation processing is one of a color shift processing, a resolution processing, a ground sampling distance processing, a noise processing, and a compression processing, or a combination of the color shift processing, the resolution processing, the ground sampling distance processing, the noise processing, and the compression processing, the target satellite information indicates an arrangement of a plurality of optical sensors mounted on the target satellite, and the color shift processing reproduces, in the reference image, a color shift based on the arrangement of the plurality of optical sensors; the target satellite information indicates a point spread function for the target image, and the resolution processing applies the point spread function to the reference image; the target satellite information indicates a ground sampling distance of the target image, and the ground sampling distance processing simulates the ground sampling distance with respect to the reference image; The target satellite information indicates noise characteristics of the remote sensing, and the noise processing adds noise based on the noise characteristics to the reference image; The target satellite information indicates a data compression technique for the target image, and the compression process applies the data compression technique to the reference image. 10. The image quality improvement system according to claim 1 or 2.

[0084] (Appendix 4) the model generation unit generates the trained model by training the pair of the degraded image and the reference image and the pair of the target image and the corresponding image; The corresponding image is obtained in an observation mode different from the observation mode in which the target image is obtained in the remote sensing, and has higher image quality than the target image. 10. A high-definition image system according to claim 1, wherein the image is imaged by a plurality of pixels.

[0085] (Appendix 5) the target satellite information indicates the arrangement of a plurality of optical sensors mounted on the target satellite; The degradation process downsamples the reference image using a different offset for each band based on the arrangement of the plurality of optical sensors. 10. The image quality improvement system according to claim 1, 2 or 4.

[0086] The target image is an image obtained by exposure to light having a wavelength in the visible light range. 6. A high-definition image system according to any one of appendices 1 to 5.

[0087] (Appendix 6) The target image is an image obtained by exposure to light rays having wavelengths in the infrared light band. 6. A high-definition image system according to any one of appendices 1 to 5. [Explanation of symbols]

[0088] 100 Image quality improvement system, 111 Target satellite information, 112 Reference image, 113 Degraded image, 121 Learning data, 122 Trained model, 123 Corresponding image, 131 Target image, 132 Estimated image, 200 Degradation processing device, 201 Processor, 202 Memory, 203 Auxiliary storage device, 204 Communication device, 205 Input / output interface, 210 Information reception unit, 220 Degradation processing unit, 230 Image output unit, 290 Storage unit, 300 Learning device, 301 Processor, 302 Memory, 303 Auxiliary storage device, 304 Communication device, 305 Input / output interface, 310 Data reception unit, 320 Model generation unit, 330 Model output unit, 390 Storage unit, 400 Inference device, 401 Processor, 402 Memory, 403 Auxiliary storage device, 404 Communication device, 405 Input / output interface, 410 image reception unit, 420 inference unit, 430 image output unit, 490 memory unit.

Claims

1. A learning device that generates a trained model for inferring an estimated image corresponding to a target image with improved image quality from a target image obtained by remote sensing of a target satellite, a data receiving unit that receives, as learning data, a pair of a reference image having a higher image quality than the target image and a degraded image that is the reference image having a lower image quality; a model generation unit that generates the trained model by training the training data; Equipped with The degraded image is generated by performing color shift processing or compression processing on the reference image. Learning device.

2. A learning method for generating a trained model for inferring an estimated image corresponding to a target image with improved image quality from a target image obtained by remote sensing of a target satellite, A step of receiving, as learning data, a pair of a reference image having a higher image quality than the target image and a degraded image which is the reference image with a lower image quality; generating the trained model by training the training data; The degraded image is generated by performing color shift processing or compression processing on the reference image. How to learn.

3. A learning program for generating a trained model for inferring an estimated image corresponding to a target image with improved image quality from a target image obtained by remote sensing of a target satellite, The learning program a data receiving process for receiving, as learning data, a pair of a reference image having a higher image quality than the target image and a degraded image, which is the reference image having a lower image quality; a model generation process for generating the trained model by learning the training data; on the computer, The degraded image is generated by performing color shift processing or compression processing on the reference image. Learning program.

4. an image receiving unit that receives a target image obtained by remote sensing of the target satellite; an inference unit that infers, from the target image using a trained model, an estimated image corresponding to the target image with improved image quality; Equipped with The trained model is generated by training a pair of a degraded image and a reference image; the reference image is an image with a higher image quality than the target image, The degraded image is generated by performing color shift processing or compression processing on the reference image. Reasoning device.

5. receiving a target image obtained by remote sensing of a target satellite; inferring, from the target image using the trained model, an estimated image corresponding to the enhanced target image; Equipped with The trained model is generated by training a pair of a degraded image and a reference image; the reference image is an image with a higher image quality than the target image, The degraded image is generated by performing color shift processing or compression processing on the reference image. Reasoning method.

6. an image receiving process for receiving a target image obtained by remote sensing of the target satellite; an inference process for inferring, from the target image using a trained model, an estimated image corresponding to the target image with improved image quality; An inference program that causes a computer to execute the following: The trained model is generated by training a pair of a degraded image and a reference image; the reference image is an image with a higher image quality than the target image, The degraded image is generated by performing color shift processing or compression processing on the reference image. Inference program.

7. an information receiving unit that receives target satellite information indicating information on remote sensing performed by a target satellite and a reference image having a higher image quality than the target image obtained by the remote sensing; a degradation processing unit that performs color shift processing or compression processing on the reference image to generate a degraded image, which is the reference image with reduced image quality; a model generation unit that generates a trained model by learning a pair of the degraded image and the reference image; an image receiving unit that receives the target image; an inference unit that infers, from the target image using the trained model, an estimated image corresponding to the target image with improved image quality; A high-definition system equipped with:

8. the degraded image is generated by performing resolution processing; The resolution processing is based on a point spread function for the target image indicated by the target satellite information, and if there is a difference in the point spread function between bands, aliasing is reproduced for the reference image. The image quality improvement system according to claim 7.

9. an information receiving unit that receives target satellite information indicating information on remote sensing performed by a target satellite and a reference image having a higher image quality than the target image obtained by the remote sensing; a degradation processing unit that performs color shift processing or compression processing on the reference image based on the target satellite information to generate a degraded image, which is the reference image with reduced image quality; a model generation unit that generates a trained model by learning a pair of the degraded image and the reference image; an image receiving unit that receives the target image; an inference unit that infers, from the target image using the trained model, an estimated image corresponding to the target image with improved image quality; A high-definition system equipped with:

Images