Image processing system, image processing method, and computer program

Abstract

This invention provides an image processing system, an image processing method, and a computer program capable of acquiring an output image by super-resolutionizing an input image. [Solution] The image processing system includes an acquisition means for acquiring an input image in which a feature is captured at a first resolution, and an output means for outputting information about an output image that includes the feature and has a second resolution higher than the first resolution, which is output from the learning model when the input image is input to the learning model. The learning model is trained to output a first learning output image that includes the predetermined region and has a second resolution when a first learning input image in which a predetermined region is captured at a first resolution is input, and to output a second learning output image that includes the predetermined region and has a second resolution when a second learning input image is input, which is obtained by converting an image in which a predetermined region is captured at a second resolution so that the resolution becomes the first resolution.

Description

Technical Field

[0001] The present invention relates to an image processing system, an image processing method, and a computer program.

Background Art

[0002] Conventionally, images in which ground features are imaged, such as aerial photographs or satellite images, have been used to perform processes such as interpretation of building changes, creation of maps, and creation of city models. In order to perform processes such as interpretation of building changes, creation of maps, and creation of city models with high accuracy, it is preferable to use images with higher resolution.

[0003] Patent Document 1 discloses a learning device that generates a learning model for inferring an estimated image corresponding to a target image with improved image quality from a target image obtained by remote sensing of a target satellite. This learning device accepts, as learning data, a set of a reference image having higher image quality than the target image and a degraded image that is a reference image with reduced image quality.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In recent years, super-resolution models have been developed that generate high-resolution images from low-resolution images. However, if a super-resolution model is trained using a high-resolution image and a separately captured low-resolution image, the position, shape, and / or color of features in each image may differ slightly, potentially resulting in an inappropriate model. On the other hand, if a super-resolution model is trained using a high-resolution image and a scaled-down version of that image, the appearance of objects in each image may differ, potentially resulting in an inappropriate model. Therefore, whether a model trained using only high-resolution and low-resolution image pairs, or only high-resolution and scaled-down image pairs, it was sometimes not possible to obtain images with a resolution high enough to sufficiently improve the accuracy of feature recognition and detection.

[0006] The present invention aims to provide an image processing system, an image processing method, and a computer program capable of acquiring an output image by super-resolutionizing an input image. [Means for solving the problem]

[0007] The image processing system according to the present invention includes an acquisition means for acquiring an input image in which a feature is captured at a first resolution, and an output means for outputting information about an output image that includes a feature and has a second resolution higher than the first resolution, which is output from the learning model when the input image is input to the learning model. The learning model is trained to output a first learning output image that includes a predetermined region and has a second resolution when a first learning input image in which a predetermined region is captured at a first resolution is input, and to output a second learning output image that includes a predetermined region and has a second resolution when a second learning input image is input, which is obtained by converting an image in which a predetermined region is captured at a second resolution so that the resolution becomes the first resolution.

[0008] Furthermore, in the image processing system according to the present invention, it is preferable that the learning model is trained to minimize the error between each of the first training output image and the second training output image and the same ground truth image in which a predetermined region is captured at a second resolution.

[0009] Furthermore, in the image processing system according to the present invention, it is preferable that the first training input image is color-coded before being input to the training model.

[0010] Furthermore, in the image processing system according to the present invention, it is preferable that the first learning input image is transformed in terms of color tone by performing standardization or by transforming the grayscale value of each pixel through histogram matching.

[0011] Furthermore, in the image processing system according to the present invention, it is preferable that pixels included in the first training output image whose loss relative to the corresponding pixel in the ground truth image satisfies predetermined conditions are excluded from the training target.

[0012] Furthermore, in the image processing system according to the present invention, it is preferable that pixels that satisfy the predetermined conditions are pixels whose loss is greater than or equal to a predetermined value, or pixels that are included in a predetermined proportion in order of decreasing loss.

[0013] Furthermore, in the image processing system according to the present invention, it is preferable that the second learning input image is generated from an image in which a predetermined area is captured at a second resolution, using a model that reproduces the degradation process.

[0014] Furthermore, the image processing method according to the present invention includes acquiring an input image in which a feature is captured at a first resolution, and outputting information about an output image that includes the feature and has a second resolution higher than the first resolution, which is output from the learning model when the input image is input to the learning model. The learning model is trained to output a first learning output image that includes a predetermined region and has a second resolution when a first learning input image in which a predetermined region is captured at a first resolution is input, and to output a second learning output image that includes a predetermined region and has a second resolution when a second learning input image is input, which is obtained by converting an image in which a predetermined region is captured at a second resolution so that the resolution becomes the first resolution.

[0015] Furthermore, the computer program according to the present invention causes the computer to acquire an input image in which a feature is captured at a first resolution, and to output information about an output image that includes the feature and has a second resolution higher than the first resolution, which is output from the learning model when the input image is input to the learning model. The learning model is trained to output a first learning output image that includes a predetermined region and has a second resolution when a first learning input image in which a predetermined region is captured at a first resolution is input, and to output a second learning output image that includes a predetermined region and has a second resolution when a second learning input image is input, which is an image in which a predetermined region is captured at a second resolution and has been converted so that the resolution becomes the first resolution. [Effects of the Invention]

[0016] The image processing system, image processing method, and computer program according to the present invention are capable of obtaining an output image by super-resolutionizing an input image. [Brief explanation of the drawing]

[0017] [Figure 1] This is a diagram illustrating an example of the configuration of image processing system 1. [Figure 2] This is a flowchart illustrating an example of the learning process flow. [Figure 3](A) is a schematic diagram for explaining ground object changes, and (B) is a schematic diagram for explaining imaging angle changes. [Figure 4] It is a flowchart showing an example of the flow of output processing.

Embodiments for Carrying out the Invention

[0018] Hereinafter, various embodiments of the present invention will be described with reference to the drawings. It should be noted that the technical scope of the present invention is not limited to these embodiments, but extends to the invention described in the claims and its equivalents.

[0019] FIG. 1 is a configuration diagram of an example of an image processing system 1 according to the present invention.

[0020] The image processing system 1 includes an information processing device 100 and a server device 200. The information processing device 100 and the server device 200 are examples of image processing devices and are communicatively connected to each other via a network N. The network N is an intranet or the Internet or the like.

[0021] The information processing device 100 is a personal computer, a notebook personal computer, a server, or the like. The information processing device 100 includes an operation device 101, a display device 102, a communication device 103, a storage device 110, a processing circuit 120, and the like.

[0022] The operation device 101 includes an input device such as a keyboard and a mouse, and an interface circuit that acquires signals from the input device. It receives operations by the user and outputs signals corresponding to the user's input to the processing circuit 120.

[0023] The display device 102 is an example of output means. The display device 102 includes a display composed of liquid crystal, organic EL, or the like, and an interface circuit that outputs image data to the display. It displays the image data on the display according to an instruction from the processing circuit 120.

[0024] The communication device 103 is an example of an output means. The communication device 103 is equipped with a wired or wireless communication interface circuit and connects the information processing device 100 to a communication network. The communication device 103 performs wired communication according to a communication protocol such as TCP / IP (Transmission Control Protocol / Internet Protocol). Alternatively, the communication device 103 may perform wireless communication according to the IEEE (Institute of Electrical and Electronics Engineers) 802.11 standard. The communication device 103 transmits information supplied from the processing circuit 120 to an external device. The communication device 103 also supplies information received from the external device to the processing circuit 120.

[0025] The storage device 110 includes, for example, semiconductor memory such as RAM (Random Access Memory) or ROM (Read Only Memory), a fixed disk device such as a hard disk, or a portable storage device such as an optical disc. The storage device 110 stores computer programs, data, etc., used for processing by the processing circuit 120. The computer program is installed in the storage device 110 from a server (not shown) via a communication device 103. Alternatively, the computer program may be installed in the storage device 110 from a computer-readable portable recording medium using a known setup program, etc. The portable recording medium is, for example, a CD-ROM or DVD-ROM. The computer program may also be distributed from a server or the like and installed in the storage device 110.

[0026] Furthermore, the storage device 110 stores the learning model 111. When an image is input, the learning model 111 outputs an image that corresponds to that image and has a higher resolution than the resolution of that image.

[0027] The processing circuit 120 is, for example, a CPU (Central Processing Unit). The processing circuit 120 may also be an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), etc. The processing circuit 120 is connected to the operating device 101, display device 102, communication device 103, and storage device 110, etc., and controls each of these parts. The processing circuit 120 reads the program stored in the storage device 110 and operates according to the read program, thereby functioning as a learning means 121, acquisition means 122, and output control means 123. The processing circuit 120 generates a learning model 111 and uses the learning model 111 to generate an output image from an input image in which features are captured, which includes features and has a resolution higher than the resolution of the input image.

[0028] Figure 2 is a flowchart showing an example of the learning process flow performed by the information processing device 100.

[0029] The following describes an example of the operation of the learning process of the information processing device 100, referring to the flowchart shown in Figure 2. The operation flow described below is executed primarily by the processing circuit 120 in cooperation with each element of the information processing device 100, based on a program pre-stored in the memory device 110.

[0030] First, the learning means 121 acquires a training dataset (step S101). The training dataset includes multiple combinations of a first training input image, in which a predetermined region is captured at a first resolution, and a ground truth image, in which the same predetermined region is captured at a second resolution. The first training input image and the ground truth image to be combined are images of the same region (minor positional errors are permitted). On the other hand, the ground truth images may include images of the same region, but are basically images of different regions. The learning means 121 acquires the training dataset by receiving it from the server device 200 via the communication device 103. Alternatively, the training dataset may be pre-stored in the storage device 110, and the learning means 121 may acquire the training dataset by reading it from the storage device 110.

[0031] The designated area is an aerial photograph, satellite image (or orthophotograph based on aerial photographs and satellite images), etc., of the Earth's surface, which is the target area for processing such as interpreting building movements, creating maps, and creating urban models, and includes features captured from above. Features include buildings, trees, rocks, etc. The first training input image and the ground truth image are images of different types (aerial photographs, satellite images, etc.), images captured by different imaging devices (imaging sensors), images captured at different timings, images captured from different positions and at different angles, or images captured under two or more of these conditions. For example, a satellite image can be used as the first training input image, and an aerial photograph with higher resolution can be used as the ground truth image. Therefore, although the first training input image and the ground truth image contain the same area, the position, shape, and / or color of features within each image are likely to be slightly different.

[0032] Next, the learning means 121 generates a second learning input image by transforming each ground truth image included in the training dataset so that its resolution becomes the first resolution (step S102). The learning means 121 generates the second learning input image from each ground truth image using a degradation process reproduction learning model that reproduces the degradation process. The degradation process reproduction learning model is trained to output an image that is naturally degraded from the input image when an image is input. The degradation process reproduction learning model outputs an image in which the resolution is reduced from the input image, and noise, out-of-focus (image blur) and / or image shake are added, or the color tone is changed. For example, BSRGAN (Blind Super Resolution Generative Adversarial Network) can be used as a degradation process reproduction learning model. For information on BSRGAN, see, for example, "Designing a Practical Degradation Model for Deep Blind Image Super-Resolution" by Kai Zhang, Jingyun Liang, Luc Van Gool, and Radu Timofte (https: / / arxiv.org / abs / 2103.14006).

[0033] The learning means 121 may generate a second learning input image by downsampling and / or interpolating the pixels of each ground truth image. In this case, the learning means 121 may add noise to each ground truth image by randomly changing the pixel values ​​of specific pixels, add blur or image blur by applying a smoothing filter to specific areas, or change the color tone by gamma correction or the like.

[0034] The image processing system 1 can generate diverse patterns of training data by using degraded versions of each ground truth image as second training input images, thereby improving the training accuracy of the training model 111. The training means 121 may also generate second training input images for each ground truth image without adding noise, blurring (image blur), or image shake, and without changing the color tone. Furthermore, since the output of the training model is magnified at a constant rate, if the resolution of the second training input image differs from the resolution of the first training input image, the loss with high-resolution images may not be calculated correctly. The image processing system 1 can correctly calculate the loss with high-resolution images by making the resolution of the second training input image the same as the resolution of the first training input image.

[0035] Next, the learning means 121 transforms the hue of each first training input image included in the training dataset (step S103). The learning means 121 performs standardization on the first training input image, for example, to transform the hue of the first training input image so that the grayscale values ​​(R, G, and B values) of each pixel included in the first training input image follow a standard normal distribution. In this case, the learning means 121 calculates the mean and variance of the grayscale values ​​of all pixels included in the first training input image for each color (red, green, and blue), and transforms the grayscale value of each pixel according to the following formula (1) based on the calculated mean and variance. I out =(I in -I mean ) / σ (1) Here, I out This is the grayscale value after conversion, I in This is the grayscale value before conversion, I mean σ is the mean value of the grayscale values, and σ is the variance of the grayscale values. As a result, the mean value of the grayscale values ​​of all pixels in the transformed first training input image becomes 0, the variance becomes 1, and the grayscale values ​​of the transformed first training input image follow a standard normal distribution.

[0036] Thus, before inputting the first training input image to the training model 111, the training means 121 transforms the color tone of the first training input image so that the grayscale values ​​of each pixel in the first training input image follow a standard normal distribution. This allows the image processing system 1 to suppress the inclusion of outlier pixels in the first training input image. Furthermore, the image processing system 1 can suppress the influence of differences in color tone between the first training input image and the ground truth image on learning (overfitting of color tone transformation). Therefore, the image processing system 1 can improve the training accuracy of the training model 111.

[0037] The learning means 121 may also convert the color tone of the first learning input image by histogram matching. In this case, the information processing device 100 generates a histogram of the grayscale values ​​of each pixel in a predetermined sample image for each color (red, green, and blue) and stores it in the storage device 110. The sample image is set to a satellite image or aerial photograph, etc., that does not contain outlier pixels and has an average color tone. The learning means 121 converts the grayscale values ​​of each pixel in the first learning input image for each color (red, green, and blue) so that the histogram of the grayscale values ​​of each pixel in the first learning input image matches the histogram of the grayscale values ​​of each pixel in the sample image.

[0038] Thus, before inputting the first training input image to the training model 111, the training means 121 converts the color tone of the first training input image by converting the grayscale value of each pixel contained in the first training input image through histogram matching. This allows the image processing system 1 to suppress the inclusion of outlier pixels in the first training input image. Furthermore, the image processing system 1 can suppress the influence of differences in color tone between the first training input image and the ground truth image on learning (overfitting of color tone conversion). Therefore, the image processing system 1 can improve the training accuracy of the training model 111.

[0039] Note that the process in step S103 may be omitted.

[0040] Next, the learning means 121 generates a learning model 111 using the first learning input image whose color tone was converted in step S103, the second learning input image generated in step S102, and the ground truth image acquired in step S101 (step S104). The learning model 111 is a model that uses, for example, a neural network or SwinIR (Image Restoration Using Swin Transformer). The learning model 111 may also be other models that use RAISR (Rapid and Accurate Image Super-Resolution), etc.

[0041] The learning means 121 trains the learning model 111 to output a first learning output image that includes a predetermined region captured in the first learning input image and has a second resolution when a first learning input image is input. The learning means 121 trains the learning model 111 so that the first learning output image approximates a ground truth image that includes the predetermined region contained in the first learning input image. For example, the learning means 121 trains the learning model 111 so that the error between the first learning output image and a ground truth image that includes the predetermined region contained in the first learning input image, i.e., the loss function, is minimized. The loss function is, for example, the sum of squared errors of each grayscale value (R value, G value, and B value) of each corresponding pixel, or the sum of the absolute values ​​of the differences. Based on the error between the first learning output image and the ground truth image, the learning means 121 changes the values ​​of parameters such as weights in the learning model 111 by backpropagation or the like.

[0042] Furthermore, the learning means 121 trains the learning model 111 to output a second learning output image that includes a predetermined region captured in the ground truth image that is the source of the second learning input image and has a second resolution when a second learning input image is input. The learning means 121 trains the learning model 111 so that the second learning output image approximates the ground truth image that is the source of the second learning input image. For example, the learning means 121 trains the learning model 111 so that the error between the second learning output image and the ground truth image that is the source of the second learning input image is minimized. Based on the error between the second learning output image and the ground truth image, the learning means 121 changes the values ​​of parameters such as weights in the learning model 111 using backpropagation or the like.

[0043] The learning means 121 inputs the first training input image into the learning model 111 and obtains the first training output image from the learning model 111. When it inputs the second training input image into the learning model 111 and obtains the second training output image from the learning model 111, it performs parameter optimization in the learning model 111. This allows the image processing system 1 to optimize the parameters of the learning model 111 by considering both the relationship between the first training input image and the ground truth image, and the relationship between the second training input image and the ground truth image.

[0044] Thus, the learning model 111 is trained to output a first learning output image that includes a predetermined region and has a second resolution when a first learning input image in which a predetermined region is captured at a first resolution is input, and to output a second learning output image that includes the predetermined region and has a second resolution when a second learning input image is input, which is obtained by converting a ground truth image in which the predetermined region is captured at a second resolution so that the resolution becomes the first resolution. As described above, the types of the first learning input image and the ground truth image, the imaging device, the imaging timing, the position of the imaging device, or the imaging angle are all different from each other. Therefore, although the first learning input image and the ground truth image contain the same region, the position, shape, and / or color of features within each image are likely to be slightly different. Consequently, the learning accuracy of a learning model trained using only the first learning input image and the ground truth image may be low.

[0045] On the other hand, the appearance of objects in the second training input image, which is generated by reducing the resolution of the ground truth image captured at the second resolution to the first resolution, differs from that of an image captured at the first resolution from the beginning. Therefore, the training accuracy of a training model trained using only the second training input image and the ground truth image may be low. However, the type of the second training input image and the ground truth image, the imaging device, the imaging timing, the position of the imaging device, or the imaging angle are all the same. Therefore, the position, shape, and / or color of the features are all the same in both the second training input image and the ground truth image. Consequently, the image processing system 1 can improve the training accuracy of the training model 111 by training the training model 111 using both the first training input image and the second training input image.

[0046] Furthermore, the image processing system 1 trains the learning model 111 so that the error between each of the first and second training output images and a single ground truth image captured at the second resolution in which a predetermined region included in the first and second training output images is minimized. This allows the image processing system 1 to more efficiently improve the learning accuracy of the learning model 111. Note that the image processing system 1 may use mutually different images as the ground truth image for the first training input image and the ground truth image for the second training input image.

[0047] The learning means 121 may exclude pixels from the learning model 111 that satisfy a predetermined condition regarding the loss of the corresponding pixel in the corresponding ground truth image, from among the pixels included in the first training output image. The predetermined condition is that the loss is sufficiently large, that is, the difference in grayscale value between the pixel and the corresponding pixel in the corresponding ground truth image is sufficiently large. For example, a pixel that satisfies the predetermined condition is a pixel whose loss of the corresponding pixel in the ground truth image is greater than or equal to a predetermined value. The predetermined value is set in advance to a value corresponding to the difference in grayscale value that a person can visually distinguish on an image (e.g., 20). The predetermined value may be dynamically set to losses of a predetermined rank (e.g., top 10%) in descending order of loss relative to the corresponding pixel in the corresponding ground truth image in a predetermined number (e.g., 10) of first training output images that were learned immediately beforehand.

[0048] Pixels that satisfy the predetermined conditions may also be pixels included in the first training output image that are ranked in descending order of loss relative to the corresponding pixels in the corresponding ground truth image. The predetermined proportion is set to, for example, the proportion of pixels that can be considered outliers (e.g., the top 1%).

[0049] Figure 3(A) is a schematic diagram illustrating changes in geographic features. In Figure 3(A), image P11 shows a first training input image in which a predetermined region was captured at a predetermined timing, and image P12 shows a ground truth image in which the same predetermined region was captured at a subsequent timing. In the example shown in Figure 3(A), a tree T exists in region S, which was an empty lot when image P11 was captured, when image P12 was captured. Similarly, between the time the first training input image was captured and the time the ground truth image was captured, construction, demolition, expansion, or renovation of a building may occur within the region included in each image. Thus, when the timing of capturing the first training input image and the ground truth image differs, the geographic features included in each image may change. The image processing system 1 can exclude pixels from the learning target that have a large loss relative to the corresponding pixel in the corresponding ground truth image, thereby excluding pixels in the first training output image whose captured geographic features have changed. Therefore, the image processing system 1 can improve the learning accuracy of the learning model 111.

[0050] Figure 3(B) is a schematic diagram illustrating the change in imaging angle. In Figure 3(B), image P21 shows a first training input image in which a predetermined region is captured at a predetermined angle, and image P22 shows a ground truth image in which the same predetermined region is captured at an angle different from the predetermined angle. In the example shown in Figure 3(B), image P21 includes building B captured from the upper left direction, and image P22 includes building B captured from the lower right direction. As shown in Figure 3(B), when the imaging angles of the first training input image and the ground truth image are different, a positional shift due to so-called tilting occurs, and one image may include a side of the building that is not captured in the other image. The image processing system 1 can exclude pixels corresponding to parts included only in one image from the learning target by excluding pixels in the first training output image that have a large loss relative to the corresponding pixels in the corresponding ground truth image. Therefore, the image processing system 1 can improve the learning accuracy of the learning model 111.

[0051] Next, the learning means 121 stores the generated learning model 111 in the storage device 110 (step S105) and terminates the learning process.

[0052] Figure 4 is a flowchart showing an example of the output processing flow performed by the information processing device 100.

[0053] The following describes an example of the output processing operation of the information processing device 100, referring to the flowchart shown in Figure 4. The operation flow described below is primarily executed by the processing circuit 120 in cooperation with each element of the information processing device 100, based on a program pre-stored in the memory device 110.

[0054] First, the acquisition means 122 acquires an input image in which features are captured at a first resolution (step S201). The acquisition means 122 acquires the input image by receiving it from the server device 200 via the communication device 103. The input image is an aerial photograph, satellite image (or orthophoto based on aerial photographs and satellite images), etc., of the ground surface that is the target area for processing such as interpreting building movements, creating maps and creating urban models, and includes features captured from above.

[0055] Next, the output control means 123 inputs the input image acquired by the acquisition means 122 to the learning model 111 and acquires the output image output from the learning model 111 when the input image was input to the learning model 111 (step S202). The output image is an image that includes features contained in the input image and has a second resolution.

[0056] Next, the output control means 123 outputs information about the acquired output image by transmitting it to an external device via the communication device 103 or by displaying it on the display device 102 (step S203), and terminates the output process. The information about the output image is, for example, the output image itself. The information about the output image may also be an image on which a predetermined correction process has been performed. The information about the output image may also be information indicating the location of features detected from the output image, objects installed on those features (for example, solar panels installed on the roof of a house), or the location of those features or objects. In this case, the output control means 123 uses known image processing techniques to detect features included in the output image, objects installed on those features, or the location of those features or objects from the output image.

[0057] As explained above, the image processing system 1 trains the learning model 111 using a first learning input image in which a predetermined region is captured at a first resolution, and a second learning input image obtained by converting a ground truth image in which a predetermined region is captured at a second resolution so that its resolution becomes the first resolution. Then, the image processing system 1 uses the learning model 111 to generate an output image having the second resolution from the input image captured at the first resolution. This makes it possible for the image processing system 1 to obtain an output image that is a super-resolution version of the input image.

[0058] Furthermore, the image processing system 1 can generate a learning model 111 using mutually different types of images as the first training input image and the ground truth image. Satellite images are captured more frequently and have lower acquisition costs compared to aerial photographs. For example, the image processing system 1 can efficiently and inexpensively generate a learning model 111 by using a combination of satellite images and aerial photographs as the first training input image and the ground truth image.

[0059] Furthermore, the image processing system 1 can detect features and / or objects attached to those features with greater accuracy by obtaining an output image with a higher resolution than the input image from the input image.

[0060] For typical cameras and subjects, the difference between low-resolution and reduced-resolution images is not significant. However, with aerial and satellite imagery, the difference between low-resolution and reduced-resolution images is large compared to typical cameras, due to the influence of external disturbances such as regional characteristics and atmospheric conditions, in addition to the shooting equipment and timing. When a super-resolution model is trained using only pairs of reduced-resolution and high-resolution images, if the difference in image distribution between the training image pairs is large, the loss will be large even after repeated training, and an appropriate trained model will not be generated. Therefore, in training a super-resolution model, the image distributions between the training image pairs must be similar. Also, when a super-resolution model is trained using only pairs of low-resolution and high-resolution images, the image distribution between the training image pairs differs due to external disturbances (imaging timing, shooting angle, etc.), so the loss is not sufficiently reduced, and an appropriate trained model is not generated. Image processing system 1 can solve the above problems by generating a trained model using both pairs of low-resolution and high-resolution images, and pairs of reduced-resolution and high-resolution images. Image processing system 1 improves interpretation accuracy by using low-resolution images that are actually to be enlarged during training, while suppressing the increase in loss by using reduced images and high-resolution images in areas where the image distributions of low-resolution and high-resolution images are far apart. Therefore, image processing system 1 can appropriately generate a training model.

[0061] While preferred embodiments have been described above, the embodiments are not limited to these. For example, the ground truth image of the first training input image used to train the learning model 111 and the ground truth image of the second training input image corresponding to the first training input image do not have to be the same image, but may be different images from each other.

[0062] Alternatively, instead of the information processing device 100, a server device 200 may have a learning means 121 and execute the learning process shown in Figure 2 to generate the learning model 111.

[0063] Alternatively, instead of the information processing device 100, the server device 200 may store the learning model 111. In that case, in step S202 of Figure 4, the output control means 123 transmits the input image to the server device 200 via the communication device 103. The server device 200 inputs the input image received from the information processing device 100 into the learning model 111 and transmits the output image output from the learning model 111 to the information processing device 100. The output control means 123 acquires the output image by receiving it from the server device 200 via the communication device 103.

[0064] Those skilled in the art will understand that various changes, substitutions, and modifications can be made without departing from the spirit and scope of the present invention. For example, the embodiments and modifications described above may be combined as appropriate within the scope of the invention. [Explanation of Symbols]

[0065] 1 Image processing system, 100 Information processing device, 102 Display device, 103 Communication device, 111 Learning model, 121 Learning means, 122 Acquisition means, 123 Output control means

Claims

1. An acquisition means for acquiring an input image in which features are captured at a first resolution, The system includes an output means that outputs information about an output image, which includes the features and has a second resolution higher than the first resolution, output from the learning model when the input image is input to the learning model. The learning model is trained to output a first learning output image that includes the predetermined region and has the second resolution when a first learning input image in which a predetermined region is captured at the first resolution is input, and to output a second learning output image that includes the predetermined region and has the second resolution when a second learning input image is input, which is obtained by converting an image in which the predetermined region is captured at the second resolution so that the resolution becomes the first resolution. An image processing system characterized by the following:

2. The image processing system according to claim 1, wherein the learning model is trained to minimize the error between each of the first training output image and the second training output image and the same ground truth image in which the predetermined region is captured at the second resolution.

3. The image processing system according to claim 1, wherein the first training input image is color-coded before being input to the training model.

4. The image processing system according to claim 3, wherein the first training input image is transformed in terms of color tone by performing standardization or by transforming the grayscale value of each pixel by histogram matching.

5. The image processing system according to claim 1, wherein pixels included in the first training output image whose loss to the corresponding pixel in the ground truth image satisfies a predetermined condition are excluded from the training target.

6. The image processing system according to claim 5, wherein the pixels that satisfy the predetermined conditions are pixels whose loss is greater than or equal to a predetermined value, or pixels that are included in a predetermined proportion in descending order of loss.

7. The image processing system according to claim 1, wherein the second training input image is generated from an image in which the predetermined region is captured at the second resolution, using a model that reproduces the degradation process.

8. An input image in which features are captured at the first resolution is obtained. The system includes outputting information about an output image that includes the features and has a second resolution higher than the first resolution, which is output from the learning model when the input image is input to the learning model. The learning model is trained to output a first learning output image that includes the predetermined region and has the second resolution when a first learning input image in which a predetermined region is captured at the first resolution is input, and to output a second learning output image that includes the predetermined region and has the second resolution when a second learning input image is input, which is obtained by converting an image in which the predetermined region is captured at the second resolution so that the resolution becomes the first resolution. An image processing method characterized by the following:

9. An input image in which features are captured at the first resolution is obtained. The computer is instructed to output information about an output image that includes the features and has a second resolution higher than the first resolution, which is output from the learning model when the input image is input to the learning model. The learning model is trained to output a first learning output image that includes the predetermined region and has the second resolution when a first learning input image in which a predetermined region is captured at the first resolution is input, and to output a second learning output image that includes the predetermined region and has the second resolution when a second learning input image is input, which is obtained by converting an image in which the predetermined region is captured at the second resolution so that the resolution becomes the first resolution. A computer program characterized by the following features.

Images