Image processing method, program, and image processing apparatus

Abstract

To provide a high-quality image in increasing the resolution of an image by using a machine learning model.SOLUTION: An image processing method includes: a step S403 of creating, on the basis of a first image, a first intermediate image with a resolution higher than that of the first image by using a first creator; a step S403 of creating, on the basis of the first image, a second intermediate image with a resolution higher than that of the first image by using a second creator; and a step S404 of creating an estimated image with a resolution higher than that of the first image on the basis of the first intermediate image and the second intermediate image. The first creator is obtained through learning not using a discriminator, and the second creator is obtained through learning using a discriminator.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for increasing the resolution of an image using a machine learning model. [Background technology]

[0002] Patent Document 1 discloses a method for increasing the resolution of an image using a machine learning model called a Generative Adversarial Network (GAN). This method is called a Super Resolution GAN (SRGAN). SRGAN performs learning using a generator that generates a high-resolution image and a classifier that identifies whether an input image is an image generated by the generator or an actual high-resolution image. Here, an actual high-resolution image means a high-resolution image that is not generated by a generator.

[0003] The generator trains weights so that the generated high-resolution image is indistinguishable from the actual high-resolution image by the classifier. This allows for the generation of more natural-looking high-resolution images with high-resolution textures. However, at the same time, there is a problem in that false structures may appear that are subjectively unnatural.

[0004] In response to this, Non-Patent Document 1 discloses a method for controlling the appearance of false structures and perceived resolution. In Non-Patent Document 1, a weighted average is taken between the weights of a generator trained to increase resolution without using a classifier (few false structures but low perceived resolution) and the weights of a generator trained using a classifier (equivalent to SRGAN, high perceived resolution but possible presence of false structures). A high-resolution image is generated by a generator using this weighted average. This method is called network interpolation. By changing the weights of the weighted average, it is possible to control the balance between the appearance of false structures and perceived resolution. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] US2018 / 0075581 [Non-patent literature]

[0006] [Non-Patent Document 1] Xintao Wang,et al.,Deep Network Interpolation for Continuous Imagery Effect Transition,Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR),2019,pp.1692-1701 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the inventors' investigations revealed that the generator that performs network interpolation as described in Non-Patent Document 1 may cause degradation in image quality in the generated high-resolution images, such as multiplexed edges that make the subject appear double, and color changes. [Means for solving the problem]

[0008] An image processing method according to one embodiment of the present invention includes the steps of: using a first generator to generate a first intermediate image based on a first image, the first intermediate image having a higher resolution than the first image; using a second generator to generate a second intermediate image based on the first image, the second intermediate image having a higher resolution than the first image; and generating an estimated image based on the first intermediate image and the second intermediate image, the estimated image having a higher resolution than the first image, wherein the first generator is obtained by learning without using a classifier, and the second generator is obtained by learning with a classifier.

[0009] Furthermore, an image processing method according to one embodiment of the present invention is characterized by comprising the steps of: converting a first image into a first feature map by inputting the first image into a generator; generating a first intermediate image having a higher resolution than the first image and a second intermediate image having a higher resolution than the first image based on the first feature map; and generating an estimated image having a higher resolution than the first image by adjusting the perceived resolution based on the first intermediate image and the second intermediate image. [Effects of the Invention]

[0010] According to the present invention, a high-quality image can be provided by increasing the resolution of an image using a machine learning model. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 2 is a diagram showing a flow of increasing the resolution in the first embodiment. [Figure 2] 1 is a block diagram of an image processing system according to a first embodiment. [Figure 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4] 1 is a flowchart of learning of a machine learning model in Example 1. [Figure 5] FIG. 2 is a schematic diagram of a subnet in the first and second embodiments. [Figure 6] 10 is a flowchart of generating an estimated high-resolution image in the first embodiment. [Figure 7] FIG. 10 is a block diagram of an image processing system according to a second embodiment. [Figure 8] FIG. 10 is an external view of an image processing system according to a second embodiment. [Figure 9] 10 is a flowchart of learning of a machine learning model in Example 2. [Figure 10] FIG. 10 is a diagram showing the flow of increasing the resolution in the second embodiment. [Figure 11] FIG. 10 is a schematic diagram of a subnet in the second embodiment. [Figure 12]10 is a flowchart of generating an estimated high-resolution image in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] A system including a processing device of the present invention will be described below with reference to the accompanying drawings. Note that the same components in each drawing are given the same reference numerals, and duplicated explanations will be omitted.

[0013] [Example 1] First, before describing in detail Example 1, which is an embodiment of the present invention, the gist of the present invention will be explained. Example 1 described below is one of the preferred embodiments of the present invention, and not all of the embodiments are necessary to realize the present invention.

[0014] In this embodiment, a generator, which is a machine learning model, converts a low-resolution image (first image) into a feature map (first feature map), and then generates two intermediate images (first intermediate image and second intermediate image) from the first feature map, each of which has a higher resolution than the low-resolution image. Hereinafter, the first intermediate image and the second intermediate image will also be referred to as the first intermediate high-resolution image and the second intermediate high-resolution image, respectively.

[0015] The generator is trained using different loss functions for the two intermediate high-resolution images. The loss functions have a first loss based on the difference between the intermediate high-resolution image and the correct high-resolution image, and a second loss defined based on the classification output of the classifier that identifies whether the input image is an image generated by the generator.

[0016] The first intermediate high-resolution image is generated using the results of training using a loss function in which the weight of the second loss relative to the first loss is smaller than that of the second intermediate high-resolution image. For example, training to generate the first intermediate high-resolution image uses only the first loss as the loss function.

[0017] The second intermediate high-resolution image may be generated using the results of training using a weighted sum of the first loss and the second loss as a loss function, resulting in an image with high-resolution texture equivalent to that trained by the SRGAN, but with the possibility of artificial structures appearing.

[0018] On the other hand, the learning method used to generate the first intermediate high-resolution image is not a GAN (or its contribution is weaker than that of the second intermediate high-resolution image), resulting in an image in which both high-resolution texture and false structures are reduced. By combining these two intermediate high-resolution images (for example, by weighted averaging), a high-resolution image (estimated image) can be generated in which the perceived resolution and the appearance of false structures are adjusted. The estimated image has a higher resolution than the low-resolution image. Hereinafter, the estimated image will also be referred to as the estimated high-resolution image.

[0019] Unlike network interpolation, this method combines high-resolution images, which are the target of the loss function, rather than the weight of the generator, and therefore can suppress image quality degradation such as edge multiplexing and color changes.In addition, since one generator simultaneously generates two intermediate high-resolution images, it can also suppress increases in calculation time.

[0020] In the following, the stage of determining the weights of the generator and discriminator, which are machine learning models, based on a training dataset, is referred to as "training," and the stage of generating an estimated high-resolution image from a low-resolution image using the generator with the trained weights is referred to as "estimation." Machine learning models include, for example, neural networks, genetic programming, and Bayesian networks. Neural networks include CNNs (Convolutional Neural Networks), GANs (Generative Adversarial Networks), and RNNs (Recurrent Neural Networks).

[0021] Next, the image processing system according to the first embodiment will be described.

[0022] 2 and 3 are a block diagram and an external view of the image processing system 100, respectively.

[0023] The image processing system 100 includes a learning device 101, a resolution increasing device 102, and a control device 103, which are connected to each other via a wired or wireless network.

[0024] The control device 103 has a storage unit 131, a communication unit 132, and a display unit 133, and transmits a request to the resolution increasing device 102 to increase the resolution of a low-resolution image in accordance with a user's instruction.

[0025] The resolution improvement device 102 has a storage unit 121, a communication unit 122, an acquisition unit 123, and a resolution improvement unit 124, and uses a generator, which is a trained machine learning model, to perform a resolution improvement process on a low-resolution image to generate an estimated high-resolution image. The functions of the acquisition unit 123 and the resolution improvement unit 124 can be implemented by one or more processors (processing means), such as CPUs. The resolution improvement device 102 acquires information on the weights of the generator from the learning device 101 and stores it in the storage unit 121.

[0026] The learning device 101 has a memory unit 111, an acquisition unit 112, a calculation unit 113, and an update unit 114, and learns the weights of the generator. The functions of the acquisition unit 112, the calculation unit 113, and the update unit 114 can be implemented by one or more processors (processing means) such as CPUs.

[0027] In the image processing system 100 configured in this manner, the control device 103 acquires the estimated high-resolution image generated by the resolution increasing device 102 and presents the result to the user via the display unit 133 .

[0028] Next, weight learning executed by learning device 101 will be described with reference to the flowchart of FIG.

[0029] The learning in Example 1 is composed of two stages: a first learning stage that does not use a classifier, and a second learning stage (GAN) that uses a classifier. The learning device 101 has a storage unit 111, an acquisition unit 112, a calculation unit 113, and an update unit 114, and each step is executed by one of these units. Note that the functions (methods) of each flowchart described below can also be realized as a program that causes a computer to execute the functions (methods).

[0030] In step S101, the acquisition unit 112 acquires one or more pairs of high-resolution images and low-resolution images from the storage unit 111. The storage unit 111 stores a learning dataset consisting of a plurality of high-resolution images and low-resolution images. Corresponding low-resolution images and high-resolution images contain the same object (subject). The low-resolution image may be generated by downsampling the high-resolution image. In the first embodiment, the number of pixels of the high-resolution image is 16 times that of the low-resolution image (four times both horizontally and vertically). However, the relationship between the number of pixels is not limited to this. The image may be either color or grayscale. Degradation other than downsampling (such as JPEG compression noise) may be added to the low-resolution image. This allows the generator to be endowed with a function for correcting image degradation in addition to a function for increasing resolution.

[0031] In step S102, the calculation unit 113 inputs the low-resolution image to a generator to generate first and second intermediate high-resolution images. The generator is, for example, a convolutional neural network (CNN), and in the first embodiment, a model having the configuration shown in FIG. 1 is used. However, the invention is not limited to this.

[0032] The initial values of the weights of the generator may be generated using random numbers or the like. In the generator shown in FIG. 1, an input low-resolution image 201 is processed by a subnetwork (sub-network) 211 to be converted into a first feature map 202. The subnetwork 211 has one or more linear sum layers. The linear sum layer has the function of calculating the linear sum of the input to the linear sum layer and the weights of the linear sum layer. The linear sum layer may be, for example, a convolutional layer, a deconvolutional layer, or a full-connection layer.

[0033] The subnet 211 also has one or more activation functions, which are nonlinear transformations. The activation functions include ReLU (Rectified Linear Unit), a sigmoid function, and a hyperbolic tangent function.

[0034] In the first embodiment, the low-resolution image 201 has fewer pixels than the corresponding high-resolution image. Therefore, in the first embodiment, the subnet 211 has an upsampling layer that increases the number of pixels in the horizontal and vertical directions. That is, the upsampling layer is a layer that has a function of performing upsampling on the input to the upsampling layer. The upsampling layer in the first embodiment has a function of performing sub-pixel convolution (also called pixel shuffler). The upsampling layer is not limited to this, and the upsampling function may be realized by deconvolution, bilinear interpolation, nearest neighbor interpolation, or the like. However, when sub-pixel convolution is used, the effect of zero padding is reduced compared to other methods, and the degree of freedom is increased by convolution with weights, thereby improving the final effect of high resolution.

[0035] In the first embodiment, the subnet 211 has the configuration shown in FIG. 5(A).

[0036] "Conv." means convolution, "sum" means pixel-by-pixel sum, and "sub-pixel conv." means sub-pixel convolution.

[0037] Moreover, "residual block" represents a residual block. The configuration of the residual block in the first embodiment is shown in FIG.

[0038] A residual block is a block that is composed of multiple linear sum layers and activation functions. The residual block is configured to sum the input to the residual block and the results of a series of operations within the residual block, and output the sum. "Concatenation" refers to concatenation in the channel direction.

[0039] In the first embodiment, the subnet 211 has eight residual blocks. However, the number of residual blocks is not limited to this. If it is desired to further improve the performance of the generator, it is advisable to increase the number of residual blocks.

[0040] In the first embodiment, there are multiple upsampling layers (subpixel convolutions). In the first embodiment, to upsample the number of pixels of a low-resolution image by 16 times, subpixel convolutions are performed twice to upsample by 4 times. If high-magnification upsampling is performed in one upsampling layer, grid-like patterns and the like are likely to occur in the high-resolution image, so it is desirable to perform low-magnification upsampling multiple times like this. Note that in the first embodiment, an example is described in which the upsampling layer is in subnet 211, but the present invention is not limited to this. The upsampling layer may be present in subnet 212 and subnet 213 instead of subnet 211.

[0041] The first feature map 202 is input to a subnet 212 to generate a first residual component 203. The first feature map 202 is also input to a subnet 213 to generate a second residual component 204.

[0042] Each of subnet 212 and subnet 213 has one or more linear sum layers. In the first embodiment, subnet 212 and subnet 213 are each configured with one convolution layer. Note that subnet 212 and subnet 213 can also be combined into one linear sum layer. For example, doubling the number of filters in the convolution layer will result in an output with twice the number of channels of low-resolution image 201 (three for color). This output can be divided into two in the channel direction to generate first residual component 203 and second residual component 204.

[0043] The first residual component 203 is summed with the low-resolution image 201 to generate a first intermediate high-resolution image 205. The second residual component 204 is summed with the low-resolution image 201 to generate a second intermediate high-resolution image 206. The low-resolution image 201 is upsampled before summation so that the number of pixels of the first residual component 203 and the second residual component 204 match. This upsampling may be performed using bilinear interpolation or bicubic interpolation, or a deconvolution layer, for example. Estimating the residual component rather than the high-resolution image itself reduces image quality degradation, such as color changes, from the low-resolution image 201.

[0044] The low-resolution image 201 may be upsampled in advance using bicubic interpolation or the like so that the number of pixels matches that of the high-resolution image, and then input to the generator. In this case, an upsampling layer is not required in the generator. However, as the number of pixels in the horizontal and vertical directions of the low-resolution image 201 increases, the number of times linear sums are calculated increases, resulting in a heavy calculation load. Therefore, it is desirable to input the low-resolution image 201 to the generator without upsampling, as in the first embodiment, and then upsample it within the generator.

[0045] In step S103, the update unit 114 updates the weights of the generator based on the first loss. The first loss in the first embodiment is a loss defined based on the difference between a high-resolution image (correct image) corresponding to the low-resolution image 201 and an intermediate high-resolution image. In the first embodiment, MSE (Mean Square Error) is used, but MAE (Mean Absolute Error) or the like may also be used.

[0046] In the first embodiment, the sum of the MSE between the first intermediate high-resolution image 205 and the high-resolution image and the MSE between the second intermediate high-resolution image 206 and the high-resolution image is used as the loss function, and the weights of the generator are updated by backpropagation.

[0047] In step S104, the update unit 114 determines whether the first learning has been completed. Completion can be determined by, for example, whether the number of iterations of learning (weight update) has reached a predetermined number, or whether the amount of change in weight during update is smaller than a predetermined value. If it is determined in step S104 that weight learning has not been completed, the process returns to step S101, and the acquisition unit 112 acquires one or more pairs of new low-resolution images 201 and high-resolution images. On the other hand, if it is determined that the first learning has been completed, the process proceeds to step S105, and the second learning is started.

[0048] In step S105, the acquisition unit 112 acquires one or more pairs of high-resolution and low-resolution images 201 from the storage unit 111.

[0049] In step S106, the calculation unit 113 inputs the low-resolution image 201 to a generator to generate a first intermediate high-resolution image 205 and a second intermediate high-resolution image 206.

[0050] In step S107, the calculation unit 113 inputs the second intermediate high-resolution image 206 and the high-resolution image to a classifier to generate a classification output. The classifier classifies whether the input image is a high-resolution image generated by the generator or an actual high-resolution image. The classifier may be a CNN or the like. The initial values of the weights of the classifier are determined by random numbers or the like. Note that the high-resolution image input to the classifier may be any actual high-resolution image, and does not need to be an image corresponding to the low-resolution image 201.

[0051] In step S108, the update unit 114 updates the weights of the classifier based on the classification output and the correct label. In the first embodiment, the correct label for the second intermediate high-resolution image 206 is set to 0, and the correct label for the actual high-resolution image is set to 1. Although sigmoid cross-entropy is used as the loss function, other functions may also be used.

[0052] In step S109, the update unit 114 updates the weights of the generator based on the first loss and the second loss. For the first intermediate high-resolution image 205, only the first loss (MSE with the corresponding high-resolution image) is taken. For the second intermediate high-resolution image 206, a weighted sum of the first loss and the second loss is taken. The second loss is the sigmoid cross-entropy between the classification output when the second intermediate high-resolution image 206 is input to the classifier and the correct label 1. Since the generator is intended to train so that the classifier does not erroneously classify the second intermediate high-resolution image 206 as an actual high-resolution image, the correct label is set to 1 (corresponding to the actual high-resolution image). The sum of the losses of the first intermediate high-resolution image 205 and the second intermediate high-resolution image 206 is used as the loss function of the generator.

[0053] By repeatedly updating the weights using this loss function, a natural-looking high-resolution image with high-resolution texture that would otherwise cause the classifier to make a false judgment is generated on the second intermediate high-resolution image 206 side. However, this can have the disadvantage of introducing false structures. On the other hand, on the first intermediate high-resolution image 205 side, both the high-resolution texture and false structures are suppressed, resulting in an output high-resolution image with fewer high-frequency components than the second intermediate high-resolution image 206.

[0054] In the first embodiment, only the first loss is used for the first intermediate high-resolution image 205, but the second loss may also be used. In this case, the weight of the second loss in the first loss should be smaller for the first intermediate high-resolution image 205 than for the second intermediate high-resolution image 206. The order of steps S108 and S109 may be reversed.

[0055] In step S110, the update unit 114 determines whether the second learning is complete. If it is determined that the second learning is incomplete, the process returns to step S105 to acquire one or more new pairs of low-resolution image 201 and high-resolution image. If the second learning is complete, the weight information is stored in the storage unit 111. Note that, since only the generator is used during estimation, the weights of only the generator may be stored.

[0056] Next, the estimation (generation of an estimated high-resolution image) performed by resolution enhancement device 102 and control device 103 will be described using the flowchart in Fig. 6. Resolution enhancement device 102 has a storage unit 121, a communication unit 122, an acquisition unit 123, and a resolution enhancement unit 124, and control device 103 has a storage unit 131, a communication unit 132, and a display unit 133, and each step is performed by any of these units.

[0057] In step S201, the communication unit 132 of the control device 103 transmits a request to perform resolution enhancement to the resolution enhancement device 102. The request also includes information specifying the low-resolution image 201 to be enhanced in resolution. Alternatively, the low-resolution image 201 itself to be enhanced in resolution may be transmitted together with the request to perform processing.

[0058] In step S202 , the communication unit 122 of the resolution increasing device 102 receives the execution request sent from the control device 103 .

[0059] In step S203, the acquisition unit 123 acquires information on the weights of the generator and the low-resolution image 201 to be subjected to resolution enhancement from the storage unit 121. The low-resolution image 201 may be acquired from another storage device connected via a wired or wireless connection.

[0060] In step S204, the resolution increasing unit 124 generates a first intermediate high-resolution image 205 and a second intermediate high-resolution image 206 from the low-resolution image 201 using the generator shown in FIG. 1. The second intermediate high-resolution image 206 is a high-resolution image that has a natural appearance and high-resolution texture, but may contain artifacts. On the other hand, the first intermediate high-resolution image 205 is a high-resolution image in which both the high-resolution texture and artifacts are suppressed. The first intermediate high-resolution image 205 has fewer high-frequency components than the second intermediate high-resolution image 206.

[0061] In step S205, the resolution increasing unit 124 generates an estimated high-resolution image 207 based on the first intermediate high-resolution image 205 and the second intermediate high-resolution image 206. In the first embodiment, the estimated high-resolution image 207 is generated by taking a weighted average of the first intermediate high-resolution image 205 and the second intermediate high-resolution image 206. Note that generation of the high-resolution image 207 is not limited to taking a weighted average of the first intermediate high-resolution image 205 and the second intermediate high-resolution image 206, and the high-resolution image 207 may be generated by replacing a partial region of the second intermediate high-resolution image 206 with the first intermediate high-resolution image 205, for example.

[0062] In step S206, the communication unit 122 transmits the estimated high-resolution image 207 to the control device 103.

[0063] In step S207, the communication unit 132 of the control device 103 acquires an estimated high-resolution image 207. The acquired estimated high-resolution image 207 is stored in the storage unit 131 or displayed on the display unit 133. Alternatively, the image may be stored in another storage device connected to the control device 103 or the resolution enhancement device 102 via a wired or wireless connection. Alternatively, the control device 103 may be configured to acquire the first intermediate high-resolution image 205 and the second intermediate high-resolution image 206, and the control device 103 may generate the estimated high-resolution image 207. In this case, the user can adjust the perceived resolution and the appearance of false structures while checking the actual image on the display unit 133.

[0064] Next, the desirable conditions for obtaining the effects of the present invention will be described.

[0065] It is desirable that the number of linear sum layers in the generator be fewer on the output side than on the input side of the upsampling layer closest to the output side. This is because if upsampling is performed early in the generator's calculations, the number of subsequent linear sum calculations increases, increasing the computational load. In Example 1, there are 40 or more linear sum layers on the input side of the upsampling layer closest to the output side, but on the output side there is only one linear sum layer for each of the first residual component 203 and the second residual component 204.

[0066] Furthermore, the generator is configured with multiple linear sum layers, and the output of at least one of the multiple linear sum layers is preferably concatenated with the input to the linear sum layer in the channel direction. This means, for example, the concatenation shown in Figure 5(b). This allows the generator to transmit more feature maps after the layer, improving the accuracy of the first intermediate high-resolution image 205 and the second intermediate high-resolution image 206.

[0067] Furthermore, it is preferable that at least half of the linear sum layers in the generator are connected to the inputs to the linear sum layers in the channel direction, thereby further improving the accuracy of the high-resolution image.

[0068] It is also desirable that each of the multiple residual blocks in the generator has three or more linear sum layers, which can improve the accuracy of high-resolution processing. Furthermore, it is desirable that each residual block has two or more activation functions, which can increase the nonlinear effect and improve the accuracy of high-resolution processing.

[0069] Furthermore, it is desirable that half or less of the multiple residual blocks in the generator have a batch regularization layer that performs batch regularization. Unlike recognition tasks, in regression tasks that estimate images from images, the effect of batch regularization on improving accuracy is small. Therefore, it is desirable to reduce the number of batch regularizations in order to reduce the computational load. If it is desired to further reduce the computational load, the generator may be configured without a batch regularization layer.

[0070] Furthermore, in training the generator, it is advisable to use a trained feature extractor that converts images into feature maps. The feature extractor converts the correct high-resolution image corresponding to the low-resolution image 201 into a second feature map, and converts the second intermediate high-resolution image 206 into a third feature map. It is advisable to train the generator by adding a third loss based on the difference between the second and third feature maps (e.g., MSE) to the loss function. This allows the loss function to take abstract features into account, making the high-resolution image look more natural.

[0071] Furthermore, the second loss is preferably based on a comparison between a value based on each of the classifier's classification outputs for multiple actual high-resolution images and the classification output for the first intermediate high-resolution image 205 or the second intermediate high-resolution image 206. This is a technique called Relativistic GAN. The value based on each of the classifier's classification outputs for multiple actual high-resolution images may be the average or median of the classification outputs. For example, sigmoid cross-entropy is calculated so that the difference between the classification output for the first intermediate high-resolution image 205 or the second intermediate high-resolution image 206 and the average classification output for the actual high-resolution image indicates the correct label (here, 1). This enables learning from a relative perspective of whether the high-resolution image generated by the generator is more realistic than the set of actual high-resolution images. Conventional GANs sometimes encounter problems when attempting to learn authenticity solely from the high-resolution images generated by the generator, ignoring the actual high-resolution images during training. However, Relativistic GAN can avoid this problem and improve the stability of learning.

[0072] With the above configuration, it is possible to provide an image processing system that can control the sense of resolution and the appearance of false structures while suppressing an increase in the calculation load and deterioration of image quality when increasing the resolution of an image using a machine learning model. In other words, it is possible to provide a high-quality image when increasing the resolution of an image using a machine learning model.

[0073] [Example 2] An image processing system according to a second embodiment of the present invention will be described.

[0074] 7 and 8 are a block diagram and an external view, respectively, of an image processing system 300. The image processing system 300 includes a learning device 301 and an imaging device 302. The imaging device 302 includes an optical system 321, an imaging element 322, an image processing unit 323, a memory unit 324, a communication unit 325, a display unit 326, and a system controller 327. The optical system 321 collects light incident from the subject space and forms a subject image. The optical system 321 has functions such as zooming, aperture adjustment, and autofocus as necessary. The imaging element 322 converts the subject image into an electrical signal by photoelectric conversion to generate a captured image. The imaging element 322 is, for example, a charge-coupled device (CCD) sensor or a complementary metal-oxide semiconductor (CMOS) sensor. The captured image is acquired as a live view of the subject space before shooting or when the user presses the shutter release button. The image is subjected to predetermined processing by the image processing unit 323 and displayed on the display unit 326.

[0075] When a user instructs digital zoom and presses the release button during shooting, the captured image (low-resolution image) is made high-resolution by a generator, which is a machine learning model, in the image processing unit 323. At this time, weights learned by the learning device 301 are used. Information about the weights is read out in advance from the learning device 301 via the communication unit 325 and stored in the storage unit 324. Details regarding the learning and estimation of the generator will be described later.

[0076] In addition, in the live view when the user instructs digital zoom, an image that has been upsampled using a high-speed method such as bilinear interpolation is displayed on the display unit 326. The captured image (estimated high-resolution image) that has been made high-resolution by the generator is stored in the storage unit 324 and displayed on the display unit 326. The above operations are controlled by the system controller 327.

[0077] Although FIG. 8 shows a so-called single-lens camera as the imaging device 302, the imaging device 302 may also be a device such as a smartphone.

[0078] Next, the learning of the weights of the generator executed by the learning device 301 will be described using the flowchart in Fig. 9. The learning device 301 has a storage unit 311, an acquisition unit 312, a calculation unit 313, and an update unit 314, and each step is executed by any of these units.

[0079] In step S301, the acquisition unit 312 acquires one or more pairs of low-resolution and high-resolution images from the storage unit 311. In the second embodiment, the number of pixels in the high-resolution image is 16 times that of the low-resolution image, but is not limited to this.

[0080] In step S302, the calculation unit 313 inputs a low-resolution image to the generator, which generates a first intermediate high-resolution image and a second intermediate high-resolution image. In the second embodiment, the generator has the configuration shown in FIG. 10. Subnet 411 converts a low-resolution image 401 into a first feature map 402, and from the first feature map 402, subnet 412 generates a first intermediate high-resolution image 403, and subnet 413 generates a second intermediate high-resolution image 404. Subnet 411 has the configuration shown in FIG. 5(A), and the residual block has the configuration shown in FIG. 11. Subnet 411 has four residual blocks. Subnet 412 and subnet 413 each have one convolution layer. However, the configuration of each subnet is not limited to this.

[0081] In step S303, the calculation unit 313 inputs the high-resolution image and the second intermediate high-resolution image 404 to the classifier, respectively, and generates a classification output.

[0082] In step S304, the update unit 314 updates the weights of the classifier based on the classification output and the correct label.

[0083] In step S305, the update unit 314 updates the weights of the generator based on the first loss and the second loss. The first loss and the second loss are the same as those described in the first embodiment.

[0084] In step S306, the update unit 314 determines whether the learning of the generator is complete. If it is determined that the learning of the weights is not complete, the process returns to step S301. If it is determined that the learning is complete, the learning is terminated and the weight information is stored in the storage unit 311.

[0085] 12, a description will be given of the process of increasing the resolution of a digitally zoomed captured image, which is performed by the image processing unit 323. The image processing unit 323 has an acquisition unit 323a, a resolution increasing unit 323b, and a calculation unit 323c, and each step is performed by any of these units.

[0086] In step S401, the acquisition unit 323a extracts a partial region (low-resolution image 401) from the captured image. Since the captured image has information on all pixels acquired by the image sensor 322, only the partial region necessary for digital zoom is extracted.

[0087] In step S402, the acquisition unit 323a acquires information on the weight of the generator from the storage unit 324. Note that the order of steps S401 and S402 does not matter.

[0088] In step S403, the resolution increasing unit 323b inputs a partial region of the captured image (low-resolution image 401) to a generator, and generates a first intermediate high-resolution image 403 and a second intermediate high-resolution image 404.

[0089] In step S404, the calculation unit 323c generates an estimated high-resolution image 405 by taking a weighted average of the first intermediate high-resolution image 403 and the second intermediate high-resolution image 404.

[0090] In step S405, the calculation unit 323c scales (upsamples or downsamples) the estimated high-resolution image 405 to a specified number of pixels. The generator is trained to upsample the number of pixels by 16 times, so it is necessary to match the digital zoom magnification specified by the user. Bicubic interpolation or the like may be used for downsampling, and anti-aliasing processing may be performed as necessary. If a digital zoom magnification of more than 4 times in one dimension is specified, the estimated high-resolution image 405 is upsampled using bicubic interpolation or the like. Alternatively, the estimated high-resolution image 405 may be input to the generator as a new low-resolution image 401.

[0091] With the above configuration, it is possible to provide an image processing system that can control the sense of resolution and the appearance of false structures while suppressing an increase in the calculation load and deterioration of image quality when increasing the resolution of an image using a machine learning model. In other words, it is possible to provide a high-quality image when increasing the resolution of an image using a machine learning model.

[0092] (Other Examples) In the above embodiments, examples have been described in which MSE or MAE is used as the first loss and the classification result by a classifier is used as the second loss, but the present invention is not limited to this. The effects of the present invention can be obtained by generating a first intermediate image and a second intermediate image with different characteristics from a first feature map based on a low-resolution image, and generating an estimated image based on the first intermediate image and the second intermediate image. This is because it makes it possible to cover up any adverse effects that occur in one of the first intermediate image and the second intermediate image with the other.

[0093] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0094] According to each embodiment, it is possible to provide an image processing device, an imaging device, an image processing method, an image processing program, and a storage medium that are capable of generating high-quality images by increasing the resolution of images using a machine learning model.

[0095] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and various combinations, modifications, and variations are possible within the scope of the present invention. For example, in the above-described embodiments, physical information or exercise information is set as goal information, but the present invention is not limited to this. The amount of studying or reading may also be set as goal information. [Explanation of symbols]

[0096] 200 low-resolution images 202 First feature map 205 First intermediate image 206 Second intermediate image 207 Estimated Images

Claims

1. generating a first intermediate image based on the first image using a first generator, the first intermediate image having a higher resolution than the first image; generating a second intermediate image based on the first image using a second generator, the second intermediate image having a higher resolution than the first image; generating an estimated image having a higher resolution than the first image based on the first intermediate image and the second intermediate image, the first generator is obtained by learning without using a discriminator; The image processing method, wherein the second generator is obtained by learning using a classifier.

2. 2. The image processing method according to claim 1, wherein the classifier identifies whether an image input in the learning is an image generated by a generator.

3. 3. The image processing method according to claim 1, wherein the first generator and the second generator include a common sub-network.

4. The image processing method of claim 3 , wherein the sub-network generates a first feature map based on the first image.

5. the first generator generates the first intermediate image by taking a sum of a first residual component generated based on the first feature map and the first image; 5. The image processing method according to claim 4, wherein the second generator generates the second intermediate image by taking a sum of a second residual component generated based on the first feature map and the first image.

6. converting a first image into a first feature map by inputting the first image into a generator; generating a first intermediate image having a higher resolution than the first image and a second intermediate image having a higher resolution than the first image based on the first feature map; and generating an estimated image having a higher resolution than the first image by adjusting the perceived resolution based on the first intermediate image and the second intermediate image.

7. The generator generating the first intermediate image by taking a sum of a first residual component generated based on the first feature map and the first image; The image processing method according to claim 6, wherein the second intermediate image is generated by taking the sum of a second residual component generated based on the first feature map and the first image.

8. 8. The image processing method according to claim 1, wherein the first intermediate image and the second intermediate image have a larger number of pixels than the first image.

9. 8. The image processing method according to claim 5, wherein the first image is upsampled before summing so that the number of pixels of the first residual component and the second residual component match.

10. 10. The image processing method according to claim 1, wherein the estimated image is generated by a weighted average of the first intermediate image and the second intermediate image.

11. obtaining a plurality of training images and a plurality of ground truth images; a step of converting the training image into a first feature map by inputting the training image into a generator, and generating a first intermediate image having a higher resolution than the training image and a second intermediate image having a higher resolution than the training image based on the first feature map; a step of identifying whether an image input to a classifier is an image generated by the generator; training the generator based on a first loss based on a difference between the correct image corresponding to the training image and the first intermediate image or the second intermediate image, and a second loss based on a classification output of the classifier when the first intermediate image or the second intermediate image is input; and generating an estimated image by inputting a first image to the generator.

12. A program causing a computer to execute the image processing method according to any one of claims 1 to 11.

13. means for generating a first intermediate image having a higher resolution than the first image based on the first image using a first generator; means for generating a second intermediate image having a higher resolution than the first image based on the first image using a second generator; means for generating an estimated image having a higher resolution than the first image based on the first intermediate image and the second intermediate image; the first generator is obtained by learning without using a discriminator; The image processing device, wherein the second generator is obtained by learning using a classifier.

14. means for converting a first image into a first feature map by inputting the first image into a generator; means for generating a first intermediate image having a higher resolution than the first image and a second intermediate image having a higher resolution than the first image based on the first feature map; and means for generating an estimated image having a higher resolution than the first image by adjusting the perceived resolution based on the first intermediate image and the second intermediate image.

Images