Image processing method, image processing device, image processing program, learning method, learning device, learning program

Abstract

The goal is to provide high-quality images through image resolution enhancement using machine learning models. [Solution] The image processing method of the present invention uses a generator which is a machine learning model. In the generator, the input low-resolution image is converted into a first feature map 202. From the first feature map 202, a first intermediate image 203 with a higher resolution than the low-resolution image and a second intermediate image 204 with a higher resolution than the low-resolution image are generated. Based on the first intermediate image 203 and the second intermediate image 204, an estimated image 207 with a higher resolution than the low-resolution image is generated.

Description

Technical Field

[0001] The present invention relates to a technique for high-resolution imaging of images using a machine learning model.

Background Art

[0002] Patent Document 1 discloses a method for high-resolution imaging of images using a machine learning model called a Generative Adversarial Network (GAN). This method is called SRGAN (Super Resolution GAN). SRGAN performs learning using a generator that generates high-resolution images and a discriminator that discriminates whether an input image is an image generated by the generator or an actual high-resolution image. Here, an actual high-resolution image means a high-resolution image that is not generated by the generator.

[0003] The generator learns weights so that the discriminator cannot distinguish between the generated high-resolution image and the actual high-resolution image. As a result, a high-resolution image with a more natural appearance having a high-resolution texture can be generated. However, at the same time, as a problem, there is a possibility that a subjectively uncomfortable pseudo structure may appear.

[0004] On the other hand, Non-Patent Document 1 discloses a method for controlling the appearance of pseudo structures and the resolution feeling. In Non-Patent Document 1, a weighted average is taken between the weights of a generator that has learned high-resolution imaging without using a discriminator (less pseudo structures but low resolution feeling) and the weights of a generator that has learned using a discriminator (equivalent to SRGAN, high resolution feeling but pseudo structures may exist). A high-resolution image is generated by a generator using this weighted average weight. This method is called network interpolation. By changing the weights of the weighted average, the balance between the appearance of pseudo structures and the resolution feeling can be controlled.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006] [Non-Patent Document 1] Xintao Wang,et al.,Deep Network Interpolation for Continuous Imagery Effect Transition,Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR),2019,pp.1692-1701 [Overview of the project] [Problems that the invention aims to solve]

[0007] However, the inventors' research has revealed that the generator using network interpolation described in Non-Patent Document 1 may result in image quality degradation in the generated high-resolution images, such as edge duplication where the subject appears doubled and color changes.

[0008] Therefore, the objective of the present invention is to provide high-quality images when using a machine learning model to increase the resolution of images. [Means for solving the problem]

[0009] The present invention provides an image processing method comprising the steps of: using a generator which is a machine learning model to convert an input low-resolution image into a first feature map; and generating a first intermediate image with a higher resolution than the low-resolution image and a second intermediate image with a higher resolution than the low-resolution image from the first feature map; The method is characterized by comprising the step of generating an estimated image with a higher resolution than the low-resolution image based on the first intermediate image and the second intermediate image. [Effects of the Invention]

[0010] According to the present invention, in the super-resolution of an image using a machine learning model, a high-quality image can be provided.

Brief Description of the Drawings

[0011] [Figure 1] It is a diagram showing the super-resolution process in Example 1. [Figure 2] It is a block diagram of the image processing system in Example 1. [Figure 3] It is an external view of the image processing system in Example 1. [Figure 4] It is a flowchart of the learning of the machine learning model in Example 1. [Figure 5] It is a schematic diagram of the sub-networks in Examples 1 and 2. [Figure 6] It is a flowchart of the generation of the estimated super-resolution image in Example 1. [Figure 7] It is a block diagram of the image processing system in Example 2. [Figure 8] It is an external view of the image processing system in Example 2. [Figure 9] It is a flowchart of the learning of the machine learning model in Example 2. [Figure 10] It is a diagram showing the super-resolution process in Example 2. [Figure 11] It is a schematic diagram of the sub-network in Example 2. [Figure 12] It is a flowchart of the generation of the estimated super-resolution image in Example 2.

Modes for Carrying Out the Invention

[0012] Hereinafter, a system including the processing device of the present invention will be described based on the accompanying drawings. In each figure, the same components are denoted by the same reference numerals, and redundant descriptions are omitted.

[0013] [Example 1] Before detailing Example 1, which is an example of the present invention, its gist will be explained. Example 1 described below is one of the preferred embodiments of the present invention, and not all of it is necessary for the realization of the present invention.

[0014] In this example, a generator, which is a machine learning model, converts a low-resolution image into a feature map (first feature map), and further generates two intermediate images (first intermediate image and second intermediate image) that are higher in resolution than the low-resolution image from the first feature map. Hereinafter, the first intermediate image and the second intermediate image are also referred to as the first intermediate high-resolution image and the second intermediate high-resolution image, respectively.

[0015] The generator is trained using different loss functions for these two intermediate high-resolution images. The loss function has a first loss based on the difference between the intermediate high-resolution image and the high-resolution image as the correct answer (correct image), and a second loss defined based on the discrimination output of a discriminator that discriminates whether the input image is an image generated by the generator.

[0016] The first intermediate high-resolution image is generated using the result of training with a loss function in which the weight of the second loss with respect to the first loss is smaller than that of the second intermediate high-resolution image. For example, the training for generating the first intermediate high-resolution image uses only the first loss as the loss function.

[0017] The second intermediate high-resolution image may be generated using the result of training with the weighted sum of the first loss and the second loss as the loss function. As a result, the second intermediate high-resolution image has a high-resolution texture equivalent to that when learned by SRGAN, but may also be an image in which a pseudo structure appears.

[0018] On the other hand, the learning method used to generate the first intermediate high-resolution image is not a GAN (or its contribution is weaker than that of the second intermediate high-resolution image), resulting in an image with reduced high-resolution texture and pseudostructure. By combining these two intermediate high-resolution images (e.g., using a weighted average), a high-resolution image (estimated image) can be generated that adjusts for perceived resolution and the appearance of pseudostructure. The estimated image has a higher resolution than the low-resolution image. Hereafter, the estimated image will also be referred to as the estimated high-resolution image.

[0019] Unlike network interpolation, which uses generator weights, this method combines high-resolution images that are the target of the loss function, thus suppressing image quality degradation such as edge multiplexing and color changes. Furthermore, since one generator produces two intermediate high-resolution images simultaneously, it can also suppress the increase in computation time.

[0020] In the following, the stage of determining the weights of the generator and discriminator, which are machine learning models, based on the training dataset will be referred to as "training," and the stage of generating estimated high-resolution images from low-resolution images using the trained weights of the generator will be referred to as "estimation." Machine learning models include, for example, neural networks, genetic programming, and Bayesian networks. Neural networks include CNNs (Convolutional Neural Networks), GANs (Generative Adversarial Networks), and RNNs (Recurrent Neural Networks).

[0021] Next, the image processing system in Example 1 will be described.

[0022] Figures 2 and 3 are a block diagram and an external view of the image processing system 100, respectively.

[0023] The image processing system 100 includes a learning device 101, a high-resolution device 102, and a control device 103, all connected to each other by a wired or wireless network.

[0024] The control device 103 has a storage unit 131, a communication unit 132, and a display unit 133, and transmits a request to the high-resolution device 102 to perform high-resolution processing on a low-resolution image according to the user's instructions.

[0025] The high-resolution image enhancement device 102 includes a storage unit 121, a communication unit 122, an acquisition unit 123, and a high-resolution image enhancement unit 124. It uses a generator, which is a trained machine learning model, to perform high-resolution processing on low-resolution images and generate estimated high-resolution images. The acquisition unit 123 and the high-resolution image enhancement unit 124 can be implemented by one or more processors (processing means) such as CPUs. The high-resolution image enhancement device 102 acquires information on the weights of the generator from the learning device 101 and stores it in the storage unit 121.

[0026] The learning device 101 has a memory unit 111, an acquisition unit 112, an arithmetic unit 113, and an update unit 114, and learns the weights of the generator. The acquisition unit 112, the arithmetic unit 113, and the update unit 114 can be implemented by one or more processors (processing means) such as CPUs.

[0027] In the image processing system 100 configured in this way, the control device 103 acquires the estimated high-resolution image generated by the high-resolution enhancement device 102 and presents the result to the user via the display unit 133.

[0028] Next, the weight learning process performed by the learning device 101 will be explained using the flowchart in Figure 4.

[0029] The learning process in Example 1 consists of two stages: a first learning process that does not use a classifier, and a second learning process (GAN) that uses a classifier. The learning device 101 has a storage unit 111, an acquisition unit 112, an arithmetic unit 113, and an update unit 114, and each step is executed by one of these units. Note that the functions (methods) of each flowchart described below can also be implemented as programs that cause a computer to execute the corresponding functions (methods).

[0030] In step S101, the acquisition unit 112 acquires one or more pairs of high-resolution and low-resolution images from the storage unit 111. The storage unit 111 stores a training dataset consisting of multiple high-resolution and low-resolution images. The corresponding low-resolution and high-resolution images contain the same object (subject) within the image. Note that the low-resolution image may be generated by downsampling the high-resolution image. In Example 1, the number of pixels in the high-resolution image is 16 times that of the low-resolution image (4 times horizontally and vertically). However, the relationship of the number of pixels is not limited to this. Also, the image may be either color or grayscale. In addition, degradation other than downsampling (such as JPEG compression noise) may be applied to the low-resolution image. This allows the generator to be given a function to correct image degradation in addition to the high-resolution function.

[0031] In step S102, the arithmetic unit 113 inputs a low-resolution image to the generator to generate first and second intermediate high-resolution images. The generator is, for example, a CNN (Convolutional Neural Network), and in Example 1, a model with the configuration shown in Figure 1 is used. However, the invention is not limited thereto.

[0032] The initial values ​​of the generator weights should be generated using random numbers or similar methods. In the generator shown in Figure 1, the input low-resolution image 201 is processed in subnet (subnetwork) 211 and converted into a first feature map 202. Subnet 211 has one or more linear sum layers. A linear sum layer has the function of taking the linear sum of the input to the linear sum layer and the weights of the linear sum layer. Examples of linear sum layers include convolutional layers, deconvolutional layers, and fully connected layers.

[0033] Furthermore, subnet 211 has one or more activation functions, which are nonlinear transformations. These activation functions include ReLU (Rectified Linear Unit), sigmoid function, and hyperbolic tangent function.

[0034] In Example 1, the low-resolution image 201 has fewer pixels than the corresponding high-resolution image. Therefore, in Example 1, subnet 211 has an upsampling layer that increases the number of pixels in the horizontal and vertical directions. That is, the upsampling layer is a layer that has the function of performing upsampling on the input to the upsampling layer. The upsampling layer in Example 1 has the function of performing sub-pixel convolution (also called Pixel Shuffler). The upsampling layer is not limited to this, and the upsampling function may be realized by deconvolution, bilinear interpolation, nearest neighbor interpolation, etc. However, when using sub-pixel convolution, the effect of zero-padding is reduced compared to other methods and the degree of freedom due to convolution with weights is increased, so the effect of final high-resolution can be enhanced.

[0035] In Example 1, subnet 211 has the configuration shown in Figure 5(A).

[0036] "conv." represents convolution, "sum" represents the sum of each pixel, and "sub-pixel conv." represents subpixel convolution.

[0037] Furthermore, "residual block" refers to the residual block. The configuration of the residual block in Example 1 is shown in Figure 5(B).

[0038] A residual block is a block composed of multiple linear sum layers and activation functions. A residual block is configured to output the sum of its inputs and the results of a series of operations performed within the residual block. "Concatenation" indicates channel-direction concatenation.

[0039] In Example 1, subnet 211 has 8 residual blocks. However, the number of residual blocks is not limited to this. If you want to improve the performance of the generator, you can increase the number of residual blocks.

[0040] In Example 1, there are multiple upsampling layers (subpixel convolutions). In Example 1, to upsample the number of pixels in a low-resolution image by 16 times, two subpixel convolutions that upsample by 4 times are performed. Performing high-magnification upsampling in a single upsampling layer tends to cause grid-like patterns in the high-resolution image, so it is desirable to perform multiple low-magnification upsamplings in this manner. Although Example 1 describes an example where the upsampling layer is located in subnet 211, the present invention is not limited to this. The upsampling layer may be located in subnets 212 and 213 instead of subnet 211.

[0041] The first feature map 202 is input to subnet 212, generating the first residual component 203. The first feature map 202 is also input to subnet 213, generating the second residual component 204.

[0042] Subnets 212 and 213 each have one or more linear sum layers. In Example 1, subnets 212 and 213 each consist of one convolutional layer. Note that subnets 212 and 213 can also be combined into a single linear sum layer. For example, doubling the number of filters in the convolutional layer will yield an output with twice the number of channels (3 for color) of the low-resolution image 201. This output may be divided into two in the channel direction to obtain a first residual component 203 and a second residual component 204.

[0043] The first residual component 203 is summed with the low-resolution image 201 to generate the first intermediate high-resolution image 205. Similarly, the second residual component 204 is summed with the low-resolution image 201 to generate the second intermediate high-resolution image 206. The low-resolution image 201 is upsampled before summing so that the number of pixels matches that of the first and second residual components 203 and 204. This upsampling can be done using bilinear interpolation, bicubic interpolation, or even inverse convolutional layers. By estimating the residual components rather than the high-resolution image itself, image quality degradation such as color changes from the low-resolution image 201 is less likely to occur.

[0044] Alternatively, the low-resolution image 201 may be upsampled beforehand using bicubic interpolation or similar methods to match the pixel count of the high-resolution image before being input to the generator. In this case, an upsampling layer is not required in the generator. However, if the number of pixels in the horizontal and vertical directions of the low-resolution image 201 increases, the number of times linear sums are calculated increases, leading to a higher computational load. Therefore, it is preferable to input the low-resolution image 201 to the generator without upsampling it, as in Example 1, and perform the upsampling internally within the generator.

[0045] In step S103, the update unit 114 updates the generator weights based on the first loss. The first loss in Example 1 is a loss defined based on the difference between the high-resolution image (ground truth image) corresponding to the low-resolution image 201 and the intermediate high-resolution image. In Example 1, MSE (Mean Square Error) is used, but MAE (Mean Absolute Error) or other methods may also be used.

[0046] In Example 1, the loss function is the sum of the MSE of the first intermediate high-resolution image 205 and the high-resolution image, and the MSE of the second intermediate high-resolution image 206 and the high-resolution image, and the generator weights are updated by backpropagation.

[0047] In step S104, the update unit 114 determines whether the first learning is complete. Completion can be determined by whether the number of iterations of learning (weight update) has reached a predetermined number, or whether the amount of change in weights during the update is less than a predetermined value. If it is determined in step S104 that the weight learning is not complete, the process returns to step S101, and the acquisition unit 112 acquires one or more new low-resolution images 201 and high-resolution images. On the other hand, if it is determined that the first learning is complete, the process proceeds to step S105, and the second learning begins.

[0048] In step S105, the acquisition unit 112 acquires one or more sets of high-resolution images and low-resolution images 201 from the storage unit 111.

[0049] In step S106, the calculation unit 113 inputs the low-resolution image 201 to the generator to generate a first intermediate high-resolution image 205 and a second intermediate high-resolution image 206.

[0050] In step S107, the arithmetic unit 113 inputs the second intermediate high-resolution image 206 and the high-resolution image to the classifier and generates a classification output. The classifier identifies whether the input image is a high-resolution image generated by the generator or an actual high-resolution image. A CNN or similar architecture may be used for the classifier. The initial values ​​of the classifier's weights are determined by random numbers or the like. Note that the high-resolution image input to the classifier can be any actual high-resolution image and does not need to be the image corresponding to the low-resolution image 201.

[0051] In step S108, the update unit 114 updates the classifier weights based on the identification output and the correct label. In Example 1, the correct label for the second intermediate high-resolution image 206 is 0, and the correct label for the actual high-resolution image is 1. Sigmoid cross-entropy is used as the loss function, but other functions may also be used.

[0052] In step S109, the update unit 114 updates the generator weights based on the first loss and the second loss. For the first intermediate high-resolution image 205, only the first loss (MSE with the corresponding high-resolution image) is taken. For the second intermediate high-resolution image 206, a weighted sum of the first loss and the second loss is taken. The second loss is the sigmoid cross-entropy between the classification output and the correct label 1 when the second intermediate high-resolution image 206 is input to the classifier. The generator wants the classifier to learn to misclassify the second intermediate high-resolution image 206 as an actual high-resolution image, so the correct label is set to 1 (corresponding to the actual high-resolution image). The sum of the losses for the first intermediate high-resolution image 205 and the second intermediate high-resolution image 206 is used as the generator's loss function.

[0053] By repeatedly updating the weights using this loss function, the second intermediate high-resolution image 206 generates a high-resolution image with a natural appearance that has a high-resolution texture that causes the classifier to misclassify. However, a drawback is that false structures may appear. On the other hand, in the first intermediate high-resolution image 205, both the high-resolution texture and false structures are suppressed, and a high-resolution image with fewer high-frequency components than the second intermediate high-resolution image 206 is output.

[0054] In Example 1, only the first loss was used for the first intermediate high-resolution image 205, but the second loss may also be used. In this case, the weight of the second loss in the first loss for the first intermediate high-resolution image 205 should be smaller than that of the second intermediate high-resolution image 206. Note that the order of steps S108 and S109 may be reversed.

[0055] In step S110, the update unit 114 determines whether the second learning is complete. If it determines that it is not complete, it returns to step S105 to acquire one or more new sets of low-resolution images 201 and high-resolution images. If it is complete, the weight information is stored in the storage unit 111. Note that since only the generator is used during estimation, only the weights of the generator may be stored.

[0056] Next, the estimation (generation of estimated high-resolution images) performed by the high-resolution enhancement device 102 and the control device 103 will be explained using the flowchart in Figure 6. The high-resolution enhancement device 102 has a storage unit 121, a communication unit 122, an acquisition unit 123, and a high-resolution enhancement unit 124, and the control device 103 has a storage unit 131, a communication unit 132, and a display unit 133, and each step is performed by one of these.

[0057] In step S201, the communication unit 132 of the control device 103 transmits a request to the high-resolution device 102 to perform high-resolution processing. The request also includes information specifying the low-resolution image 201 to be processed. Alternatively, the low-resolution image 201 to be processed may be transmitted along with the request to perform the processing.

[0058] In step S202, the communication unit 122 of the high-resolution device 102 receives the execution request transmitted from the control device 103.

[0059] In step S203, the acquisition unit 123 acquires the weight information of the generator and a low-resolution image 201 for high-resolution processing from the storage unit 121. The low-resolution image 201 may be acquired from another storage device connected via wired or wireless connection.

[0060] In step S204, the high-resolution unit 124 generates a first intermediate high-resolution image 205 and a second intermediate high-resolution image 206 from the low-resolution image 201 using the generator shown in Figure 1. The second intermediate high-resolution image 206 is a high-resolution image with a natural appearance and high-resolution texture, but false structures may appear. On the other hand, the first intermediate high-resolution image 205 is a high-resolution image in which both high-resolution texture and false structures are suppressed. The first intermediate high-resolution image 205 has fewer high-frequency components than the second intermediate high-resolution image 206.

[0061] In step S205, the high-resolution unit 124 generates an estimated high-resolution image 207 based on the first intermediate high-resolution image 205 and the second intermediate high-resolution image 206. In Embodiment 1, the estimated high-resolution image 207 is generated by a weighted average of the first intermediate high-resolution image 205 and the second intermediate high-resolution image 206. Note that the generation of the high-resolution image 207 is not limited to a weighted average of the first intermediate high-resolution image 205 and the second intermediate high-resolution image 206; it may also be generated by replacing a portion of the second intermediate high-resolution image 206 with the first intermediate high-resolution image 205, for example.

[0062] In step S206, the communication unit 122 transmits the estimated high-resolution image 207 to the control device 103.

[0063] In step S207, the communication unit 132 of the control device 103 acquires the estimated high-resolution image 207. The acquired estimated high-resolution image 207 is stored in the storage unit 131 or displayed on the display unit 133. Alternatively, it may be stored in another storage device connected via wired or wireless connection from the control device 103 or the high-resolution device 102. Furthermore, the control device 103 may be configured to acquire a first intermediate high-resolution image 205 and a second intermediate high-resolution image 206, and the control device 103 may generate the estimated high-resolution image 207. In this case, the user can adjust the perceived resolution and the appearance of false structures while checking the actual image on the display unit 133.

[0064] Next, we will describe the desirable conditions for obtaining the effects of the present invention.

[0065] In a generator, it is desirable that the number of linear sum layers on the output side is less than the number of linear sum layers on the input side of the upsample layer located on the output side. This is because if upsampling is performed early in the generator's calculations, the number of subsequent linear sums increases, increasing the computational load. In Example 1, there are more than 40 linear sum layers on the input side of the upsample layer on the output side, but on the output side there is only one linear sum layer for each of the first residual component 203 and the second residual component 204.

[0066] Furthermore, the generator is configured to have multiple linear sum layers, and it is desirable that the output of at least one of the multiple linear sum layers be configured to be concatenated in the channel direction with the input to the linear sum layer. This means, for example, the concatenation shown as “concatenation” in Figure 5(b). This allows the generator to transmit more feature maps after the layer, improving the accuracy of the first intermediate high-resolution image 205 and the second intermediate high-resolution image 206.

[0067] Furthermore, it is desirable that at least half of the multiple linear sum layers in the generator be coupled to the input to the linear sum layer in the channel direction. This further improves the accuracy of high-resolution processing.

[0068] Furthermore, it is desirable that each of the multiple residual blocks in the generator has three or more linear sum layers. This improves the accuracy of high-resolution enhancement. In addition, it is desirable that the residual blocks have two or more activation functions. This increases the nonlinear effect and improves the accuracy of high-resolution enhancement.

[0069] Furthermore, it is desirable that less than half of the multiple residual blocks in the generator have a batch regularization layer that performs batch regularization. Unlike recognition tasks, the effect of batch regularization on improving accuracy is small in regression tasks that estimate images from images. Therefore, it is desirable to reduce the number of batch regularizations in order to suppress the computational load. If further reduction in computational load is desired, the generator may be configured without a batch regularization layer.

[0070] Furthermore, in training the generator, it is advisable to use a pre-trained feature extractor that converts images into feature maps. The feature extractor converts the ground truth high-resolution image corresponding to the low-resolution image 201 into a second feature map, and the second intermediate high-resolution image 206 into a third feature map. It is advisable to add a third loss based on the difference between the second and third feature maps (e.g., MSE) to the loss function and train the generator. This allows the loss function to take into account abstract features as well, resulting in a more natural appearance of the high-resolution image.

[0071] Furthermore, the second loss should preferably be based on a comparison between the classifier's classification output for each of the multiple actual high-resolution images and the classification output of the first intermediate high-resolution image 205 or the second intermediate high-resolution image 206. This is a technique called a Relativistic GAN. The value based on the classifier's classification output for each of the multiple actual high-resolution images can be the mean or median of the multiple classification outputs. For example, the sigmoid cross-entropy is calculated such that the difference between the classification output of the first intermediate high-resolution image 205 or the second intermediate high-resolution image 206 and the mean of the classification outputs of the actual high-resolution images represents the correct label (1 in this case). This allows for learning from a relative perspective of whether the high-resolution images generated by the generator are more or less realistic compared to the set of actual high-resolution images. In conventional GANs, a problem sometimes arises where the realism is learned only from the high-resolution images generated by the generator, ignoring the actual high-resolution images during training. However, this problem can be avoided and learning stability can be improved using Relativistic GANs.

[0072] With the above configuration, it is possible to provide an image processing system that can control resolution and the appearance of false structures while suppressing increased computational load and degradation of image quality when using machine learning models to increase image resolution. In other words, it is possible to provide high-quality images when increasing image resolution using machine learning models.

[0073] [Example 2] The image processing system in Embodiment 2 of the present invention will be described below.

[0074] Figures 7 and 8 are a block diagram and an external view of the image processing system 300, respectively. The image processing system 300 includes a learning device 301 and an imaging device 302. The imaging device 302 includes an optical system 321, an image sensor 322, an image processing unit 323, a storage unit 324, a communication unit 325, a display unit 326, and a system controller 327. The optical system 321 collects light incident from the subject space and forms a subject image. The optical system 321 has functions such as zoom, aperture adjustment, and autofocus as needed. The image sensor 322 converts the subject image into an electrical signal by photoelectric conversion and generates an image. The image sensor 322 is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor. The image is acquired from a live view of the subject space before shooting or when the shutter release is pressed by the user, processed by the image processing unit 323, and displayed on the display unit 326.

[0075] When the user instructs the camera to use digital zoom and presses the shutter release during shooting, the captured image (low-resolution image) is upscaled to high resolution by a machine learning model called a generator in the image processing unit 323. At this time, weights learned by the learning device 301 are used. This weight information is read in advance from the learning device 301 via the communication unit 325 and stored in the storage unit 324. Details regarding the learning and estimation of the generator will be described later.

[0076] Furthermore, in live view when the user commands digital zoom, the image upsampled using high-speed techniques such as bilinear interpolation is displayed on the display unit 326. The high-resolution captured image (estimated high-resolution image) generated by the generator is stored in the storage unit 324 and displayed on the display unit 326. The above operations are controlled by the system controller 327.

[0077] Figure 8 shows a so-called single-lens reflex camera as the imaging device 302, but the imaging device 302 could also be a device like a smartphone.

[0078] Next, the learning of the generator weights performed by the learning device 301 will be explained using the flowchart in Figure 9. The learning device 301 has a storage unit 311, an acquisition unit 312, an arithmetic unit 313, and an update unit 314, and each step is performed by one of these units.

[0079] In step S301, the acquisition unit 312 acquires one or more sets of low-resolution and high-resolution images from the storage unit 311. In Embodiment 2, the number of pixels in the high-resolution image is 16 times that of the low-resolution image, but is not limited to this.

[0080] In step S302, the calculation unit 313 inputs a low-resolution image to the generator and generates a first intermediate high-resolution image and a second intermediate high-resolution image. In Embodiment 2, the generator has the configuration shown in Figure 10. Subnet 411 converts the low-resolution image 401 into a first feature map 402, and from the first feature map 402, subnet 412 generates a first intermediate high-resolution image 403, and subnet 413 generates a second intermediate high-resolution image 404. Subnet 411 has the configuration shown in Figure 5(A), and the residual blocks have the configuration shown in Figure 11. Subnet 411 has four residual blocks. Subnets 412 and 413 each consist of one convolutional layer. However, the configuration of each subnet is not limited to this.

[0081] In step S303, the calculation unit 313 inputs the high-resolution image and the second intermediate high-resolution image 404 to the classifier and generates a classification output.

[0082] In step S304, the update unit 314 updates the classifier weights based on the identification output and the correct label.

[0083] In step S305, the update unit 314 updates the generator weights based on the first loss and the second loss. The first loss and the second loss are the same as described in Embodiment 1.

[0084] In step S306, the update unit 314 determines whether the generator's learning is complete. If it is determined that the weight learning is not complete, the unit returns to step S301. If it is determined that it is complete, the learning is terminated and the weight information is stored in the storage unit 311.

[0085] Next, the process of increasing the resolution of a digitally zoomed image, which is performed by the image processing unit 323, will be explained using the flowchart in Figure 12. The image processing unit 323 includes an acquisition unit 323a, a resolution enhancement unit 323b, and a calculation unit 323c, and each step is performed by one of these units.

[0086] In step S401, the acquisition unit 323a extracts a partial region (low-resolution image 401) from the captured image. Since the captured image contains information from all pixels acquired by the image sensor 322, only the partial region necessary for digital zoom is extracted.

[0087] In step S402, the acquisition unit 323a acquires the weight information of the generator from the storage unit 324. The order of steps S401 and S402 does not matter.

[0088] In step S403, the high-resolution unit 323b inputs a portion of the captured image (low-resolution image 401) to the generator and generates a first intermediate high-resolution image 403 and a second intermediate high-resolution image 404.

[0089] In step S404, the calculation unit 323c generates an estimated high-resolution image 405 by weighted averaging of the first intermediate high-resolution image 403 and the second intermediate high-resolution image 404.

[0090] In step S405, the calculation unit 323c scales (upsamples or downsamples) the estimated high-resolution image 405 to the specified number of pixels. Since the generator is trained to upsample the number of pixels by 16 times, it is necessary to match the magnification of the digital zoom specified by the user. Bicubic interpolation or the like may be used for downsampling, and anti-aliasing processing may be performed as needed. If a digital zoom magnification of more than 4 times in one dimension is specified, the estimated high-resolution image 405 is upsampled using bicubic interpolation or the like. Alternatively, the estimated high-resolution image 405 may be input to the generator as a new low-resolution image 401.

[0091] With the above configuration, it is possible to provide an image processing system that can control resolution and the appearance of false structures while suppressing increased computational load and degradation of image quality when using machine learning models to increase image resolution. In other words, it is possible to provide high-quality images when increasing image resolution using machine learning models.

[0092] (Other examples) In the above embodiments, examples were described in which MSE or MAE was used as the first loss and the identification result by a classifier was used as the second loss, but the present invention is not limited thereto. The effects of the present invention can be obtained by generating a first intermediate image and a second intermediate image with different characteristics from a first feature map based on a low-resolution image, and then generating an estimated image based on the first intermediate image and the second intermediate image. This is because it is possible to cover the drawbacks that occur in one of the first intermediate image and the second intermediate image with the other.

[0093] Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0094] According to each embodiment, it is possible to provide an image processing apparatus, an imaging apparatus, an image processing method, an image processing program, and a storage medium that can generate high-quality images when increasing the resolution of images using a machine learning model.

[0095] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various combinations, modifications, and changes are possible within the scope of its gist. For example, in the embodiments described above, an example was given in which physical information or exercise information was set as target information, but the present invention is not limited to this. Study volume or reading volume may also be set as target information. [Explanation of Symbols]

[0096] 200 low-resolution images 202 Feature Map of the First Category 205 First intermediate image 206 Second intermediate image 207 Estimated Images

Claims

1. The process involves using a machine learning model generator to convert an input low-resolution image into a first feature map, and generating a first intermediate image with a higher resolution than the low-resolution image and a second intermediate image with a higher resolution than the low-resolution image from the first feature map, An image processing method characterized by comprising the step of generating an estimated image with a higher resolution than the low-resolution image based on the first intermediate image and the second intermediate image.

2. The image processing method according to claim 1, characterized in that the first intermediate image has fewer high-frequency components than the second intermediate image.

3. The image processing method according to claim 1 or 2, characterized in that the first intermediate image and the second intermediate image have more pixels than the low-resolution image.

4. The generator has an upsampling layer, The image processing method according to any one of claims 1 to 3, characterized in that the upsampling layer upsamples the input to the upsampling layer and outputs it.

5. The generator has one or more linear sum layers, The linear sum layer outputs a linear sum of the inputs to the linear sum layer and the weights of the linear sum layer. The image processing method according to claim 4, characterized in that the number of linear sum layers is less on the output side than the input side of the upsample layers included in the generator that are located furthest to the output side of the generator.

6. The generator generates the first intermediate image by summing the first residual component generated from the first feature map with the low-resolution image. The image processing method according to any one of claims 1 to 5, characterized in that a second intermediate image is generated by summing a second residual component generated from the first feature map with the low-resolution image.

7. The image processing method according to claim 6, characterized in that the low-resolution image is upsampled before summing so that the number of pixels matches the first residual component and the second residual component.

8. The aforementioned generator has multiple linear sum layers, The linear sum layer outputs a linear sum of the inputs to the linear sum layer and the weights of the linear sum layer. The image processing method according to any one of claims 1 to 7, characterized in that the output of at least one of the plurality of linear sum layers is connected in the channel direction to the input to the linear sum layer.

9. The aforementioned generator has multiple linear sum layers, The linear sum layer outputs a linear sum of the inputs to the linear sum layer and the weights of the linear sum layer. The image processing method according to any one of claims 1 to 8, characterized in that at least half of the outputs of the plurality of linear sum layers are coupled in the channel direction to the inputs of each of the linear sum layers.

10. The generator has multiple residual blocks, each having multiple linear sum layers. The linear sum layer outputs a linear sum of the inputs to the linear sum layer and the weights of the linear sum layer. The image processing method according to any one of claims 1 to 9, characterized in that the residual block outputs the sum of the input to the residual block and the result calculated by the residual block on the input.

11. The image processing method according to claim 10, characterized in that the residual block has two or more activation functions that perform nonlinear transformations.

12. The generator has multiple residual blocks, each having multiple linear sum layers. The linear sum layer outputs a linear sum of the inputs to the linear sum layer and the weights of the linear sum layer. The residual block outputs the sum of the input to the residual block and the result calculated by the residual block on the input. The image processing method according to any one of claims 1 to 11, characterized in that half or less of the multiple residual blocks have a batch regularization layer for performing batch regularization.

13. The image processing method according to any one of claims 1 to 12, characterized in that the generator does not have a batch regularization layer that performs batch regularization.

14. The image processing method according to any one of claims 1 to 13, characterized in that the estimated image is generated by a weighted average of the first intermediate image and the second intermediate image.

15. The generator is a machine learning model trained using losses based on a first loss and a second loss. The first loss is defined based on a comparison between the intermediate image generated by the generator and the ground truth image corresponding to the intermediate image. The second loss is defined based on the identification result of whether the intermediate image generated by the generator is an image generated by the generator, The image processing method according to any one of claims 1 to 14, characterized in that the weight of the second loss relative to the first loss in the loss defined using the first intermediate image is smaller than the weight of the second loss relative to the first loss in the loss defined using the second intermediate image.

16. A process of acquiring multiple low-resolution images and multiple high-resolution images, The process involves a generator converting the low-resolution image into a first feature map, and generating a first intermediate image with a higher resolution than the low-resolution image and a second intermediate image with a higher resolution than the low-resolution image from the first feature map. A step of using a classifier to determine whether the input image is an image generated by the generator, The process includes a step of training the generator based on a first loss based on the difference between the high-resolution image corresponding to the low-resolution image and the first intermediate image or the second intermediate image, and a second loss based on the identification output of the classifier when the first intermediate image or the second intermediate image is input. A learning method characterized in that the weight of the second loss relative to the first loss in the loss defined using the first intermediate image is smaller than the weight of the second loss relative to the first loss in the loss defined using the second intermediate image.

17. The learning method according to claim 16, characterized in that the weight of the second loss in the loss defined using the first intermediate image is 0.

18. The process involves converting an image into a feature map using a feature extractor. The learning method according to 16 or 17, characterized in that, in the step of learning the generator, the generator is learned based on a third loss based on the difference between a second feature map obtained by transforming the high-resolution image by the feature extractor and a third feature map obtained by transforming the second intermediate image.

19. The learning method according to any one of claims 16 to 18, characterized in that the second loss is based on a comparison between a value based on each of the multiple identification outputs of the classifier when a plurality of the high-resolution images are input and the identification output of the classifier when the first intermediate image or the second intermediate image is input.

20. It has a processing means for generating an estimated image by increasing the resolution of an input low-resolution image, The processing means is The process involves using a machine learning model generator to convert an input low-resolution image into a first feature map, and generating a first intermediate image with a higher resolution than the low-resolution image and a second intermediate image with a higher resolution than the low-resolution image from the first feature map, An image processing apparatus characterized by performing the steps of: generating an estimated image with a higher resolution than the low-resolution image based on the first intermediate image and the second intermediate image.

21. A program characterized by causing a computer to execute the image processing method described in any one of claims 1 to 15.

22. It has a processing means for training a machine learning model, The processing means is A process of acquiring multiple low-resolution images and multiple high-resolution images, A generator converts the low-resolution image into a first feature map, and generates a first intermediate image with a higher resolution than the low-resolution image and a second intermediate image with a higher resolution than the low-resolution image from the first feature map. A means for identifying whether the image input to the classifier is an image generated by the generator, The process involves training the generator based on a first loss based on the difference between the high-resolution image corresponding to the low-resolution image and the first intermediate image or the second intermediate image, and a second loss based on the identification output of the classifier when the first intermediate image or the second intermediate image is input. A learning device characterized in that the weight of the second loss relative to the first loss in the loss defined using the first intermediate image is smaller than the weight of the second loss relative to the first loss in the loss defined using the second intermediate image.

23. A program characterized by causing a computer to execute the learning method described in any one of claims 16 to 19.

Images