IMAGE PROCESSING METHOD, IMAGE PROCESSING APPARATUS, IMAGE PROCESSING PROGRAM, LEARNING METHOD, LEARNING APPARATUS, LEARNING PROGRAM
By generating two intermediate high-resolution images with distinct loss functions, the method addresses edge multiplexing and false structures in image resolution enhancement, resulting in high-quality images with controlled resolution and reduced calculation load.
Patent Information
- Application Number
- JP2020152151
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-09-10
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-09-10
AI Technical Summary
Existing image resolution enhancement methods using machine learning models, such as SRGAN, suffer from issues like edge multiplexing and color changes, and the appearance of false structures in generated high-resolution images.
A method involving a generator that produces two intermediate high-resolution images using different loss functions, one focusing on texture and the other on natural appearance, combined to generate a final high-resolution image that suppresses image quality deterioration and reduces calculation time.
The method produces high-quality images with improved resolution and reduced false structures, while minimizing calculation load and image quality issues.
Smart Images

Figure 0007580986000001 
Figure 0007580986000002 
Figure 0007580986000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for increasing the resolution of an image using a machine learning model. [Background technology]
[0002] Patent Document 1 discloses a method for increasing the resolution of an image using a machine learning model called a Generative Adversarial Network (GAN). This method is called a Super Resolution GAN (SRGAN). SRGAN performs learning using a generator that generates a high-resolution image and a classifier that identifies whether an input image is an image generated by the generator or an actual high-resolution image. Here, the actual high-resolution image means a high-resolution image that is not generated by the generator.
[0003] The generator learns weights so that the generated high-resolution image cannot be distinguished from the actual high-resolution image by the classifier. This allows the generation of high-resolution images with high-resolution textures that look more natural. However, at the same time, there is a problem in that false structures that are subjectively unnatural may appear.
[0004] In response to this, Non-Patent Document 1 discloses a method for controlling the appearance of false structures and the sense of resolution. In Non-Patent Document 1, a weighted average is taken between the weights of a generator that has learned to increase resolution without using a classifier (few false structures but low sense of resolution) and the weights of a generator that has learned using a classifier (equivalent to SRGAN, high sense of resolution but possible presence of false structures). A high-resolution image is generated by a generator using this weighted average. This method is called network interpolation. By changing the weight of the weighted average, it is possible to control the balance between the appearance of false structures and the sense of resolution. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] US2018 / 0075581 [Non-patent literature]
[0006] [Non-Patent Document 1] Xintao Wang,et al.,Deep Network Interpolation for Continuous Imagery Effect Transition,Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR),2019,pp.1692-1701 Summary of the Invention [Problem to be solved by the invention]
[0007] However, the inventors' research has revealed that the generator that performs the network interpolation described in Non-Patent Document 1 may generate high-resolution images with degradation in image quality, such as multiplexing of edges that makes the subject appear double and color changes.
[0008] Therefore, an object of the present invention is to provide a high-quality image when increasing the resolution of an image using a machine learning model. [Means for solving the problem]
[0009] The present invention In one aspect The image processing method involves inputting a low-resolution image into a generator, which is a machine learning model, to convert it into a first feature map. And a process of The above Based on the first feature map The generator generating a first intermediate image having a higher resolution than the low resolution image and a second intermediate image having a higher resolution than the low resolution image; generating an estimated image having a higher resolution than the low-resolution image based on the first intermediate image and the second intermediate image. death, the generator includes one or more upsample layers and one or more linear sum layers; the upsample layer generates an output by upsampling an input to the upsample layer; the linear sum layer outputs a linear sum of the inputs to the linear sum layer and the weights in the linear sum layer; The number of linear sum layers on the output side of the upsample layer that is located on the most output side among the one or more upsample layers is less than the number of linear sum layers on the input side of the upsample layer. It is characterized by: Effect of the Invention
[0010] According to the present invention, it is possible to provide a high-quality image by increasing the resolution of an image using a machine learning model. [Brief description of the drawings]
[0011] [Figure 1] FIG. 2 is a diagram showing a procedure for increasing the resolution in the first embodiment. [Diagram 2] 1 is a block diagram of an image processing system according to a first embodiment. [Diagram 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4] 1 is a flowchart of learning of a machine learning model in the first embodiment. [Diagram 5] FIG. 2 is a schematic diagram of a subnet in the first and second embodiments. [Figure 6] 4 is a flowchart of generation of an estimated high-resolution image in the first embodiment. [Figure 7] FIG. 11 is a block diagram of an image processing system according to a second embodiment. [Figure 8] FIG. 11 is an external view of an image processing system according to a second embodiment. [Figure 9] 11 is a flowchart of learning of a machine learning model in the second embodiment. [Figure 10] FIG. 11 is a diagram showing a procedure for increasing the resolution in the second embodiment. [Figure 11] FIG. 11 is a schematic diagram of a subnet in the second embodiment. [Figure 12] 10 is a flowchart of generation of an estimated high-resolution image in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] Hereinafter, a system including a processing apparatus according to the present invention will be described with reference to the accompanying drawings. In each drawing, the same components are designated by the same reference numerals, and duplicated explanations will be omitted.
[0013] [Example 1] First, the gist of the present invention will be described before describing in detail Example 1. Example 1 described below is one of the preferred embodiments of the present invention, and not all of them are necessary for realizing the present invention.
[0014] In this embodiment, a generator, which is a machine learning model, converts a low-resolution image into a feature map (first feature map), and further generates two intermediate images (first intermediate image and second intermediate image) having a higher resolution than the low-resolution image from the first feature map. Hereinafter, the first intermediate image and the second intermediate image are also referred to as the first intermediate high-resolution image and the second intermediate high-resolution image, respectively.
[0015] The generator is trained using different loss functions for the two intermediate high-resolution images. The loss functions have a first loss based on the difference between the intermediate high-resolution image and the correct high-resolution image, and a second loss defined based on the classification output of a classifier that identifies whether the input image is an image generated by the generator.
[0016] The first intermediate high-resolution image is generated using the results of learning with a loss function in which the weight of the second loss is smaller than that of the first loss than that of the second intermediate high-resolution image. For example, the learning for generating the first intermediate high-resolution image uses only the first loss as a loss function.
[0017] The second intermediate high-resolution image may be generated using a result trained using a weighted sum of the first loss and the second loss as a loss function, such that the second intermediate high-resolution image has high-resolution texture equivalent to that trained by the SRGAN, but may also contain spurious structures.
[0018] On the other hand, the learning method for generating the first intermediate high-resolution image is not GAN (or its contribution is weaker than that of the second intermediate high-resolution image), so the image has both high-resolution texture and reduced false structure. By combining these two intermediate high-resolution images (for example, by weighted averaging), a high-resolution image (estimated image) can be generated in which the sense of resolution and the appearance of false structure are adjusted. The estimated image has a higher resolution than the low-resolution image. Hereinafter, the estimated image is also referred to as the estimated high-resolution image.
[0019] This method combines high-resolution images, which are the target of the loss function, instead of the weight of the generator as in network interpolation, so it can suppress image quality degradation such as edge multiplexing and color changes. In addition, since one generator simultaneously generates two intermediate high-resolution images, it can also suppress increases in calculation time.
[0020] In the following, the stage of determining the weights of the generator and discriminator, which are machine learning models, based on the training dataset, is called "learning," and the stage of generating an estimated high-resolution image from a low-resolution image by the generator using the trained weights is called "estimation." Machine learning models include, for example, neural networks, genetic programming, and Bayesian networks. Neural networks include CNNs (Convolutional Neural Networks), GANs (Generative Adversarial Networks), and RNNs (Recurrent Neural Networks).
[0021] Next, the image processing system according to the first embodiment will be described.
[0022] 2 and 3 are a block diagram and an external view of the image processing system 100, respectively.
[0023] The image processing system 100 includes a learning device 101, a resolution improvement device 102, and a control device 103, which are connected to each other via a wired or wireless network.
[0024] The control device 103 has a storage unit 131, a communication unit 132, and a display unit 133, and transmits a request to the resolution increasing device 102 to increase the resolution of a low-resolution image in accordance with an instruction from a user.
[0025] The high-resolution device 102 has a storage unit 121, a communication unit 122, an acquisition unit 123, and a high-resolution unit 124, and performs high-resolution processing on a low-resolution image using a generator, which is a trained machine learning model, to generate an estimated high-resolution image. The acquisition unit 123 and the high-resolution unit 124 can implement their functions using one or more processors (processing means), such as CPUs. The high-resolution device 102 acquires information on the weights of the generator from the learning device 101 and stores it in the storage unit 121.
[0026] The learning device 101 has a memory unit 111, an acquisition unit 112, a calculation unit 113, and an update unit 114, and learns the weights of a generator. The functions of the acquisition unit 112, the calculation unit 113, and the update unit 114 can be implemented by one or more processors (processing means) such as CPUs.
[0027] In the image processing system 100 configured in this manner, the control device 103 acquires the estimated high-resolution image generated by the resolution increasing device 102 , and presents the result to the user via the display unit 133 .
[0028] Next, weight learning executed by learning device 101 will be described with reference to the flowchart of FIG.
[0029] The learning in the first embodiment is composed of two stages: a first learning step that does not use a classifier, and a second learning step (GAN) that uses a classifier. The learning device 101 has a storage unit 111, an acquisition unit 112, a calculation unit 113, and an update unit 114, and each step is executed by any of these units. Note that the functions (methods) of each flowchart described below can also be realized as a program that causes a computer to execute the functions (methods).
[0030] In step S101, the acquisition unit 112 acquires one or more pairs of high-resolution images and low-resolution images from the storage unit 111. A learning data set consisting of a plurality of high-resolution images and low-resolution images is stored in the storage unit 111. The corresponding low-resolution image and high-resolution image have the same object (subject) in the image. The low-resolution image may be generated by downsampling the high-resolution image. In the first embodiment, the number of pixels of the high-resolution image is 16 times that of the low-resolution image (four times horizontally and vertically). However, the relationship of the number of pixels is not limited to this. The image may be either color or grayscale. Furthermore, degradation other than downsampling (such as JPEG compression noise) may be added to the low-resolution image. This allows the generator to be endowed with a function of correcting image degradation in addition to a function of increasing resolution.
[0031] In step S102, the calculation unit 113 inputs the low-resolution image to a generator to generate first and second intermediate high-resolution images. The generator is, for example, a Convolutional Neural Network (CNN), and in the first embodiment, a model having a configuration as shown in FIG. 1 is used. However, the invention is not limited to this.
[0032] The initial values of the weights of the generator may be generated using random numbers or the like. In the generator shown in Fig. 1, an input low-resolution image 201 is converted into a first feature map 202 by being processed in a subnetwork (sub-network) 211. The subnetwork 211 has one or more linear sum layers. The linear sum layer has a function of taking a linear sum between the input to the linear sum layer and the weights of the linear sum layer. The linear sum layer is, for example, a convolution layer, a deconvolution layer, a full connection layer, or the like.
[0033] The subnetwork 211 also has one or more activation functions that are nonlinear transformations. The activation functions include ReLU (Rectified Linear Unit), a sigmoid function, and a hyperbolic tangent function.
[0034] In the first embodiment, the low-resolution image 201 has fewer pixels than the corresponding high-resolution image. Therefore, in the first embodiment, the subnet 211 has an upsampling layer that increases the number of pixels in the horizontal and vertical directions. That is, the upsampling layer is a layer that has a function of performing upsampling on the input to the upsampling layer. The upsampling layer in the first embodiment has a function of performing sub-pixel convolution (also called Pixel Shuffler). The upsampling layer is not limited to this, and the upsampling function may be realized by deconvolution, bilinear interpolation, nearest neighbor interpolation, or the like. However, when sub-pixel convolution is used, the effect of the final high resolution can be improved because the degree of freedom due to the reduction of the influence of zero padding and the convolution with weights is increased compared to other methods.
[0035] In the first embodiment, the subnetwork 211 has the configuration shown in FIG.
[0036] "Conv." stands for convolution, "sum" stands for pixel-by-pixel sum, and "sub-pixel conv." stands for sub-pixel convolution.
[0037] Moreover, "residual block" refers to a residual block. The configuration of the residual block in the first embodiment is shown in FIG.
[0038] A residual block is a block that is composed of multiple linear sum layers and activation functions. The residual block is configured to take the sum of the input to the residual block and the result of a series of operations within the residual block, and output it. "Concatenation" refers to concatenation in the channel direction.
[0039] In the first embodiment, the subnet 211 has eight residual blocks. However, the number of residual blocks is not limited to this. If it is desired to further improve the performance of the generator, it is advisable to increase the number of residual blocks.
[0040] In the first embodiment, there are multiple upsampling layers (subpixel convolution). In the first embodiment, in order to upsample the number of pixels of a low-resolution image by 16 times, subpixel convolution is performed twice to upsample by 4 times. If high-magnification upsampling is performed in one upsampling layer, a lattice pattern or the like is likely to occur in the high-resolution image, so it is desirable to perform low-magnification upsampling multiple times in this way. Note that in the first embodiment, an example is described in which the upsampling layer is in subnet 211, but the present invention is not limited to this. The upsampling layer may be present in subnet 212 and subnet 213 instead of subnet 211.
[0041] The first feature map 202 is input to a subnetwork 212 to generate a first residual component 203. The first feature map 202 is also input to a subnetwork 213 to generate a second residual component 204.
[0042] Each of the subnet 212 and the subnet 213 has one or more linear sum layers. In the first embodiment, each of the subnet 212 and the subnet 213 is configured with one convolution layer. The subnet 212 and the subnet 213 can be combined into one linear sum layer. For example, by doubling the number of filters in the convolution layer, an output having twice the number of channels of the low-resolution image 201 (3 in the case of color) can be obtained. This output can be divided into two in the channel direction to obtain a first residual component 203 and a second residual component 204.
[0043] The first residual component 203 is added to the low-resolution image 201 to generate a first intermediate high-resolution image 205. The second residual component 204 is added to the low-resolution image 201 to generate a second intermediate high-resolution image 206. The low-resolution image 201 is upsampled before the summation so that the number of pixels of the first residual component 203 and the second residual component 204 match. This upsampling may be performed by bilinear or bicubic interpolation, or a deconvolution layer may be used. By estimating the residual component instead of the high-resolution image itself, image quality degradation such as color change from the low-resolution image 201 is less likely to occur.
[0044] Note that the low-resolution image 201 may be upsampled in advance using bicubic interpolation or the like so that the number of pixels matches that of the high-resolution image, and then input to the generator. In this case, an upsampling layer is not required in the generator. However, as the number of pixels in the horizontal and vertical directions of the low-resolution image 201 increases, the number of times linear sums are calculated increases, and the calculation load increases. For this reason, it is desirable to input the low-resolution image 201 to the generator without upsampling as in the first embodiment, and to upsample it within the generator.
[0045] In step S103, the update unit 114 updates the weights of the generator based on the first loss. The first loss in the first embodiment is a loss defined based on the difference between a high-resolution image (correct image) corresponding to the low-resolution image 201 and an intermediate high-resolution image. In the first embodiment, MSE (Mean Square Error) is used, but MAE (Mean Absolute Error) or the like may also be used.
[0046] In the first embodiment, the sum of the MSE between the first intermediate high-resolution image 205 and the high-resolution image and the MSE between the second intermediate high-resolution image 206 and the high-resolution image is used as the loss function, and the weights of the generator are updated by backpropagation.
[0047] In step S104, the update unit 114 determines whether the first learning is complete. Completion can be determined by, for example, whether the number of iterations of learning (updating the weight) has reached a preset number, or whether the amount of change in the weight at the time of updating is smaller than a preset value. If it is determined in step S104 that the weight learning is not complete, the process returns to step S101, and the acquisition unit 112 acquires one or more pairs of new low-resolution images 201 and high-resolution images. On the other hand, if it is determined that the first learning is complete, the process proceeds to step S105, and the second learning is started.
[0048] In step S105 , the acquisition unit 112 acquires one or more pairs of a high-resolution image and a low-resolution image 201 from the storage unit 111 .
[0049] In step S 106 , the calculation unit 113 inputs the low-resolution image 201 to a generator to generate a first intermediate high-resolution image 205 and a second intermediate high-resolution image 206 .
[0050] In step S107, the calculation unit 113 inputs the second intermediate high-resolution image 206 and the high-resolution image to a classifier to generate a classification output. The classifier classifies whether the input image is a high-resolution image generated by a generator or an actual high-resolution image. The classifier may be a CNN or the like. The initial values of the weights of the classifier are determined by random numbers or the like. Note that the high-resolution image input to the classifier may be any actual high-resolution image, and does not need to be an image corresponding to the low-resolution image 201.
[0051] In step S108, the update unit 114 updates the weights of the classifier based on the classification output and the correct label. In the first embodiment, the correct label for the second intermediate high resolution image 206 is set to 0, and the correct label for the actual high resolution image is set to 1. Although sigmoid cross entropy is used as the loss function, other functions may be used.
[0052] In step S109, the update unit 114 updates the weights of the generator based on the first loss and the second loss. For the first intermediate high-resolution image 205, only the first loss (MSE with the corresponding high-resolution image) is taken. For the second intermediate high-resolution image 206, a weighted sum of the first loss and the second loss is taken. The second loss is the sigmoid cross entropy between the classification output when the second intermediate high-resolution image 206 is input to the classifier and the correct label 1. Since the generator is to learn so that the classifier erroneously determines that the second intermediate high-resolution image 206 is an actual high-resolution image, the correct label is set to 1 (corresponding to the actual high-resolution image). The sum of the losses of the first intermediate high-resolution image 205 and the second intermediate high-resolution image 206 is set as the loss function of the generator.
[0053] By repeatedly updating the weights using this loss function, a high-resolution image with a natural appearance and high-resolution texture that may cause the classifier to make a false judgment is generated on the second intermediate high-resolution image 206 side. However, as a drawback, false structures may appear. On the other hand, on the first intermediate high-resolution image 205 side, both the high-resolution texture and false structures are suppressed, and a high-resolution image with fewer high-frequency components than the second intermediate high-resolution image 206 is output.
[0054] In the first embodiment, only the first loss is used for the first intermediate high resolution image 205, but the second loss may also be used. In this case, the weight of the second loss in the first loss should be smaller for the first intermediate high resolution image 205 than for the second intermediate high resolution image 206. The order of steps S108 and S109 may be reversed.
[0055] In step S110, the update unit 114 determines whether the second learning is complete. If it is determined that the second learning is incomplete, the process returns to step S105 to acquire one or more new pairs of low-resolution image 201 and high-resolution image. If the second learning is complete, the weight information is stored in the storage unit 111. Note that since only the generator is used during estimation, the weight of only the generator may be stored.
[0056] Next, the estimation (generation of an estimated high-resolution image) performed by resolution enhancement device 102 and control device 103 will be described with reference to the flowchart in Fig. 6. Resolution enhancement device 102 has a storage unit 121, a communication unit 122, an acquisition unit 123, and a resolution enhancement unit 124, and control device 103 has a storage unit 131, a communication unit 132, and a display unit 133, and each step is performed by any of these units.
[0057] In step S201, the communication unit 132 of the control device 103 transmits a request for executing resolution increase to the resolution increase device 102. The request for execution also includes information specifying the low-resolution image 201 to be increased in resolution. Alternatively, the low-resolution image 201 itself to be increased in resolution may be transmitted together with the request for execution of the process.
[0058] In step S202 , the communication unit 122 of the resolution enhancement device 102 acquires the execution request sent from the control device 103 .
[0059] In step S203, the acquisition unit 123 acquires information on the weights of the generator and the low-resolution image 201 to be subjected to resolution enhancement from the storage unit 121. The low-resolution image 201 may be acquired from another storage device connected via a wired or wireless connection.
[0060] In step S204, the resolution increasing unit 124 generates a first intermediate high-resolution image 205 and a second intermediate high-resolution image 206 from the low-resolution image 201 using the generator shown in FIG. 1. The second intermediate high-resolution image 206 is a high-resolution image that has a high-resolution texture and looks natural, but may contain false structures. On the other hand, the first intermediate high-resolution image 205 is a high-resolution image in which both the high-resolution texture and false structures are suppressed. The first intermediate high-resolution image 205 has fewer high-frequency components than the second intermediate high-resolution image 206.
[0061] In step S205, the resolution increasing unit 124 generates an estimated high resolution image 207 based on the first intermediate high resolution image 205 and the second intermediate high resolution image 206. In the first embodiment, the estimated high resolution image 207 is generated by a weighted average of the first intermediate high resolution image 205 and the second intermediate high resolution image 206. Note that the generation of the high resolution image 207 is not limited to the weighted average of the first intermediate high resolution image 205 and the second intermediate high resolution image 206, and the high resolution image 207 may be generated by replacing a partial region of the second intermediate high resolution image 206 with the first intermediate high resolution image 205, for example.
[0062] In step S206, the communication unit 122 transmits the estimated high-resolution image 207 to the control device 103.
[0063] In step S207, the communication unit 132 of the control device 103 acquires an estimated high-resolution image 207. The acquired estimated high-resolution image 207 is stored in the storage unit 131 or displayed on the display unit 133. Alternatively, the estimated high-resolution image 207 may be stored in another storage device connected from the control device 103 or the resolution increasing device 102 via a wired or wireless connection. The control device 103 may be configured to acquire the first intermediate high-resolution image 205 and the second intermediate high-resolution image 206, and the estimated high-resolution image 207 may be generated by the control device 103. In this case, the user can adjust the sense of resolution and the appearance of false structures while checking the actual image on the display unit 133.
[0064] Next, the conditions desirable for obtaining the effects of the present invention will be described.
[0065] It is desirable that the number of linear sum layers in the generator is smaller on the output side than on the input side of the upsampling layer located closest to the output side. If upsampling is performed early in the generator's calculations, the number of subsequent linear sum calculations increases, increasing the calculation load. In the first embodiment, there are 40 or more linear sum layers on the input side of the upsampling layer located closest to the output side, but there is only one linear sum layer for each of the first residual component 203 and the second residual component 204 on the output side.
[0066] In addition, the generator is configured with multiple linear sum layers, and it is preferable that the output of at least one of the multiple linear sum layers is concatenated with the input to the linear sum layer in the channel direction. This means the concatenation represented by "concatenation" in Fig. 5(b), for example. This allows the generator to transmit more feature maps after the layer, improving the accuracy of the first intermediate high-resolution image 205 and the second intermediate high-resolution image 206.
[0067] Furthermore, it is preferable that at least half of the linear sum layers in the generator are connected to the inputs to the linear sum layers in the channel direction, thereby further improving the accuracy of the resolution enhancement.
[0068] In addition, it is preferable that each of the multiple residual blocks in the generator has three or more linear sum layers. This can improve the accuracy of high-resolution. Furthermore, it is preferable that the residual block has two or more activation functions. This increases the nonlinear effect and improves the accuracy of high-resolution.
[0069] In addition, it is desirable that half or less of the multiple residual blocks in the generator have a batch regularization layer that performs batch regularization. Unlike recognition tasks, in regression tasks that estimate images from images, the effect of improving accuracy by batch regularization is small. Therefore, it is desirable to reduce the number of batch regularizations in order to reduce the computational load. If it is desired to further reduce the computational load, the generator may be configured not to have a batch regularization layer.
[0070] In addition, in training the generator, a trained feature extractor that converts an image into a feature map may be used. The feature extractor converts the correct high-resolution image corresponding to the low-resolution image 201 into a second feature map, and converts the second intermediate high-resolution image 206 into a third feature map. The generator may be trained by adding a third loss based on the difference between the second and third feature maps (e.g., MSE) to the loss function. This allows the loss function to take abstract features into account, making the high-resolution image look more natural.
[0071] Moreover, the second loss is preferably based on a comparison between a value based on each of the classification outputs of the classifier for a plurality of actual high-resolution images and the classification output of the first intermediate high-resolution image 205 or the second intermediate high-resolution image 206. This is a method called Relativistic GAN. The value based on each of the classification outputs of the classifier for a plurality of actual high-resolution images may be the average value or median value of the plurality of classification outputs. For example, sigmoid cross entropy is taken so that the difference between the classification output of the first intermediate high-resolution image 205 or the second intermediate high-resolution image 206 and the average value of the classification output of the actual high-resolution image indicates the correct label (here, 1). This allows learning from a relative perspective of whether the high-resolution image generated by the generator is more realistic than the set of actual high-resolution images. In conventional GANs, there was a problem that the actual high-resolution images were ignored during learning, and realism was learned only from the high-resolution images generated by the generator. However, Relativistic GAN can avoid this problem and improve the stability of learning.
[0072] With the above configuration, it is possible to provide an image processing system that can control the sense of resolution and the appearance of false structures while suppressing an increase in the calculation load and deterioration of image quality when increasing the resolution of an image using a machine learning model. In other words, it is possible to provide a high-quality image when increasing the resolution of an image using a machine learning model.
[0073] [Example 2] Second Embodiment An image processing system according to a second embodiment of the present invention will be described.
[0074] 7 and 8 are a block diagram and an external view of the image processing system 300, respectively. The image processing system 300 includes a learning device 301 and an imaging device 302. The imaging device 302 includes an optical system 321, an imaging element 322, an image processing unit 323, a storage unit 324, a communication unit 325, a display unit 326, and a system controller 327. The optical system 321 collects light incident from the subject space to form a subject image. The optical system 321 has functions such as zooming, aperture adjustment, and autofocus as necessary. The imaging element 322 converts the subject image into an electrical signal by photoelectric conversion to generate a captured image. The imaging element 322 is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor. The captured image is obtained as a live view of the subject space before shooting or when the user presses the release button, and is subjected to a predetermined process by the image processing unit 323 and displayed on the display unit 326.
[0075] When a user instructs digital zoom and presses the release button during shooting, the captured image (low-resolution image) is made high-resolution by a generator, which is a machine learning model, in the image processing unit 323. At this time, weights learned by the learning device 301 are used. Information on the weights is read out in advance from the learning device 301 via the communication unit 325 and stored in the storage unit 324. Details regarding the learning and estimation of the generator will be described later.
[0076] In addition, in the live view when the user instructs digital zoom, an image upsampled by a high-speed method such as bilinear interpolation is displayed on the display unit 326. The captured image with high resolution by the generator (estimated high-resolution image) is stored in the storage unit 324 and displayed on the display unit 326. The above operations are controlled by the system controller 327.
[0077] Although FIG. 8 shows a so-called single-lens camera as the imaging device 302, the imaging device 302 may be a device such as a smartphone.
[0078] Next, the learning of the weights of the generator executed by the learning device 301 will be described with reference to the flowchart in Fig. 9. The learning device 301 has a storage unit 311, an acquisition unit 312, a calculation unit 313, and an update unit 314, and each step is executed by any of these units.
[0079] In step S301, the acquisition unit 312 acquires one or more pairs of a low-resolution image and a high-resolution image from the storage unit 311. In the second embodiment, the number of pixels of the high-resolution image is 16 times that of the low-resolution image, but is not limited to this.
[0080] In step S302, the calculation unit 313 inputs the low-resolution image to the generator, and generates a first intermediate high-resolution image and a second intermediate high-resolution image. In the second embodiment, the generator has the configuration shown in FIG. 10. The subnet 411 converts the low-resolution image 401 into a first feature map 402, and from the first feature map 402, the subnet 412 generates the first intermediate high-resolution image 403, and the subnet 413 generates the second intermediate high-resolution image 404. The subnet 411 has the configuration shown in FIG. 5(A), and the residual block has the configuration shown in FIG. 11. The subnet 411 has four residual blocks. The subnet 412 and the subnet 413 each have one convolution layer. However, the configuration of each subnet is not limited to this.
[0081] In step S303, the calculation unit 313 inputs the high-resolution image and the second intermediate high-resolution image 404 to a classifier, and generates a classification output.
[0082] In step S304, the update unit 314 updates the weights of the classifier based on the classification output and the correct label.
[0083] In step S305, the update unit 314 updates the weights of the generator based on the first loss and the second loss. The first loss and the second loss are the same as those described in the first embodiment.
[0084] In step S306, the update unit 314 determines whether the learning of the generator is complete. If it is determined that the learning of the weights is not complete, the process returns to step S301. If it is determined that the learning is complete, the learning is terminated and the weight information is stored in the storage unit 311.
[0085] 12, a description will be given of the resolution enhancement of a digitally zoomed captured image, which is executed by the image processing unit 323. The image processing unit 323 has an acquisition unit 323a, a resolution enhancement unit 323b, and a calculation unit 323c, and each step is executed by any of these units.
[0086] In step S401, the acquisition unit 323a extracts a partial region (low-resolution image 401) from a captured image. Since the captured image has information of all pixels acquired by the image sensor 322, only a partial region necessary for digital zoom is extracted.
[0087] In step S402, the acquisition unit 323a acquires information on the weights of the generator from the storage unit 324. Note that the order of steps S401 and S402 does not matter.
[0088] In step S403, the resolution increasing unit 323b inputs a partial region of the captured image (low resolution image 401) to a generator, and generates a first intermediate high resolution image 403 and a second intermediate high resolution image 404.
[0089] In step S404, the calculation unit 323c generates an estimated high-resolution image 405 by taking a weighted average of the first intermediate high-resolution image 403 and the second intermediate high-resolution image 404.
[0090] In step S405, the calculation unit 323c scales (upsamples or downsamples) the estimated high-resolution image 405 to a specified number of pixels. The generator has been trained to upsample the number of pixels by 16 times, so it is necessary to match it to the digital zoom magnification specified by the user. Bicubic interpolation or the like is used for downsampling, and anti-aliasing processing may be performed as necessary. If a digital zoom of a magnification exceeding 4 times in one dimension is specified, the estimated high-resolution image 405 is upsampled by bicubic interpolation or the like. Alternatively, the estimated high-resolution image 405 may be input to the generator as a new low-resolution image 401.
[0091] With the above configuration, it is possible to provide an image processing system that can control the sense of resolution and the appearance of false structures while suppressing an increase in the calculation load and deterioration of image quality when increasing the resolution of an image using a machine learning model. In other words, it is possible to provide a high-quality image when increasing the resolution of an image using a machine learning model.
[0092] (Other Examples) In the above embodiments, the first loss is MSE or MAE, and the second loss is a classification result by a classifier. However, the present invention is not limited to this. The effect of the present invention can be obtained by generating a first intermediate image and a second intermediate image having different characteristics from each other from a first feature map based on a low-resolution image, and generating an estimated image based on the first intermediate image and the second intermediate image. This is because it is possible to cover the adverse effects that occur in either the first intermediate image or the second intermediate image with the other.
[0093] The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-mentioned embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.
[0094] According to each embodiment, it is possible to provide an image processing device, an imaging device, an image processing method, an image processing program, and a storage medium capable of generating high-quality images by increasing the resolution of an image using a machine learning model.
[0095] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various combinations, modifications, and changes are possible within the scope of the gist of the present invention. For example, in the above-mentioned embodiment, the example of setting physical information or exercise information as the target information has been described, but the present invention is not limited to this. The amount of studying or reading may be set as the target information. [Explanation of symbols]
[0096] 200 low resolution images 202 First feature map 205 First intermediate image 206 Second intermediate image 207 Estimated Image
Claims
1. converting the low-resolution image into a first feature map by inputting the low-resolution image into a generator that is a machine learning model; generating, by the generator, a first intermediate image having a higher resolution than the low resolution image and a second intermediate image having a higher resolution than the low resolution image based on the first feature map; generating an estimated image having a higher resolution than the low-resolution image based on the first intermediate image and the second intermediate image; the generator includes one or more upsample layers and one or more linear sum layers; the upsample layer generates an output by upsampling an input to the upsample layer; the linear sum layer outputs a linear sum of the inputs to the linear sum layer and the weights in the linear sum layer; An image processing method, characterized in that the number of linear sum layers on the output side of the upsample layer located closest to the output side among the one or more upsample layers is less than the number of linear sum layers on the input side of the upsample layer.
2. A step of converting a low-resolution image into a first feature map by inputting the low-resolution image into a generator that is a machine learning model; generating, by the generator, a first intermediate image having a higher resolution than the low resolution image and a second intermediate image having a higher resolution than the low resolution image based on the first feature map; generating an estimated image having a higher resolution than the low-resolution image based on the first intermediate image and the second intermediate image; the generator includes a plurality of residual blocks each including a plurality of linear sum layers; 11. An image processing method according to claim 10, wherein the residual block outputs a sum of an input to the residual block and a result of the input being calculated by the residual block.
3. A step of converting a low-resolution image into a first feature map by inputting the low-resolution image into a generator that is a machine learning model; generating, by the generator, a first intermediate image having a higher resolution than the low resolution image and a second intermediate image having a higher resolution than the low resolution image based on the first feature map; generating an estimated image having a higher resolution than the low-resolution image by taking a weighted average of the first intermediate image and the second intermediate image.
4. 4. The image processing method according to claim 1, wherein the first intermediate image has fewer high frequency components than the second intermediate image.
5. 5. The image processing method according to claim 1, wherein the first intermediate image and the second intermediate image have a larger number of pixels than the low-resolution image.
6. The generator includes: generating the first intermediate image by taking a sum of a first residual component generated based on the first feature map and the low-resolution image; The image processing method according to any one of claims 1 to 5, characterized in that the second intermediate image is generated by taking a sum of a second residual component generated based on the first feature map and the low-resolution image.
7. 7. The image processing method according to claim 6, wherein the low-resolution image is upsampled before summing so that the number of pixels of the first residual component and the second residual component match.
8. the generator having one or more upsample layers; 8. A method according to any preceding claim, wherein the upsample layer generates an output by upsampling an input to the upsample layer.
9. The generator includes one or more linear sum layers; the linear sum layer outputs a linear sum of the inputs to the linear sum layer and the weights in the linear sum layer; 9. The image processing method according to claim 8, wherein the number of linear sum layers on the output side of the upsampling layer located closest to the output side among the one or more upsampling layers is less than the number of linear sum layers on the input side of the upsampling layer.
10. 10. The image processing method of claim 9, wherein an input to the linear sum layer and a calculation result obtained based on the linear sum and the activation function are connected in a channel direction.
11. the generator has a plurality of linear sum layers including the linear sum layer; 11. The image processing method according to claim 9, wherein at least half of the inputs of the plurality of linear sum layers are connected to the calculation results obtained based on the linear sums and activation functions corresponding to the inputs in a channel direction.
12. the generator includes a plurality of residual blocks each including a plurality of linear sum layers including the linear sum layer; 12. The image processing method according to claim 9, wherein the residual block outputs a sum of an input to the residual block and a result of the input being calculated by the residual block.
13. 13. The image processing method according to claim 12, wherein the residual block includes two or more activation functions that perform a nonlinear transformation.
14. 14. The image processing method according to claim 12, wherein the number of residual blocks including a layer that performs batch regularization is less than half of the plurality of residual blocks.
15. 15. The image processing method according to claim 1, wherein the generator does not have a layer that performs batch regularization.
16. 15. The image processing method according to claim 1, wherein the estimated image is generated by a weighted average of the first intermediate image and the second intermediate image.
17. The generator is a machine learning model trained using a loss based on a first loss and a second loss, The first loss is defined based on an intermediate image generated by the generator and a ground truth image corresponding to the intermediate image; The second loss is defined based on a result of identifying whether the intermediate image generated by the generator is an image generated by the generator; 17. The image processing method according to claim 1, wherein a weight of the second loss relative to the first loss in the loss defined using the first intermediate image is smaller than a weight of the second loss relative to the first loss in the loss defined using the second intermediate image.
18. acquiring a plurality of low resolution images and a plurality of high resolution images; A step of converting the low-resolution image into a first feature map by inputting the low-resolution image into a generator that is a machine learning model, and generating a first intermediate image having a higher resolution than the low-resolution image and a second intermediate image having a higher resolution than the low-resolution image based on the first feature map; A step of identifying whether an image input to a classifier is an image generated by the generator; and learning the generator based on a first loss based on a difference between the high-resolution image corresponding to the low-resolution image and the first intermediate image or the second intermediate image, and a second loss based on a classification output of the classifier when the first intermediate image or the second intermediate image is input, A learning method, characterized in that a weight of the second loss with respect to the first loss in a loss defined using the first intermediate image is smaller than a weight of the second loss with respect to the first loss in a loss defined using the second intermediate image.
19. The learning method according to claim 18 , wherein a weight of the second loss in a loss defined using the first intermediate image is 0.
20. converting the image into a feature map using a feature extractor; The learning method according to claim 18 or 19, characterized in that in the step of training the generator, the generator is trained based on a third loss based on a difference between a second feature map obtained by converting the high-resolution image by the feature extractor and a third feature map obtained by converting the second intermediate image.
21. 21. The learning method according to claim 18, wherein the second loss is obtained based on a value based on each of a plurality of classification outputs of the classifier when a plurality of the high-resolution images are input, and based on a classification output of the classifier when the first intermediate image or the second intermediate image is input.
22. A processing means for generating an estimated image by increasing the resolution of an input low-resolution image, The processing means includes: A process of converting a low-resolution image into a first feature map by inputting the low-resolution image into a generator that is a machine learning model, and generating a first intermediate image having a higher resolution than the low-resolution image and a second intermediate image having a higher resolution than the low-resolution image based on the first feature map; generating an estimated image having a higher resolution than the low-resolution image based on the first intermediate image and the second intermediate image; the generator includes a plurality of residual blocks each including a plurality of linear sum layers; 13. An image processing apparatus comprising: a residual block that outputs a sum of an input to the residual block and a result of the input being calculated by the residual block.
23. A program causing a computer to execute the image processing method according to any one of claims 1 to 17.
24. A processing means for learning a machine learning model, The processing means includes: acquiring a plurality of low resolution images and a plurality of high resolution images; converting the low-resolution image into a first feature map using a generator, and generating a first intermediate image having a higher resolution than the low-resolution image and a second intermediate image having a higher resolution than the low-resolution image based on the first feature map; A means for identifying whether an image input to the classifier is an image generated by the generator; learning the generator based on a first loss based on a difference between the high-resolution image corresponding to the low-resolution image and the first intermediate image or the second intermediate image, and a second loss based on a classification output of the classifier when the first intermediate image or the second intermediate image is input; A learning device, characterized in that a weight of the second loss with respect to the first loss in a loss defined using the first intermediate image is smaller than a weight of the second loss with respect to the first loss in a loss defined using the second intermediate image.
25. A program for causing a computer to execute the learning method according to any one of claims 18 to 21.
Citation Information
Patent Citations
Super resolution using a generative adversarial network
US20180075581A1
Display device
WO2019030615A1
