Image processing apparatus, and image processing method

Abstract

To provide a technique to acquire restoration accuracy in processing of restoring an image before deterioration with higher accuracy and at low cost.SOLUTION: An image processing apparatus acquires restoration accuracy in processing of restoring a teacher image from a deteriorated image obtained by deteriorating the teacher image, acquires the frequency characteristics of the teacher image, and performs learning processing for a learning model based on the restoration accuracy and the frequency characteristics.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a technique for restoring an image to its original state before it was deteriorated. [Background technology]

[0002] In recent years, convolutional neural networks (CNNs) have been applied to many image restoration applications. Image restoration specifically refers to the process of restoring an original image from a degraded image, such as noise removal, blur removal, super-resolution, and loss completion.

[0003] A large amount of training data is required for learning using CNN for degradation restoration. When training data is collected randomly, various types of images can be obtained, such as nature photos including landscapes and animals, portraits or sports photos of people, and photos of man-made objects such as architecture and products. However, the training data collected in this way contains many low-frequency components, which leads to a decrease in performance of degradation restoration. To address this issue, a method called "hard sample mining" is known that collects high-difficulty patches that contain many high-frequency components that are desirable for degradation restoration and builds a dataset (Non-Patent Document 1). The processing content is shown below.

[0004] First, a degradation element is added to a teacher image to generate a degraded image. This corresponds to a process of simulating recapturing or regenerating the teacher image under conditions where degradation occurs. Next, a degradation restoration process is applied to the degraded image to generate a degraded restored image. Finally, the error between the teacher image and the degraded restored image is evaluated. The magnitude of this error is the difficulty level. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Wang, Shizun, et al. “SamplingAug: On the Importance of Patch Sampling Augmentation for Single Image Super-Resolution.” arXiv preprint arXiv:2111.15185 (2021). Summary of the Invention [Problem to be solved by the invention]

[0006] For highly accurate difficulty calculation, it is necessary to accurately reproduce the imaging process in the imaging device. This reproduction process includes physical simulation, sensor simulation, and development process. Therefore, there is a problem that the calculation cost becomes huge. In the present invention, a technology is provided for obtaining the restoration accuracy in the process of restoring an image before deterioration with higher accuracy and lower cost. [Means for solving the problem]

[0007] One aspect of the present invention is characterized in that it comprises a first acquisition means for acquiring restoration accuracy in a process of restoring a teacher image from a degraded image obtained by adding degradation to the teacher image, a second acquisition means for acquiring frequency characteristics of the teacher image, and a first learning means for performing a learning process of a learning model based on the restoration accuracy and the frequency characteristics. Effect of the Invention

[0008] According to the present invention, it is possible to provide a technique for obtaining higher restoration accuracy and lower cost in a process for restoring an image before deterioration. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an image processing system. [Diagram 2] FIG. 4 is a diagram for explaining an outline of a power spectrum used when calculating frequency characteristics. [Diagram 3]FIG. 1 is a block diagram showing an example of the functional configuration of an image processing system. [Figure 4] 4 is a flowchart of the operation of the image processing system. [Diagram 5] FIG. 3 is a block diagram showing an example of the functional configuration of a construction unit 311. [Figure 6] 4 is a flowchart of the operation of the image processing system. [Figure 7] FIG. 13 is a diagram showing an example of a histogram. [Figure 8] FIG. 13 is a diagram showing an example of theoretical characteristics. [Figure 9] FIG. 13 is a diagram showing an example of a mixing characteristic. [Figure 10] FIG. 13 is a diagram for explaining a specific example of a modification of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] <CNNについて> First, a Convolutional Neural Network (CNN) used in the general image processing technology to which deep learning is applied, which is used in each of the following embodiments, will be described. CNN is a technology in which a filter generated by training or learning is convolved with an input image, followed by nonlinear calculation, is repeated. The filter is also called a local receptive field. An image obtained by convolving the input image with the filter and then performing nonlinear calculation is called a feature map. Moreover, the learning is performed using training data (training images or data sets) consisting of a pair of an input image and an output image. Simply put, learning is to generate, from the training data, the values ​​of a filter that can convert an input image into a corresponding output image with high accuracy. This will be described in detail later.

[0012] When the input image has RGB color channels or the feature map is composed of multiple images, the filter used for convolution also has multiple channels accordingly. In other words, the convolution filter is expressed as a four-dimensional array that includes the number of channels in addition to the vertical and horizontal sizes and number of images. The process of nonlinear calculation after convolving the filter with the image (or feature map) is expressed in units of layers, such as the nth layer feature map or the nth layer filter. In addition, for example, a CNN that repeats the filter convolution and nonlinear calculation three times has a three-layer network structure. Such nonlinear calculation processing can be formulated as the following equation (1).

[0013]

number

[0014] Here, in the above formula (1), W n is the nth (1≦n≦N) layer filter, b n is the bias of the nth layer, f is a nonlinear operator, and X nis the feature map of the n-th layer, and * is the convolution operator. Note that the (l) on the upper right indicates the l-th filter or feature map. The filters and biases are generated by the learning described later and are collectively called "network parameters". As the non-linear operation, for example, a sigmoid function or ReLU (Rectified Linear Unit) is used. For example, ReLU is given by the following equation (2).

[0015]

Number

[0016] Here, as shown in the above equation (2), in ReLU, for the elements of the input vector X that are negative, 0 is output as the output value, and for the elements of the input vector X that are positive, the value of the element is output as the output value.

[0017] As networks using CNN, ResNet in the field of image recognition and its application RED-Net in the field of super-resolution are well-known. In both cases, by stacking multiple layers of CNN and performing convolution of filters multiple times, the accuracy of processing is improved. For example, ResNet features a network structure with a path that shortcuts the convolutional layer, thereby realizing a multi-layer network of 152 layers and achieving high-precision recognition approaching the human recognition rate. The reason why the processing is made more accurate by multi-layer CNN is simply that by repeating the non-linear operation multiple times, the non-linear relationship between the input and output can be expressed.

[0018] <Learning of CNN> Next, the learning of CNN will be described. The learning of CNN is generally performed by minimizing the objective function represented by the following equation (3) using a set (learning data) of an input learning image (student image) and an output learning image (teacher image) corresponding to the input learning image.

[0019]

Number

[0020] In the above formula (3), L is a loss function that measures the error between the correct answer and its estimate. Y i is the i-th output training image, X i is the i-th input training image. F is a function that collectively represents the calculations (equation (1)) performed in each layer of CNN. θ is the network parameters (filter and bias). ||Z||2 is the L2 norm, or more simply, the square root of the sum of the squares of the elements of vector Z. n is the total number of training data used for training. Since the number of training data is generally large, in the Stochastic Gradient Descent (SGD) method, a portion of the training images is randomly selected and used for training. This reduces the computational load in training. Various methods are known for minimizing (optimizing) the objective function, such as the momentum method, AdaGrad method, AdaDelta method, and Adam method. The Adam method is given by the following equation (4).

[0021]

number

[0022] In the above formula (4), θ i t is the i-th network parameter at the t-th iteration, and g is θ i t is the gradient of the loss function L with respect to . Additionally, m and v are moment vectors, α is the base learning rate, β1 and β2 are hyperparameters, and ε is a small constant. Note that since there are no guidelines for selecting an optimization method in learning, essentially any method can be used, but it is known that differences in convergence between methods will result in differences in learning time.

[0023] [First embodiment] In this embodiment, a method is described for estimating the restoration accuracy (difficulty) in a process for restoring an undegraded image from a degraded (image degradation) image, and generating a data set (learning data set) used for learning a "learning model for estimating an undegraded image from a degraded image" based on the result of the estimation. The learning model includes, for example, learned parameters of the learning model and information representing a network structure in the learning model.

[0024] The "difficulty level" according to this embodiment is calculated based on an input image and a degraded image obtained by adding degradation to the input image. Conventional methods include a procedure for adding costly image degradation that reproduces the image capture process by a camera, which results in a huge calculation cost for difficulty level calculation. In contrast, the method according to this embodiment calculates difficulty level using a model that has been trained to output a difficulty level corresponding to the frequency characteristics of the input image, so that the calculation cost for difficulty level calculation can be reduced compared to conventional methods. In this embodiment, "degradation" is described by taking "reduced resolution" as an example.

[0025] First, a configuration example of an image processing system according to this embodiment will be described with reference to the block diagram of Fig. 1. As shown in Fig. 1, the image processing system according to this embodiment includes a cloud server 200 and an edge device 100, and the cloud server 200 and the edge device 100 are configured to be able to communicate data with each other via the Internet.

[0026] First, the cloud server 200 will be described. The cloud server 200 generates a data set (learning data set) used for learning a learning model for estimating an undegraded image from a degraded image, and performs a learning process for the learning model using the generated learning data set. Then, the cloud server 200 provides the edge device 100 with the trained learning model in response to a request from the edge device 100.

[0027] CPU 201 executes various processes using computer programs and data stored in RAM 203. As a result, CPU 201 controls the overall operation of cloud server 200, and executes or controls various processes that will be described as processes performed by cloud server 200.

[0028] ROM 202 stores setting data for cloud server 200, computer programs and data related to the startup of cloud server 200, computer programs and data related to the basic operation of cloud server 200, and the like.

[0029] The RAM 203 has an area for storing computer programs and data loaded from the ROM 202 or the mass storage device 204. The RAM 203 also has an area for storing computer programs and data received from the edge device 100 via the network I / F 205. The RAM 203 also has a work area used when the CPU 201 executes various processes. In this way, the RAM 203 can provide various areas as appropriate.

[0030] The mass storage device 204 is a mass information storage device such as a hard disk drive or SSD. The mass storage device 204 stores an OS (operating system), computer programs and data for causing the CPU 201 to execute or control various processes described as processes performed by the cloud server 200, and the like. The computer programs and data stored in the mass storage device 204 are loaded into the RAM 203 as appropriate under the control of the CPU 201, and become targets for processing by the CPU 201.

[0031] The network I / F 205 is an interface for performing data communication with the edge device 100 via the Internet. The CPU 201, the ROM 202, the RAM 203, the mass storage device 204, and the network I / F 205 are all connected to a system bus 206.

[0032] Next, a description will be given of the edge device 100. The edge device 100 acquires a learning model provided by the cloud server 200, and acquires the output of the learning model to which an input image has been input as an undegraded image of the input image.

[0033] The CPU 101 executes various processes using computer programs and data stored in the RAM 102. As a result, the CPU 101 controls the overall operation of the edge device 100, and executes or controls various processes that will be described as processes performed by the edge device 100.

[0034] The RAM 102 has an area for storing computer programs and data loaded from the ROM 103, the mass storage device 104, or the external storage device 30. The RAM 102 also has an area for storing computer programs and data received from the cloud server 200 via the network I / F 106, and an area for storing captured images received from the imaging device 10 via the general-purpose I / F 105. The RAM 102 also has a work area used when the CPU 101 executes various processes. In this way, the RAM 102 can provide various areas as appropriate.

[0035] The ROM 103 stores setting data for the edge device 100, computer programs and data related to the startup of the edge device 100, computer programs and data related to the basic operation of the edge device 100, and the like.

[0036] The mass storage device 104 is a mass information storage device such as a hard disk drive or SSD. The mass storage device 104 stores an OS (operating system), computer programs and data for causing the CPU 101 to execute or control various processes described as processes performed by the edge device 100. The computer programs and data stored in the mass storage device 104 are loaded into the RAM 103 as appropriate under the control of the CPU 101, and become targets for processing by the CPU 101.

[0037] The network I / F 106 is an interface for performing data communication with the cloud server 200 via the Internet. The general-purpose I / F 105 is, for example, a serial bus interface such as USB, IEEE1394, or HDMI (registered trademark). The imaging device 10, the input device 20, the external storage device 30, and the display device 40 are connected to the general-purpose I / F 105.

[0038] The imaging device 10 is a device capable of capturing moving images or still images. For example, the imaging device 10 outputs an image of each frame in a moving image as a captured image. Also, for example, the imaging device 10 outputs a still image captured periodically or irregularly as a captured image.

[0039] The input device 20 is a user interface such as a keyboard, a mouse, or a touch panel, and a user can input various information and instructions to the edge device 100 by operating it.

[0040] The external storage device 30 is, for example, a memory card, a CF card, an SD card, a USB memory, or other various storage media, and may store some of the computer programs and data described as being stored in the large-capacity storage device 104. The computer programs and data stored in the external storage device 30 are loaded into the RAM 103 as appropriate under the control of the CPU 101, and become the subject of processing by the CPU 101.

[0041] The display device 40 has a liquid crystal screen or a touch panel screen, and displays the results of processing by the CPU 101 as images, characters, etc. The display device 40 may be a projection device such as a projector that projects images and characters. The CPU 101, RAM 102, ROM 103, mass storage device 104, general-purpose I / F 105, and network I / F 106 are all connected to a system bus 107.

[0042] Note that the configurations of the cloud server 200 and the edge device 100 shown in Fig. 1 are merely examples, and are not intended to be limiting. Furthermore, the configuration of the image processing system according to this embodiment is not limited to the configuration shown in Fig. 1. For example, the functions of the cloud server 200 may be realized by processing by a plurality of computer devices.

[0043] In addition, the functions of the cloud server 200 and / or the edge device 100 may be incorporated into the imaging device 10, and the imaging device 10 may perform processes such as generating a learning dataset, learning a learning model, and acquiring the output of a learning model to which an image is input as an undegraded image of the image.

[0044] Next, a functional configuration example of the image processing system according to this embodiment is shown in the block diagram of Fig. 3. Of the functional units shown in Fig. 3, the functional units other than the main inference unit 313 are functional units possessed by the cloud server 200, and the main inference unit 313 is a functional unit possessed by the edge device 100.

[0045] In this embodiment, a case will be described in which each functional unit shown in FIG. 3 is implemented by software (computer program). In the following, each functional unit shown in FIG. 3 may be described as the subject of processing. In reality, however, the functions of the functional units other than the inference unit 313 are realized by CPU 201 executing a computer program corresponding to the functional units, and the functions of the functional units are realized by CPU 101 executing a computer program corresponding to the inference unit 313. Note that one or more of the functional units shown in FIG. 3 may be implemented by hardware.

[0046] The operation of the image processing system according to this embodiment will be described with reference to the flowchart in Fig. 4. In the following, resolution reduction will be taken up as an example of degradation, and an example of inferring super-resolution processing for restoring an image that is the source of the resolution reduction will be described.

[0047] In step S401, the acquisition unit 301 acquires the main learning parameters in the RAM 203. The main learning is to perform super-resolution learning using a learning dataset generated by a process described later. In the super-resolution learning, a degraded image is generated by adding resolution degradation to a teacher image, and the super-resolution learning learns to restore the teacher image that is the source of the degradation of the degraded image. The main learning parameters include the "degradation kernel," "image reduction ratio," and "learning patch size" used in this learning.

[0048] The acquisition unit 301 may acquire the main learning parameters from the mass storage device 204, or may acquire them from an external device via the network I / F 205, and the source from which the main learning parameters are acquired is not limited to a specific source.

[0049] In step S402, the acquisition unit 302 acquires an input image. The acquisition unit 302 may acquire a captured image output from the imaging device 10 as an input image, or may acquire an input image stored in advance in the large-capacity storage device 204. The source of the input image is not limited to a specific source.

[0050] The acquisition unit 302 then divides the acquired input image into a plurality of patches, each having a "learning patch size," and acquires the patches as teacher images. That is, the acquisition unit 302 acquires a plurality of teacher images (the size of each teacher image is the "learning patch size" included in the learning parameters) from the input image. Note that the input image itself may be used as the teacher image without dividing the patches.

[0051] In step S403, the extraction unit 303 selects some of the multiple teacher images acquired by the acquisition unit 302 as selected teacher images, and constructs a set of the selected teacher images as a partial data set. How the selected teacher images are selected from the multiple teacher images acquired by the acquisition unit 302 is not limited to a specific selection method, and may be selected at random or according to some criteria, for example.

[0052] In step S404, the adding unit 304 generates a degraded image by adding degradation to each selected teacher image included in the partial data set. For example, the adding unit 304 adds degradation to the selected teacher image by convolving a degradation kernel that reproduces the loss of high-frequency components in the lens of the image capture device 10 with the selected teacher image, and then applying image reduction that reproduces the light collection in the sensor of the image capture device 10.

[0053] A method of convolving the degradation kernel and adding degradation will be described below. An image captured using an ideal optical system that does not cause degradation is defined as f. Here, f(x, y) represents the pixel value of the pixel at pixel position (x, y) in the image f. On the other hand, an image captured using an optical system that causes degradation is defined as g. Here, g(x, y) represents the pixel value of the pixel at pixel position (x, y) in the image g. Also, the degradation kernel is defined as h. The degradation kernel h is determined from the shooting conditions such as the lens type, camera type, aperture, focal length, and subject distance. Here, h(x, y) represents the value of the degradation kernel corresponding to the pixel position (x, y). The relationship of the following formula (5) holds between f(x, y), g(x, y), and h(x, y).

[0054]

number

[0055] Image reduction is achieved by thinning out pixels at a given rate. For example, if the "image reduction rate" included in the learning parameters is 1 / 4, then pixels are thinned out at intervals of 4 pixels. Note that the image reduction method is not limited to a specific method.

[0056] In step S405, the restoration unit 305 performs a "restoration process for restoring an image before deterioration of the degraded image" on each degraded image generated by the adding unit 304, and generates a degraded restored image as a result. For the restoration process, for example, a super-resolution network that has learned the restoration process is used. The super-resolution network used is one created based on CNN. In addition to the above, for example, a linear interpolation algorithm such as the Bicubic method may be applied to the restoration process.

[0057] In step S406, the calculation unit 306 calculates, for each selected teacher image included in the partial data set, the difference (error) between the selected teacher image and the degraded restored image corresponding to the selected teacher image using the following equation (6).

[0058]

number

[0059] Here, d is the error, MSE(A,B) is the mean square error between image A and image B, HR is the selected teacher image, and SR is the degraded restored image corresponding to the selected teacher image. Note that other image quality evaluation indices such as peak signal-to-noise ratio (PSNR) and L1 loss may be used instead of MSE.

[0060] In step S407, the calculation unit 307 calculates the frequency characteristics of each selected teacher image included in the partial data set. First, the calculation unit 307 calculates an MTF (Modulated Transfer Function) using the following equation (7).

[0061]

number

[0062] Here, g(x,y) is the intensity at coordinates (x,y) on the power spectrum of the selected teacher image, and r is the distance from the center to the edge of the power spectrum. This formula (7) represents an operation of integrating one section of an area previously divided into a given number of sections in the direction from the center to the edge after averaging the power spectrum in the circumferential direction. A conceptual diagram is shown in FIG. 2. Here, the amount of high-frequency components increases in the direction from the center to the edge. Note that the formula for calculating MTF is not limited to the above formula (7), and other calculation methods may be used as long as the evaluation value has a correlation with the amount of adverse effects. Then, the calculation unit 307 uses the following formula (8) to obtain a vector p whose elements are values ​​obtained by integrating the MTF for each section, as the frequency characteristic.

[0063]

number

[0064] Here, c is the interval width, and N is the number of intervals. As shown in the following formula (9), the bin number a corresponding to the frequency at which the intensity of the MTF calculated from the degraded image is equal to or lower than a given threshold and the bin number b corresponding to the frequency at which the intensity of the MTF calculated from the teacher image is equal to or lower than a given threshold may be set as the integral interval, and the vector p may be calculated.

[0065]

number

[0066] The bin number a may be calculated based on the image reduction rate. For example, when the image reduction rate is 1 / 4, the lower limit bin number a is also 1 / 4 of the number of sections N. In step S408, the learning unit 308 performs a process of updating the parameters (weighting coefficients, etc.) of the error calculation model so that the output of the error calculation model to which the frequency characteristics of each selected teacher image are input becomes the error calculated in step S406 for the selected teacher image (learning process of the error calculation model). For example, a multilayer perceptron is used as the error calculation model, but the present invention is not limited to this, and other calculation models such as other types of neural networks and linear regression formulas may also be used.

[0067] In this manner, in this embodiment, the error calculation model is trained using not all of the teacher images obtained from the input image, but some of the teacher images. However, in this embodiment, the error calculation model can be trained more quickly than when the error calculation model is trained using all of the teacher images obtained from the input image.

[0068] In step S409, the calculation unit 309 obtains the frequency characteristics of each of the multiple teacher images acquired by the acquisition unit 302 in the same manner as the calculation unit 307. In step S410, the inference unit 310 obtains, for each of the multiple teacher images acquired by the acquisition unit 302, the output (error) of the error calculation model to which the frequency characteristics of the teacher image are input.

[0069] In step S411, the construction unit 311 identifies (selects) as the main learning teacher image a teacher image having an error (i.e., the above-mentioned difficulty) calculated in step S410 equal to or greater than a threshold value from among the multiple teacher images acquired by the acquisition unit 302. In other words, the construction unit 311 identifies as the main learning teacher image a teacher image having a restoration accuracy from a degraded image equal to or greater than a threshold value from among the multiple teacher images acquired by the acquisition unit 302. The construction unit 311 then generates a degraded image by adding degradation to the main learning teacher image in the same manner as in step S404, and constructs a set of the main learning teacher image and the degraded image generated for the main learning teacher image as a learning dataset.

[0070] In step S412, the main learning unit 312 performs a learning process of the degradation restoration model using the learning data set. That is, the main learning unit 312 performs a process (learning process of the degradation restoration model) of updating parameters (weighting coefficients, etc.) of the degradation restoration model so that the output of the degradation restoration model to which each degraded image included in the learning data set is input becomes a main learning teacher image corresponding to the degraded image. For example, a CNN is used as the degradation restoration model, but it is not limited to this, and other calculation models such as other types of neural networks and linear regression formulas may be used.

[0071] In step S413, the inference unit 313 acquires the degradation restoration model generated by the cloud server 200. The inference unit 313 also acquires the processing target image into the RAM 102. The inference unit 313 may acquire a captured image output from the imaging device 10 as the processing target image, or may acquire an image stored in the mass storage device 204 as the processing target image. The inference unit 313 may also acquire the processing target image from an external device (such as the cloud server 200) via the Internet. In this way, the acquisition source of the processing target image is not limited to a specific acquisition source.

[0072] The inference unit 313 then acquires the output of the degradation restoration model to which the processing target image has been input as an image before degradation of the processing target image (in this embodiment, an image subjected to super-resolution processing), and outputs the acquired image. The output destination of the image by the inference unit 313 is not limited to a specific output destination. For example, the inference unit 313 may display the acquired image on the display device 40, may store it in the mass storage device 104 or the external storage device 30, or may transmit it to an external device via the Internet.

[0073] As an example of the operation of such an image processing system, the following operation example is conceivable. When a user operates a web browser installed in the edge device 100 using the input device 20 to input an instruction for super-resolution processing, the web browser (edge ​​device 100) transmits a request to acquire a degradation restoration model to the cloud server 200. The cloud server 200, which has received the request, transmits the learned degradation restoration model to the edge device 100. The edge device 100 performs super-resolution processing using the degradation restoration model received from the cloud server 200, and displays the result of the super-resolution processing on the screen of the web browser.

[0074] <Modification> 3 is an example, and can be modified or changed as appropriate. For example, one functional unit can be divided into multiple functional units, or two or more functional units can be integrated into one functional unit.

[0075] In addition, in this embodiment, a case where low resolution is applied has been described as an example of degradation, but this is not limited to this, and general image degradation such as blur, noise, compression, color thinning, fog / haze / rain / snow, aberration, and loss can also be applied.

[0076] In this embodiment, learning is performed from scratch using a group of teacher images prepared in advance, but the processing of this embodiment may be performed based on already-learned network parameters.

[0077] In this embodiment, the extraction unit 303 randomly selects teacher images from the teacher image group to configure the partial data set. However, the extraction unit 303 may select teacher images from the teacher image group so that the "average frequency characteristics" of the teacher image group and the partial data set match to configure the partial data set.

[0078] The partial data set may be extracted so that the average frequency characteristics of all images is approximately equal to the average frequency characteristics of the input image group. Specifically, the average frequency characteristics is calculated for a randomly selected partial data set. Then, a given number of images having frequencies that are missing compared to the average frequency characteristics of the input image group are extracted from the teacher image group and added to the partial data set. This operation is repeated until there are no more missing frequencies.

[0079] [Second embodiment] In the following embodiments and modifications including this embodiment, the differences from the first embodiment will be described, and unless otherwise specified below, they are assumed to be the same as the first embodiment. In the first embodiment, an aspect of reducing the calculation cost of difficulty calculation by using an error calculation model that has learned the process of difficulty calculation when constructing a learning dataset has been described.

[0080] In this embodiment, a method for constructing a learning dataset so as to include many high-frequency components that are considered to be desirable for super-resolution will be described. Specifically, a teacher image and a corresponding degraded image to be included in the learning dataset are selected so as to follow a specific difficulty distribution suitable for super-resolution learning. This allows for high-sharpness, low-artifact super-resolution learning. As a result, as in the first embodiment, the process of adding expensive degradation that reproduces the imaging process by a camera is omitted, and therefore calculations can be realized at a lower cost than conventional methods.

[0081] An example of the functional configuration of the construction unit 311 according to this embodiment is shown in the block diagram of Fig. 5. The operation of the image processing system according to this embodiment will be described below with reference to the flowchart of Fig. 6. Note that the same step numbers are used in Fig. 6 for the same processing steps as those shown in Fig. 4, and descriptions of these processing steps will be omitted.

[0082] In step S601, the acquisition unit 501 acquires a reference dataset including a plurality of reference images into the RAM 203. The reference images are images that include many high-frequency components that are desirable for super-resolution learning. The acquisition unit 501 may acquire the reference dataset from the mass storage device 204, or may acquire the reference dataset from an external device via the network I / F 205. The acquisition unit 501 may also acquire a set of captured images captured by the imaging device 10 as the reference dataset. In this way, the source of the reference dataset is not limited to a specific source.

[0083] In step S602, the calculation unit 502 obtains the frequency characteristics of each reference image included in the reference data set in the same manner as the calculation unit 307. In step S603, the inference unit 503 obtains, for each reference image included in the reference data set, the output (error) of the "error calculation model generated by the learning unit 308" to which the frequency characteristics of the reference image are input, in the same manner as the inference unit 310.

[0084] The inference unit 503 then generates a histogram of the errors acquired for each reference image included in the reference data set as a reference characteristic. An example of a histogram generated by the inference unit 503 is shown in Fig. 7. In Fig. 7, the horizontal axis represents the error (bin) and the vertical axis represents the frequency. The histogram in Fig. 7 has fewer low-frequency components and more high-frequency components than a histogram obtained from randomly collected images.

[0085] In step S604, the acquisition unit 504 acquires the theoretical characteristic. The theoretical characteristic is a histogram in which the number of selected teacher images increases as the error (difficulty level) increases. An example of the theoretical characteristic is shown in FIG. 8. In FIG. 8, the horizontal axis represents the error (bin) and the vertical axis represents the number of selected images. As shown in FIG. 8, the histogram is configured to adopt (acquire) a minimum number of low-frequency components and a maximum number of high-frequency components. Here, it is assumed that the number of bins in the reference characteristic and the number of bins in the theoretical characteristic are the same (=Nb).

[0086] In step S605, the mixer 505 obtains a mixed characteristic by using the reference characteristic and the theoretical characteristic. If the frequency corresponding to the x-th bin (error) in the reference characteristic is i(x) and the number of selections corresponding to the x-th bin (error) in the theoretical characteristic is j(x), the mixer 505 obtains a mixed characteristic k(x) by using the following formula (10).

[0087]

number

[0088] Here, the mixed characteristic k(x) represents the number of selections corresponding to the x-th bin (error). An example of the mixed characteristic is shown in Figure 9. In Figure 9, the horizontal axis represents x and the vertical axis represents k(x). The low frequency components are histograms obtained based on the ideal characteristics, and the high frequency components are histograms obtained based on the reference characteristics.

[0089] In addition, in the formula (10), an example is shown in which the reference characteristic and the theoretical characteristic are mixed by multiplying them, but the method of mixing the reference characteristic and the theoretical characteristic is not limited to this. For example, the mixer 505 may mix the reference characteristic and the theoretical characteristic by adding them, or may mix the reference characteristic and the theoretical characteristic by taking a weighted average of the reference characteristic and the theoretical characteristic.

[0090] In step S606, the extraction unit 506 acquires the multiple teacher images acquired by the acquisition unit 302, the errors acquired by the inference unit 310 for each of the multiple teacher images, and the mixing characteristics calculated by the mixing unit 505. The extraction unit 506 then divides the interval between the maximum error and the minimum error among the errors acquired by the inference unit 310 into Nb subintervals by equally dividing the interval into Nb. The extraction unit 506 then selects k(x) teacher images from the teacher images corresponding to the errors belonging to the x (1≦x≦Nb)th subinterval among the multiple teacher images acquired by the acquisition unit 302, and includes the selected teacher images and the degraded images corresponding to the teacher images in the learning dataset. The method of selecting k(x) teacher images from the teacher images corresponding to the errors belonging to the xth subinterval is not limited to a specific method, and may be selected randomly or according to some criteria.

[0091] Thus, according to this embodiment, it is possible to construct a learning data set that approximately matches the distribution of reference characteristics or theoretical characteristics that contain many high-frequency components that are considered desirable for super-resolution, and it is possible to perform super-resolution learning with higher sharpness and fewer artifacts than the first embodiment.

[0092] In addition, since there is no need to apply degradation or restoration processing to the reference dataset, the costly process of applying degradation that reproduces the image capture process by a camera, as in conventional methods, is omitted, resulting in lower cost calculations compared to conventional methods.

[0093] In this embodiment, the histogram is created by the cloud server 200, but the user may create it manually by operating an operation unit not shown, or the histogram may be received from an external device via the network I / F 205.

[0094] Also, instead of the mixed characteristic, the data set may be constructed so as to substantially match the reference characteristic or the theoretical characteristic. For example, the extraction unit 506 may select i(x) teacher images from the teacher images corresponding to the error belonging to the x-th subsection among the multiple teacher images acquired by the acquisition unit 302, and include the selected teacher images and the degraded images corresponding to the teacher images in the learning data set. Also, for example, the extraction unit 506 may select j(x) teacher images from the teacher images corresponding to the error belonging to the x-th subsection among the multiple teacher images acquired by the acquisition unit 302, and include the selected teacher images and the degraded images corresponding to the teacher images in the learning data set.

[0095] In this embodiment, the characteristic of the data set is a histogram of difficulty, but other data may be used as long as it represents the statistics of the difficulty of the data set.

[0096] <Modification> In the first and second embodiments, the calculation unit 307 calculates the frequency characteristic by averaging the power spectrum in the circumferential direction and then integrating one section of the area divided into a given number of sections in the direction from the center to the edge, but the frequency characteristic may be calculated for each direction. A specific example is shown in Fig. 10. The calculation method of the frequency characteristic described below is used by the calculation unit 307.

[0097] First, half-rays 1002 in multiple directions are set, each of which passes through the center 1001 of the power spectrum. Note that the half-rays 1002 do not have to pass through the center. Also, the half-rays 1002 may be straight lines, but may also be curved lines.

[0098] Next, the half line 1002 is divided into a plurality of intervals, the average of the power spectrum values ​​on each interval is calculated, and a vector p having these as elements is obtained. Equation (11) for obtaining vector p is shown below.

[0099]

number

[0100] where m ij is the power spectrum value average of the j-th section on the half line 1002 in the i-th direction, M is the number of elements in each section, and N is the number of sections. Note that the average may include values ​​around each section. Note that a differential image of the teacher image may be used instead of the power spectrum. In this case, a vector p having elements each representing a differential value for each direction is calculated.

[0101] Furthermore, the numerical values, processing timing, processing order, processing subject, data (information) acquisition method / destination / source / storage location, etc. used in each of the above embodiments and each of the modified examples are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0102] In addition, any part or all of the embodiments and modifications described above may be used in appropriate combination. In addition, any part or all of the embodiments and modifications described above may be used selectively.

[0103] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0104] The invention of this specification includes the following image processing device, image processing method, and computer program.

[0105] (Item 1) A first acquisition means for acquiring a restoration accuracy in a process of restoring a teacher image from a degraded image obtained by adding degradation to the teacher image; A second acquisition means for acquiring a frequency characteristic of the teacher image; a first learning means for performing a learning process of a learning model based on the restoration accuracy and the frequency characteristic; An image processing device comprising:

[0106] (Item 2) 2. The image processing device according to item 1, wherein the first acquisition means acquires a difference between a degraded restored image obtained by processing for restoring the degraded image and the teacher image as the restoration accuracy.

[0107] (Item 3) 3. The image processing device according to item 1 or 2, wherein the first learning means learns the learning model so that an output of the learning model to which the frequency characteristic is input becomes the restoration accuracy.

[0108] (Item 4) moreover, A selection means for selecting some teacher images from the plurality of teacher images as main learning teacher images based on a restoration accuracy obtained by inputting frequency characteristics of each of the plurality of teacher images into the learning model; a generating means for generating a data set including the main training teacher image and a degraded image obtained by adding degradation to the main training teacher image; 4. The image processing device according to any one of items 1 to 3, comprising:

[0109] (Item 5) moreover, a selection means for determining a distribution of the number of teacher images to be selected for each restoration accuracy from among the plurality of teacher images based on a distribution of restoration accuracy obtained by inputting the frequency characteristics of each of the plurality of reference images into the learning model and a distribution of the number of selections corresponding to the restoration accuracy that has been created in advance, and selecting some of the teacher images from the plurality of teacher images as main learning teacher images based on the determined distribution; a generating means for generating a data set including the main training teacher image and a degraded image obtained by adding degradation to the main training teacher image; 4. The image processing device according to any one of items 1 to 3, comprising:

[0110] (Item 6) moreover, A selection means for selecting some teacher images from the plurality of teacher images as main learning teacher images based on a distribution of restoration accuracy obtained by inputting frequency characteristics of each of the plurality of reference images into the learning model; A generating means for generating a data set including the main training teacher image and an image obtained by adding degradation to the main training teacher image; 4. The image processing device according to any one of items 1 to 3, comprising:

[0111] (Item 7) moreover, A selection means for selecting some teacher images as main learning teacher images from the plurality of teacher images based on a distribution of the number of selected images corresponding to the restoration accuracy created in advance; A generating means for generating a data set including the main training teacher image and an image obtained by adding degradation to the main training teacher image; 4. The image processing device according to any one of items 1 to 3, comprising:

[0112] (Item 8) moreover, 8. The image processing device according to any one of items 4 to 7, further comprising a second learning means for performing a learning process of a learning model for estimating an undegraded image from an image to which degradation has been added, using the data set.

[0113] (Item 9) 9. The image processing device according to any one of items 1 to 8, wherein the first acquisition means acquires, as the teacher image, a portion of a plurality of patches obtained by dividing an input image.

[0114] (Item 10) 10. The image processing device according to item 9, wherein the first acquisition means acquires the portion of patches as the teacher image so that an average of frequency characteristics of the plurality of patches coincides with an average of frequency characteristics of the portion of patches.

[0115] (Item 11) The image processing device described in item 9, characterized in that the first acquisition means acquires some of the patches as the teacher image so that an average of the frequency characteristics of the input image group coincides with an average of the frequency characteristics of some of the patches obtained by dividing the input image.

[0116] (Item 12) An image processing method performed by an image processing device, comprising: a first acquisition step in which a first acquisition means of the image processing device acquires a restoration accuracy in a process of restoring a teacher image from a deteriorated image obtained by adding deterioration to the teacher image; A second acquisition step in which a second acquisition means of the image processing device acquires a frequency characteristic of the teacher image; a first learning step in which a first learning means of the image processing device performs a learning process of a learning model based on the restoration accuracy and the frequency characteristic; An image processing method comprising:

[0117] (Item 13) 12. A computer program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 11.

[0118] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0119] 301: Acquisition unit 302: Acquisition unit 303: Extraction unit 304: Assignment unit 305: Restoration unit 306: Calculation unit 307: Calculation unit 308: Learning unit 309: Calculation unit 310: Inference unit 311: Construction unit 312: Main learning unit 313: Main inference unit

Claims

1. a first acquisition means for acquiring a restoration accuracy in a process of restoring a teacher image from a degraded image obtained by adding degradation to the teacher image; A second acquisition means for acquiring a frequency characteristic of the teacher image; a first learning means for performing a learning process of a learning model based on the restoration accuracy and the frequency characteristic; An image processing device comprising:

2. 2 . The image processing apparatus according to claim 1 , wherein the first acquisition means acquires, as the restoration accuracy, a difference between a degraded restored image obtained by a process for restoring the degraded image and the teacher image.

3. 2 . The image processing apparatus according to claim 1 , wherein the first learning means learns the learning model so that an output of the learning model to which the frequency characteristic is input corresponds to the restoration accuracy.

4. moreover, A selection means for selecting some teacher images from the plurality of teacher images as main learning teacher images based on a restoration accuracy obtained by inputting frequency characteristics of each of the plurality of teacher images into the learning model; a generating means for generating a data set including the main training teacher image and a degraded image obtained by adding degradation to the main training teacher image; The image processing device according to claim 1 , further comprising:

5. moreover, a selection means for determining a distribution of the number of teacher images to be selected for each restoration accuracy from among the plurality of teacher images based on a distribution of restoration accuracy obtained by inputting the frequency characteristics of each of the plurality of reference images into the learning model and a distribution of the number of selections corresponding to the restoration accuracy that has been created in advance, and selecting some of the teacher images from the plurality of teacher images as main learning teacher images based on the determined distribution; a generating means for generating a data set including the main training teacher image and a degraded image obtained by adding degradation to the main training teacher image; The image processing device according to claim 1 , further comprising:

6. moreover, A selection means for selecting some teacher images from the plurality of teacher images as main learning teacher images based on a distribution of restoration accuracy obtained by inputting frequency characteristics of each of the plurality of reference images into the learning model; A generating means for generating a data set including the main training teacher image and an image obtained by adding degradation to the main training teacher image; The image processing device according to claim 1 , further comprising:

7. moreover, A selection means for selecting some teacher images as main learning teacher images from the plurality of teacher images based on a distribution of the number of selected images corresponding to the restoration accuracy created in advance; A generating means for generating a data set including the main training teacher image and an image obtained by adding degradation to the main training teacher image; The image processing device according to claim 1 , further comprising:

8. moreover, 5. The image processing device according to claim 4, further comprising a second learning means for performing a learning process of a learning model for estimating an undegraded image from an image to which degradation has been imparted, using the data set.

9. The image processing apparatus according to claim 1 , wherein the first acquisition means acquires, as the teacher image, a portion of a plurality of patches obtained by dividing an input image.

10. 10. The image processing device according to claim 9, wherein the first acquisition means acquires the portion of patches as the teacher image so that an average of frequency characteristics of the plurality of patches coincides with an average of frequency characteristics of the portion of patches.

11. The image processing device according to claim 9, characterized in that the first acquisition means acquires some of the patches as the teacher image so that an average of the frequency characteristics of a group of input images matches an average of the frequency characteristics of some of the multiple patches obtained by dividing the input images.

12. An image processing method performed by an image processing device, comprising: a first acquisition step in which a first acquisition means of the image processing device acquires a restoration accuracy in a process of restoring a teacher image from a degraded image obtained by adding degradation to the teacher image; a second acquisition step in which a second acquisition means of the image processing device acquires a frequency characteristic of the teacher image; a first learning step in which a first learning means of the image processing device performs a learning process of a learning model based on the restoration accuracy and the frequency characteristic; An image processing method comprising:

13. A computer program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 11.

Images