Method for generating learning data, learning method, learning data production device, learning device, and program

Abstract

To provide a method for generating learning data and the like that can achieve accurate correction while reducing the possibility of the occurrence of a harmful effect, even when an image to be input to a neural network includes a high-luminance subject and a subject greatly blurred due to aberration of an optical system.SOLUTION: A method for manufacturing learning data to be used for learning of a neural network includes: a first acquisition step of acquiring an original image; a second acquisition step of acquiring a first image as a training image created by adding blur to the original image; and a third acquisition step of acquiring a second image as a correct answer image created by adding blur to the original image. The amount of blur added to the second image is smaller than that to the first image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a method for generating learning data used in deep learning, and a learning method using deep learning. [Background technology]

[0002] Patent Document 1 discloses a method for suppressing undershoot and ringing that accompanies higher resolution and higher contrast (sharpening) by taking into account the effects of gamma correction when training a multilayer neural network that uses RAW images as input. Non-Patent Document 1 discloses a network configuration that can be generally applied to various regression problems. Non-Patent Document 1 also discloses using a network to perform upsampling of input images, JPEG deblocking (removal of compression noise), denoising, non-blind deblurring, or inpainting. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-121252 [Non-patent literature]

[0004] [Non-Patent Document 1] X. Mao, C. Shen, Y. Yang, “Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections”, https: / / arxiv.org / abs / 1606.08921. Summary of the Invention [Problem to be solved by the invention]

[0005] However, the methods disclosed in Patent Document 1 and Non-Patent Document 1 may not be able to suppress undershooting, ringing, and other problems depending on the input image, and these problems may occur in the corrected image (estimated image). Specifically, these problems are likely to occur when the input image contains a highly luminous object or when the object is significantly blurred due to aberrations in the optical system, and particularly when the object's luminance value is high and saturated. When these images are used as training data, problems such as undershooting and ringing associated with the amount of correction may occur.

[0006] Therefore, the present invention aims to provide a method for producing training data that can achieve high-precision correction while reducing the possibility of adverse effects occurring, even when the image input to a neural network contains a highly bright subject or a subject that is significantly blurred due to aberrations in the optical system. [Means for solving the problem]

[0007] A method for generating training data according to one aspect of the present invention is a method for generating training data used in training a neural network, and includes the steps of acquiring an original image, generating a training image based on the original image and information relating to a first blur, and generating a ground truth image based on the original image and information relating to a second blur, wherein the information relating to the second blur is generated based on the information relating to the first blur.

[0008] Other objects and features of the present invention are illustrated in the following examples. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a method for producing training data that can achieve high-precision correction while reducing the possibility of adverse effects occurring, even when the image input to the neural network contains a highly luminous subject or a subject that is significantly blurred due to aberrations in the optical system. [Brief explanation of the drawings]

[0010] [Figure 1]1 is a flowchart of a method for producing training data in the first embodiment. [Figure 2] FIG. 1 is a block diagram of an image processing system according to first and third embodiments. [Figure 3] FIG. 1 is an external view of an image processing system according to first and third embodiments. [Figure 4] FIG. 2 is an explanatory diagram relating to optical information in each example. [Figure 5] FIG. 10 is an explanatory diagram of a correction map in Examples 1 and 3. [Figure 6] 10 is a flowchart of a learning process in each embodiment. [Figure 7] FIG. 1 is a diagram illustrating a convolutional neural network in each example. [Figure 8] FIG. 2 is an explanatory diagram of color components of an image in each embodiment. [Figure 9] 1 is a flowchart of an estimation step in Examples 1 and 2. [Figure 10] FIG. 10 is a block diagram of an image processing system according to a second embodiment. [Figure 11] FIG. 10 is an external view of an image processing system according to a second embodiment. [Figure 12] 10 is a flowchart of a method for producing training data in the second embodiment. [Figure 13] FIG. 10 is an explanatory diagram of a correction map in the second embodiment. [Figure 14] 11 is a flowchart of a method for producing training data in the third embodiment. [Figure 15] 10A and 10B are explanatory diagrams of optical performance and blur amount in Example 3. [Figure 16] 10 is a flowchart of an estimation step in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and redundant explanations will be omitted.

[0012] First, the terms used in each embodiment are defined below. Each embodiment relates to a method for solving a regression problem using deep learning and estimating various output images from an input image. Deep learning is machine learning using a multi-layered neural network. By learning network parameters (weights and biases) from pairs of a large number of training images and their corresponding ground truth images (desired outputs), highly accurate estimation is possible even for unknown input images.

[0013] Image processing using a multilayer neural network involves two processes: a process for updating the network parameters (weights and biases), and a process for making inferences about unknown inputs using the updated parameters. Hereinafter, the former will be called the learning process, and the latter will be called the estimation process.

[0014] Next, the names of the images used in the learning and estimation processes are determined. The image input to the network is referred to as the input image, and the input image used in the learning process, for which the correct answer image is known, is referred to as the training image. The image output from the network is referred to as the output image, and the output image in the estimation process is referred to as the estimated image. The input image to the network and the correct answer image are RAW images. Here, a RAW image is undeveloped image data output from an imaging device, in which the light intensity and signal value of each pixel have a roughly linear relationship. The RAW image is developed before the user views it, during which gamma correction is performed. Gamma correction is, for example, a process of raising the input signal value to a power, with an exponent such as 1 / 2.2 being used. Furthermore, the manufacturing method for learning data (learning images) in each embodiment generates an adjusted correct answer image used in learning to achieve high-precision correction while suppressing adverse effects. In each embodiment, an image equivalent to a lossless image that is the basis for generating the correct answer image or training image is referred to as the original image.

[0015] Before proceeding to a detailed description of each embodiment, the gist of the present invention will be described. The objective of the present invention is to construct a neural network capable of performing correction without adverse effects, even when an image contains a highly luminous subject or a subject that is significantly blurred due to optical system aberration, and to provide the necessary training data. To this end, the present invention is characterized in that, instead of using the original image as the correct image for the training data, an adjusted correct image is used. This adjusted correct image is an image in which blur is added to the original image in areas where adverse effects are likely to occur, such as high-luminance areas, high-contrast areas, and areas with large aberrations. By such adjustment, the sharpness of the correct image and the training images differs depending on the position on the screen. Alternatively, the training images may be generated by sharpening them. By using either method, the correct image can be adjusted to have a blur amount (sharpness) between the original image and the training image, thereby reducing the difference between the correct image and the training image in areas where adverse effects are likely to occur. By training using the correct image and training images created in this way, it is possible to generate a learning model that achieves high-precision correction while reducing the possibility of adverse effects. [Example]

[0016] An image processing system according to a first embodiment of the present invention will be described. In this embodiment, first, training data to be used for training a multi-layer neural network is generated. Then, training is performed using the generated training data, and correction processing is performed using the trained model.

[0017] Fig. 2 is a block diagram of an image processing system 100 in this embodiment. Fig. 3 is an external view of the image processing system 100. The image processing system 100 includes a learning device (image processing device) 101, an imaging device 102, an image estimation device (image processing device) 103, a display device 104, a recording medium 105, an output device 106, a network 107, and a learning data production device (image processing device) 108.

[0018] The training data production device 108 is a device that produces training data used in the training process, and includes a storage unit 108a, an acquisition unit 108b, and a generation unit 108c. The acquisition unit 108b acquires an original image (third image) and characteristic information (such as image height) of the optical system used to generate the training data. The generation unit 108c generates a training image (first image) and a reference image (second image) based on the original image. That is, the generation unit 108c functions as an acquisition unit that acquires training images and as a generation unit that generates reference images with optical characteristics different from those of the training images. The training images and reference images generated by the generation unit 108c are stored in the storage unit 108a. Details regarding the generation of training data will be described later with reference to the flowchart of FIG. 1.

[0019] The learning device 101 is an image processing device that executes a learning process, and includes a storage unit 101a, an acquisition unit 101b, a generation unit (generation means) 101c, and an update unit (update means) 101d. The acquisition unit 101b acquires training images and reference images. The generation unit 101c inputs the training images (first images) into a multi-layer neural network to generate an output image (processed image). The update unit 101d updates the network parameters of the neural network based on the difference (error) between the output image generated by the generation unit 101c and the reference image. Details of the learning process will be described later using a flowchart. The learned network parameters are stored in the storage unit 101a.

[0020] The imaging device 102 has an optical system 102a and an imaging element 102b. The optical system 102a collects light incident on the imaging device 102 from the subject space. The imaging element 102b receives (photoelectrically converts) an optical image (subject image) formed via the optical system 102a to obtain a captured image. The imaging element 102b is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor. The captured image obtained by the imaging device 102 contains blur due to aberration and diffraction of the optical system 102a and noise due to the imaging element 102b.

[0021] The image estimation device 103 is a device that executes the estimation process and includes a storage unit 103a, an acquisition unit 103b, and a correction unit (estimation means) 103c. The image estimation device 103 acquires a captured image and performs blur correction that suppresses adverse effects to generate an estimated image. A multilayer neural network is used for blur correction, and information on network parameters is read from the storage unit 103a. The network parameters are learned by the learning device 101, and the image estimation device 103 reads the network parameters from the storage unit 101a via the network 107 in advance and stores them in the storage unit 103a. The stored network parameters may be their actual values ​​or may be in an encoded format. Details regarding network parameter learning and blur correction processing using the network parameters will be described later.

[0022] The output image is output to at least one of a display device 104, a recording medium 105, and an output device 106. The display device 104 is, for example, a liquid crystal display or a projector. A user can perform editing work while checking the image being processed via the display device 104. The recording medium 105 is, for example, a semiconductor memory, a hard disk, a server on a network, etc. The output device 106 is, for example, a printer. The image estimation device 103 has a function of performing development processing and other image processing as necessary.

[0023] Next, a method for producing training data executed by the training data production device 108 will be described with reference to Fig. 1. Fig. 1 is a flowchart of the method for producing training data. Each step in Fig. 1 is executed by each unit (storage unit 108a, acquisition unit 108b, generation unit 108c) of the training data production device 108.

[0024] First, in step S101 of FIG. 1, the acquisition unit 108b acquires multiple original images (third images) (first acquisition step). The original images may be stored in advance in the storage unit 108a and acquired from there, or they may be acquired from an external device (not shown). In this embodiment, the original images are undeveloped RAW images. However, this embodiment is not limited to this, and developed images may also be used. The multiple original images are images of various subjects, i.e., images having edges of various strengths and directions, textures, gradations, flat areas, etc. The original images may be real images or images generated by CG (Computer Graphics). Preferably, the original images have a signal value higher than the brightness saturation value of the image sensor 102b. This is because even in real subjects, there are subjects that do not fall within the brightness saturation value when photographed by the image capture device 102 under specific exposure conditions.

[0025] Next, in step S102, the acquisition unit 108b acquires optical information of the optical system 102a used to impart (add) blur to the original image acquired in step S101 (fourth acquisition step). The optical system 102a has different aberrations and diffractions depending on multiple lens states (zoom, aperture, and focal length states), image height, and azimuth. Therefore, it is necessary to impart blur due to aberrations and diffractions that differ depending on the lens state, image height, and azimuth for each original image. In step S102, the acquisition unit 108b acquires a PSF (point spread function) of the optical system 102a to be used for blurring in the next step S103. The optical information acquired by the acquisition unit 108b at this time may be two-dimensional data corresponding to the distribution of the PSF, or may be a set of one-dimensional vectors obtained by decomposing the PSF using singular value decomposition or the like. Alternatively, the PSF may be approximated by fitting a specific function, and multiple coefficient data obtained during the approximation may be acquired as optical information. In this case, the PSF can be obtained by reconstructing the image using the acquired coefficient data and the approximation function. In this embodiment, a case where a PSF is used to add blur will be described, but an OTF (optical transfer function) may be used instead of a PSF. When an OTF is acquired as optical information, it may be acquired as two-dimensional data, or as one-dimensional vector or coefficient data, just like in the case of a PSF.

[0026] Figure 4 shows the positional relationship between the original image and the PSF, with the rectangular frame in Figure 4 representing the original image and the white circle representing the position of the PSF to be acquired or generated. When changing the PSF within the original image as shown in Figure 4, a PSF corresponding to each position is acquired or generated. While Figure 4 uses 9 x 9 pixels, this can be reduced to reduce the size, or increased to prioritize accuracy.

[0027] Next, in step S103, the generation unit 108c generates a training image (first image) using the optical information acquired in step S102 (second acquisition step). If the acquired optical information is not a PSF but includes coefficient data or the like, a PSF is generated. Then, the original image is blurred using the PSF. In this embodiment, blurring is achieved by convolving the original image with the PSF as a blurring filter. To change the blurring method depending on the image height, the PSF is changed for each pixel of the original image, and a linear sum of the PSF and the corresponding area of ​​the original image is calculated for all pixels. For example, when using a PSF corresponding to a position such as that shown in FIG. 4, to obtain the blurred pixel value of a pixel (a pixel of interest) located between white circles, first obtain the PSFs corresponding to the four neighboring white circles. Then, the four acquired PSFs are interpolated to calculate the PSF corresponding to the pixel of interest, and the linear sum of this PSF and the original image is calculated, and the resulting value is used as the blurred pixel value of the pixel of interest. As the interpolation method, existing methods such as bilinear, bicubic, nearest neighbor, etc. may be used.

[0028] Next, the blurred original image is reduced. At this time, if the original image has a signal value higher than the brightness saturation value of the image sensor 102b, the signal is clipped at the brightness saturation value of the image sensor 102b. In particular, when a real image is used as the original image, blurring has already occurred due to aberration and diffraction, so by reducing the image, the effect of blurring can be reduced and a high-resolution (high-quality) image can be obtained. Note that if the original image contains sufficient high-frequency components, reduction may not be necessary.

[0029] Next, in step S104, the generation unit 108c generates a correction map to be used in generating a correct image. FIG. 5 is a schematic diagram of the correction map in this embodiment, in which the original image is divided into four regions, A, B, C, and D, according to the image height. In FIG. 5, the region inside the solid line is region A, the region surrounded by the solid line and the dashed line is region B, the region surrounded by the dashed line and the two-dot chain line is region C, and the region outside the two-dot chain line is region D. In this embodiment, a correct image is generated by adding blur to the original image based on the performance of the optical system. Any index related to the performance of the optical system may be used for generating the correction map, but in this embodiment, the correction map is generated using the OTF of the optical system.

[0030] First, a Fourier transform is performed on the PSF corresponding to the white circles in Figure 4 to generate the OTF at the position corresponding to the white circle. Next, the absolute value of each OTF is taken to calculate the MTF (Modulation Transfer Function). Then, the average value of the MTF up to the Nyquist frequency of the image sensor is calculated, and this value is used as an index to generate a correction map. Index values ​​at positions other than the white circles are obtained by interpolating index values ​​corresponding to nearby white circles. Note that here, the index is calculated using the average MTF up to the Nyquist frequency, but an average MTF may be calculated in a different frequency band, or a specific frequency may be used as the index. The correction map in this embodiment corresponds to the amount of blur at each pixel, as it is used when blurring the target image.

[0031] In FIG. 5, the image is divided into four areas according to image height, and these areas are divided according to the average MTF value, which is an index value. For example, the area where the average MTF is 0.8 or more is Area A, the area where the average MTF is 0.6 or more but less than 0.8 is Area B, the area where the average MTF is 0.4 or more but less than 0.6 is Area C, and the area where the average MTF is less than 0.4 is Area D. Here, if the optical system 102a is a coaxial system, the optical performance will be symmetrical with respect to the optical axis, and the correction map will also be rotationally symmetrical as shown in FIG. 5. In addition, the performance of an optical system generally tends to be higher in the center of the image closer to the optical axis (the lower the image height), and lower in the peripheral area of ​​the image farther from the optical axis (the higher the image height), and therefore the correction map also often has a distribution such as that shown in FIG. 5. That is, when the image height is a first image height, the correct image has a first sharpness (first blur amount), and when the image height is a second image height that is higher than the first image height, the correct image has a second sharpness that is lower than the first sharpness (second blur amount that is larger than the first blur amount).

[0032] Note that when considering variations in the optical low-pass filter, the pixel aperture of the image sensor 102b, and the manufacturing process of the optical system 102a, this symmetry disappears, and the correction map does not necessarily have circular symmetry. Furthermore, while the OTF is calculated to calculate the index in this step, any other numerical value may be used as the index as long as it reflects the optical performance. For example, the peak value of the PSF can be used as the index. The peak value of the PSF is the integral value of the real part of the OTF, and therefore is an index correlated with the MTF. Alternatively, Parseval's theorem can be used to calculate the sum of squares of the PSF and then calculate the average value of the sum of squares of the MTF. In this way, if the index is calculated directly from the PSF without passing through the OTF to generate the correction map, a Fourier transform is not required, thereby reducing the amount of calculation. Furthermore, although the correction map in this embodiment is divided into four regions as shown in FIG. 5, the number of regions may be increased or decreased. Furthermore, although the index is divided into stages, the index value may be used as the correction map directly or as a continuous value.

[0033] Next, in step S105, the generation unit 108c generates a correct image (second image) based on the correction map created in step S104 (third acquisition process). When performing blur correction using deep learning, the greater the difference between the training image and the correct image in the learning data used for learning, the greater the likelihood of adverse effects occurring during estimation. The lower the performance of the optical system 102a, the more the training image is blurred in step S103. Therefore, if the original image is used as the correct image without modification, the training image and the correct image will deviate. In this embodiment, the possibility of adverse effects occurring is reduced by reducing this difference. In the correction map of FIG. 5, region A is a high-performance region, so the original image corresponding to this region is used as is without blurring it. As a result, for high-performance regions where the likelihood of adverse effects occurring is originally low, the amount of correction can be maintained by maintaining the difference between the training image and the correct image.

[0034] Next, for region B, the performance is lower than region A but higher than region C, so a small amount of blur is applied to the original image. For example, a PSF corresponding to each pixel is generated by reducing it by 30%, and the reduced PSF is applied to the original image to generate a correct image. Next, for region C, the performance is lower than region B, so the amount of blur is further increased. For example, a PSF is generated by reducing it by 60% of the PSF applied to the training image, and the reduced PSF is applied to the original image to generate a correct image. Then, for region D, the performance is the lowest, so the original image is blurred with the PSF applied to the training image without reduction to generate a correct image. The amount of PSF reduction for this correction map is one example and is not limited to this. Also, although region A is not blurred, the entire region may be blurred. However, the amount of blur applied to the correct image must be equal to or less than the amount of blur applied when generating the training image. In other words, the training image has lower sharpness (greater blur) than the correct image. Therefore, when blurring the ground truth image, the PSF used to generate the training images is used at the same size or reduced size.

[0035] In addition, if the training images are blurred and then reduced in size in step S103, the target image is also reduced by the same factor. At this time, if the original image has a signal value higher than the luminance saturation value of the image sensor 102b, the signal is clipped at the luminance saturation value of the image sensor 102b.

[0036] Next, in step S106, a pair of the training image generated in step S103 and the ground truth image generated in step S105 is stored as learning data in the storage unit 108a. The generated training image and ground truth image may be divided and output as multiple patches. A patch refers to an image having a predetermined number of pixels (e.g., 64 × 64 pixels). The number of pixels of the ground truth patch and the training patch does not necessarily have to match. When outputting as patches, the training image and ground truth image output via the flowchart of FIG. 1 may be divided, or the original image may be decomposed into patches in step S101 before the processing from step S102 onwards is performed.

[0037] As described above, in this embodiment, when generating a target image, the original image is not simply reduced and output, but blur is intentionally added (sharpness is reduced) to areas where the optical performance of the optical system is low. When blurring, blurring may be performed using a rotationally symmetric blur such as a Gaussian distribution, but blurring based on deterioration of the optical system is more preferable, and using the PSF of the optical system as in this embodiment allows for more accurate correction.

[0038] Next, a network parameter learning method (a method for manufacturing a trained model) executed by the learning device 101 in this embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart related to network parameter learning. Each step in Fig. 6 is mainly executed by the acquisition unit 101b, the generation unit 101c, or the update unit 101d of the learning device 101.

[0039] First, in step S201 of FIG. 6, the acquisition unit 101b acquires a correct answer patch (first correct answer image) and a training patch (first training image). The correct answer patch is an image with relatively little blur, and the training patch is an image with relatively more blur. In this embodiment, mini-batch learning is used to learn the network parameters of the multi-layer neural network. Therefore, in step S101, multiple pairs of correct answer patches and training patches are acquired. However, this embodiment is not limited to this, and online learning or batch learning may also be used.

[0040] Next, in step S202, the acquisition unit 101b acquires information related to the correction strength (correction strength information) used in the learning process. In this embodiment, the correction strength information is information linked to the learning data, and enables the amount of correction to be controlled during estimation. For example, if two pieces of correction strength information, "strong" and "weak," can be set during estimation, it is necessary to learn using the corresponding learning data (training image and correct image) and prepare two types of network parameters. Note that in this embodiment, a case where these two pieces of information are used will be described, but the number of pieces that can be set may be increased, or the information may be set continuously rather than discretely.

[0041] Next, in step S203, the generation unit 101c selects at least one training image from the multiple training images acquired in step S201, inputs the selected training image to the network, and calculates (generates) an output image. Selecting all of the multiple training images (inputting all of the training images to the network and using all of the outputs to update the network parameters) is called batch learning. This method imposes a significant computational load as the number of training images increases. Selecting only one training image (using only one training image to update the network parameters, with a different training image used for each update) is called online learning. This method does not increase the computational load even when the total number of training images increases, but is susceptible to noise present in a single training image. For this reason, it is preferable to use a mini-batch method, which selects a small number of training images (mini-batches) from the multiple training images and uses them to update the network parameters. For the next update, a different small number of training images is selected and used. Repeating this process can mitigate the weaknesses of batch learning and online learning.

[0042] Here, with reference to FIG. 7, the processing performed in a multi-layer neural network will be described. FIG. 7 is a diagram showing a convolutional neural network (CNN). However, this embodiment is not limited to this, and for example, a residual network can be adopted for the CNN, or a GAN (Generative Adversarial Network) or the like can also be used. Note that for simplicity, only one input training image 201 is depicted in FIG. 7, but in reality, an output image is generated for each of multiple selected training images. The training image 201 is an image in which RAW images are arranged in a three-dimensional direction for each color component.

[0043] FIG. 8 is an explanatory diagram of the color components of an image. In this embodiment, the training image is a Bayer array image as shown in FIG. 8(A). Here, RGB represents red, green, and blue, respectively. FIG. 8(B) shows a configuration in which only the color components are rearranged from the Bayer array in FIG. 8(A). Since there are two types of G, G1 and G2, each is extracted and arranged. The four images in FIG. 8(B) are arranged in a three-dimensional direction to form a four-channel image, which is the training image 201 in FIG. 7. While this process is not necessarily required, because aberration and diffraction vary with wavelength, it is easier to correct by arranging color components with the same blur. Furthermore, if RGB are arranged in the same dimension, pixels with locally different brightnesses will be mixed, which can easily reduce estimation accuracy. For this reason, it is preferable to separate the training image into each color component. Note that while the Bayer array is shown here, the same applies to other arrays (such as a honeycomb structure). Note that rearrangement of the color components is not necessary for monochrome images. In this embodiment, an example is shown in which multiple color components are learned and estimated at once, but each color may be learned and estimated individually. In Fig. 1, the training image 201 is a 4x4 four-channel image for the sake of simplicity, but the image size in both length and width is not limited to this.

[0044] In this embodiment, the training images and the ground truth images each have multiple color components arranged periodically, and a step of generating color component images consisting only of each color component of the training images or the ground truth images may be provided. Here, the step of generating color component images is performed on the training images before inputting them to the neural network, and on the ground truth images before calculating the error. CNN has a multi-layer structure, and linear and nonlinear transformations are performed in each layer. Linear transformation is expressed as the sum of the convolution of the input image (or feature map) with a filter and a bias (bias in Figure 7). Network parameters (filter weights and biases) in each layer are updated through a learning process. Nonlinear transformation is a transformation using a nonlinear function called an activation function (AF in Figure 7). Examples of activation functions include a sigmoid function and a hyperbolic tangent function. In this embodiment, the ReLU (Rectified Linear Unit) expressed by the following equation (1) is used.

[0045]

number

[0046] In equation (1), max represents a MAX function that outputs the maximum value of the arguments.

[0047] The training image 201 input to the input layer is convolved with each of the multiple filters 202 and summed with a bias in the first convolutional layer. The number of channels in each filter 202 matches that of the training image 201. If the number of channels in the training image 201 is two or more, the filter becomes a three-dimensional filter (the third dimension represents the number of channels). The filter's vertical and horizontal dimensions are arbitrary. The result of the convolution and summation is nonlinearly transformed using an activation function, and a first feature map 203 is output to the first hidden layer. Here, the number of channels in the first feature map 203 (the number of arrays in the three-dimensional direction) is the same as the number of filters 202. Next, the first feature map 203 is input to the second convolutional layer, where, as described above, the convolution with each of the multiple filters 204 and summed with a bias are calculated. The result is nonlinearly transformed, and the same process is repeated for each convolutional layer. Generally, a CNN with three or more convolutional layers is considered deep learning. The result output from the final convolutional layer is the CNN output image 211. Note that in the final convolutional layer, it is not necessary to perform nonlinear transformation using an activation function.

[0048] Next, in step S204, the generation unit 101c calculates the error between the output image 211 and the correct image 221. At this time, in this embodiment, gamma correction is performed on the output image 211 and the correct image 221 before calculating the error. Gamma correction is, for example, a process of raising the input signal value to a power, and 1 / 2.2 or the like is used as the exponent. The correct image 221, like the training image 201, is arranged for each color component and stacked in the channel direction. In this embodiment, the generation unit 101c calculates the error L using the following equation (2).

[0049]

number

[0050] In equation (2), t represents the signal value of the correct image 221, y represents the signal value of the output image 211, j represents the pixel number, N represents the total number of pixels, and g represents gamma correction. Although equation (2) uses the Euclidean norm, any other index may be used as long as it represents the difference between the correct image and the output image. Note that in this embodiment, gamma correction is performed on the output image 211 and the correct image 221 before calculating the error, but this process is not essential, and the error may be calculated without performing gamma correction.

[0051] Next, in step S205, the update unit 101d calculates the update amount of the network parameters from the error calculated in step S204, and updates the network parameters. Here, backpropagation is used. In backpropagation, the update amount is calculated based on the differentiation of the error. However, this embodiment is not limited to this.

[0052] Next, in step S206, the update unit 101d determines whether a predetermined termination condition is satisfied, i.e., whether the optimization of the network parameters is completed. The predetermined termination condition here refers to, for example, when the learning process has reached a predetermined time, when the number of parameter updates has reached a predetermined number, or when training images and correct images that are not used for parameter update are prepared and the error between the output image and the correct image falls below a predetermined value. Alternatively, the user may instruct the end of optimization. If the predetermined termination condition is not satisfied, the process returns to step S203, where the update unit 101d acquires a new mini-batch and updates the network parameters. On the other hand, if the predetermined termination condition is satisfied, the process proceeds to step S207.

[0053] In step S207, the network parameters updated by the update unit 101d are output to the storage medium 105. In this embodiment, since the network parameters are learned for each different correction intensity information, the network parameters and the corresponding correction intensity information are stored together in the storage medium 105. Through the above learning process, it is possible to obtain a multi-layer neural network that can reduce the possibility of adverse effects occurring even when correction processing is performed on an image that includes an area with poor optical performance.

[0054] Next, the estimation process executed by the image estimation device 103 will be described with reference to Fig. 9. Fig. 9 is a flowchart of the estimation process.

[0055] First, in step S301, the acquisition unit 103b acquires a captured image from the imaging device 102 or the recording medium 105. The captured image is an undeveloped RAW image. If the signal values ​​of the RAW image are encoded, the correction unit 103c performs a decoding process. The acquisition unit 103b also acquires correction intensity information from the imaging device 102 or the recording medium 105. As described above, the correction intensity information is a parameter associated with the strength of correction, such as "strong" or "weak," and can be freely selected by the user through settings in the imaging device. The correction intensity information may be stored in the captured image as header information of the captured image. In the following description, the term "header information" refers to additional information of the image and may also be footer information. Alternatively, the imaging device 102 may automatically determine the correction intensity information according to the captured scene, rather than having the user select it.

[0056] Subsequently, in step S302, the correction unit 103c acquires network parameters corresponding to the correction intensity information acquired in step S301 from the information. The network parameters are read from the storage unit 101a of the learning device 101. Alternatively, a plurality of network parameters may be stored in the storage unit 103a of the image estimation device 103 and read from the storage unit 103a. The acquired network parameters are those in which the correction intensity information acquired in step S301 matches or is closest to the correction intensity information used in the learning process.

[0057] Next, in step S303, the correction unit 103c acquires an input image to be input to the CNN from the captured image. The input image is arranged for each color component and stacked in a three-dimensional direction, similar to the training image. Note that the size of the input image in the estimation process does not necessarily have to match the size of the training image in the learning process.

[0058] Next, in step S304, the correction unit 103c generates an estimated image based on the input image and the network parameters. To generate the estimated image, the CNN shown in Fig. 7 is used, as in the learning process. However, the output image 211 in Fig. 7 becomes the estimated image, and subsequent processing such as error calculation with respect to the correct image is not performed.

[0059] Next, in step S305, the correction unit 103c determines whether estimation has been completed for a predetermined region of the captured image. If estimation has not been completed, the process returns to step S303, where the correction unit 103c acquires a new input image from the predetermined region of the captured image. In the CNN used for estimation, if the size of the output image is smaller than the input image, it is necessary to acquire an input image that overlaps the predetermined region. The predetermined region is the entire captured image or a part of it. Since the captured image is a RAW image, in addition to the image obtained by receiving light, header information (information such as the number of pixels of the image and the shooting time) and optical black information of the image sensor may be included. Since the header information and optical black are unrelated to aberration and diffraction blur, they may be excluded from the predetermined region.

[0060] Next, in step S306, the correction unit 103c combines the generated estimated images and outputs a captured image in which blur due to aberration and diffraction has been corrected. If necessary, the correction unit 103c outputs the captured image including header information and optical black information.

[0061] The above estimation process allows for correction of blurring due to aberration and diffraction while suppressing undershoot and ringing, even when the estimation process is performed on captured images that include images significantly degraded by the performance of the optical system. After the estimation process, the user can optionally perform editing such as exposure correction, and then perform a development process to obtain a final developed image. In this embodiment, a method for performing correction by switching network parameters based on correction intensity information has been described. However, multiple network parameters may be acquired and multiple output images may be generated by inputting an input image to each network. This allows for the generation of multiple output images with different correction intensities, which can then be interpolated to generate an output image with an intermediate correction intensity. Conversely, only one correction intensity information may be used, and only specific network parameters may be stored in the imaging device 102 or the recording medium 105.

[0062] Preferably, the second image has a different sharpness depending on the image height. More preferably, the second image has a first sharpness when the image height is a first image height, and a second sharpness lower than the first sharpness when the image height is a second image height higher than the first image height.

[0063] Preferably, the second image has a different sharpness depending on the luminance value. More preferably, the second image has a third sharpness when the luminance value is a first luminance value, and a fourth sharpness higher than the third sharpness when the luminance value is a second luminance value lower than the first luminance value.

[0064] Preferably, the first image has a lower sharpness than the second image.

[0065] Preferably, the method for producing training data further includes a step of acquiring a third image (original image). In the image generating step, the second image is generated by blurring the third image. More preferably, in the image generating step, the second image is generated by blurring the third image by varying the amount of blur according to the optical characteristics of the optical system. [Example]

[0066] Next, an image processing system according to a second embodiment of the present invention will be described. Fig. 10 is a block diagram of an image processing system 300 in this embodiment. Fig. 11 is an external view of the image processing system 300. The image processing system 300 includes a server 301 and an imaging device 302 connected via a network 303.

[0067] The server 301 includes a learning unit 310 and a training data generation unit (training data production device) 330. The learning unit 310 includes a memory unit 311, an acquisition unit 312, a generation unit 313, and an update unit 314, and learns network parameters for correcting blur due to aberration and diffraction using a neural network. The training data generation unit 330 includes a memory unit 331, an acquisition unit 332, and a generation unit 333. The acquisition unit 332 acquires original images and characteristic information of the optical system used to generate the training data. The generation unit 333 generates training images and reference images based on the original images. In other words, the generation unit 333 functions as an acquisition means for acquiring training images and a generation means for generating reference images with different optical characteristics from the training images. The training images and reference images generated by the generation unit 333 are stored in the memory unit 331.

[0068] The imaging device 302 captures an image of a subject space, acquires the captured image, and corrects blur due to aberration and diffraction in the captured image using the read network parameters. The imaging device 302 has an optical system 321 and an image sensor 322. The image estimation unit 323 has an acquisition unit 323a and an estimation unit 323b, and corrects the captured image using network parameters stored in a storage unit 324. The network parameters are learned in advance by a learning unit 310 and stored in a storage unit 311. The imaging device 302 reads the network parameters from the storage unit 311 via the network 303 and stores them in the storage unit 324. The captured image (output image) with blur due to aberration and diffraction corrected is stored in a recording medium 325. When a user issues an instruction regarding display of the output image, the saved output image is read and displayed on a display unit 326. Note that a captured image already saved in the recording medium 325 may be read, and blur correction may be performed by the image estimation unit 323. The above series of controls are performed by a system controller 327.

[0069] Next, the learning data generation process executed by the learning data generation unit 330 will be described with reference to Fig. 12. Fig. 12 is a flowchart of the learning data generation process. Each step in Fig. 12 is executed by each unit of the learning data generation unit 330 (storage unit 331, acquisition unit 332, generation unit 333).

[0070] In Example 1, in areas where the optical performance of the optical system is low, blurring is intentionally added to the original image when generating a ground truth image. This reduces the difference from the training image in areas where the optical performance is low, and ensures the difference between the training image and the ground truth image in areas where the optical performance is high, thereby generating training data with a reduced possibility of adverse effects. On the other hand, in this example, a method for producing training data that focuses on the luminance values ​​of the image and reduces the possibility of adverse effects such as undershoot and ringing occurring around high luminance areas will be described.

[0071] Steps S401, S402, and S403 in Fig. 12 are similar to steps S101, S102, and S103 in Fig. 1, and therefore description thereof will be omitted. Note that, when information on the brightness saturation value of the image sensor 322 is used in step S404, the acquisition unit 332 may acquire the information on the brightness saturation value from the image capture device 302 in step S401 or step S402. Alternatively, the information on the brightness saturation value may be acquired from the header information of the original image acquired in step S401.

[0072] Next, in step S404, a correction map used to generate a target image is generated. FIG. 13(A) is a schematic diagram of the correction map in this embodiment, in which the original image is divided into four regions, A, B, C, and D, according to brightness values. In FIG. 13(A), the region inside the solid line is Region D, the region inside the dashed line excluding Region D is Region C, the region inside the two-dot chain line excluding Regions D and C is Region B, and the region outside the two-dot chain line is Region A. FIG. 13(B) is a cross-sectional view of the original image corresponding to the thick solid line in FIG. 13(A). The solid line, dashed line, and two-dot chain line in FIG. 13(B) correspond to the solid line, dashed line, and two-dot chain line in FIG. 13(A), respectively. In this embodiment, brightness values ​​equal to or greater than the brightness saturation value of the image sensor 322 are Region D, brightness values ​​equal to or greater than 60% of the brightness saturation value but less than the brightness saturation value are Region C, brightness values ​​equal to or greater than 20% but less than 60% of the brightness saturation value are Region B, and brightness values ​​less than 20% of the brightness saturation value are Region A. The correction map of this embodiment is divided into four regions according to the brightness value as shown in Fig. 13(A), but the number of regions may be increased or decreased. Also, although the index is divided into stages, the index value may be used as the correction map as it is, or may be used as a continuous value.

[0073] Next, in step S405, a correct image is generated based on the correction map created in step S404. In the correction map of Figure 13(A), area A has a low brightness value and is unlikely to cause adverse effects, so the correct image corresponding to this area is used as is without blurring the original image. As a result, for high-performance areas that are originally unlikely to cause adverse effects, the amount of correction can be maintained by maintaining the difference between the training image and the correct image.

[0074] Next, for region B, the brightness is higher than region A but lower than region C, so a slight blur is added to the original image. For example, a PSF corresponding to each pixel is reduced by 20%, and the reduced PSF is applied to the original image to generate a correct image. Next, for region C, the brightness is higher than region B, so the amount of blur is further increased. For example, a PSF reduced by 50% from the PSF applied to the training image is generated, and the reduced PSF is applied to the original image to generate a correct image. Then, for region D, the brightness is the highest, so the original image is blurred without reduction using the PSF applied to the training image to generate a correct image. That is, for the correct image, if the brightness value is the first brightness value, the sharpness is the third sharpness (third blur amount), and if the brightness value is the second brightness value lower than the first brightness value, the sharpness is the fourth sharpness higher than the third sharpness (fourth blur amount smaller than the third blur amount).

[0075] Note that the amount of reduction of the PSF for the correction map described above is an example and is not limited to this. Although region A is not blurred, the entire region may be blurred. However, the amount of blur applied to the reference image must be equal to or less than the amount of blur applied when generating the training images. In other words, the training images have lower sharpness (greater blur) than the reference image. Therefore, when blurring the reference image, a PSF equal to or reduced from the PSF used when generating the training images is used. Furthermore, if the training images are reduced after blurring in step S403, the reference image is also reduced by the same factor. If the original image has a signal value higher than the luminance saturation value of the image sensor 322, the signal is clipped at the luminance saturation value of the image sensor 322.

[0076] Subsequently, in step S406, a pair of the training image generated in step S403 and the correct image generated in step S405 is stored as learning data in storage unit 331. Step S406 is similar to step S106, and therefore a detailed description thereof will be omitted.

[0077] In this embodiment, the learning process is performed by the learning unit 310, and the estimation process is performed by the image estimation unit 323. In the learning process in this embodiment, learning data (correct image and training image) created in the flow of FIG. 12 is used. Other details are the same as those in the flowchart shown in FIG. 6 of the first embodiment, so details are omitted. Furthermore, the estimation process is also the same as those in the flowchart shown in FIG. 9 of the first embodiment, so explanations are omitted.

[0078] As described above, by generating a correction map according to brightness values ​​and generating a ground truth image based on that correction map, training data can be produced that can reduce the likelihood of problems such as undershoot and ringing occurring around high-brightness areas. By generating network parameters using the training data created in this way and using that information to perform estimation processing, it is possible to accurately correct blurring due to aberration and diffraction while reducing the likelihood of the aforementioned problems occurring even in captured images with high-brightness areas. While the correction map is divided into four parts in this embodiment, it may also be divided into two parts: brightness saturated areas and non-brightness saturated areas. In this case, for example, if a ground truth image is generated by blurring the original image with a PSF applied to the training image for the brightness saturated areas, there will be no difference between the training image and the ground truth image for the brightness saturated areas. By training a network using this training data and performing estimation processing using the network parameters, it is possible to correct areas other than brightness saturated areas without correcting the brightness saturated areas. In particular, brightness saturated areas in captured images are areas where information is missing, and attempting to correct them directly, like other areas, is likely to result in problems. Therefore, by performing this type of training, it is possible to achieve a network that can output high-quality estimated images. In this embodiment, the correction map is generated using the luminance values ​​of the original image, but the correction map may also be generated based on changes in luminance values. In this case, a differential image can be generated by applying a differential filter, Prewitt filter, or Sobel filter to the original image. Alternatively, a second-order differentiated image may be generated using a Laplacian filter or the like. In this way, by generating a correction map based on changes in luminance, it is possible to generate training data that can suppress adverse effects that occur near high-contrast edges.

[0079] Furthermore, in the first embodiment, the generation of a correction map according to performance was described, and in the present embodiment, the generation of a correction map according to the luminance of an image was described. However, both of these may be performed. That is, in step S404, not only is a correction map according to luminance created, but also a correction map according to optical information of the optical system is created, and these two correction maps are integrated. For example, when integrating the correction maps of FIG. 5 and FIG. 13(A), an integrated map can be created by adjusting the area to be more blurred. That is, a method is to change the area that is area B in FIG. 5 and area D in FIG. 13(A) to area B. Alternatively, a correction map may be created by integrating the average values ​​of the PSF reduction rates in each correction map. The integration method is not limited to these methods, and another method may be used to generate an integrated correction map. By using two correction maps in this way, learning data that takes into account the influence of both the performance of the optical system and the luminance value of the image can be generated. [Example]

[0080] Next, an image processing system according to a third embodiment of the present invention will be described. The configuration and learning process of the image processing system according to this embodiment are the same as those of the first embodiment, but the learning data generation process (method for producing learning data) and estimation process are different.

[0081] The method for producing training data executed in this embodiment will be described with reference to Fig. 14. Fig. 14 is a flowchart of the method for producing training data.

[0082] In the first embodiment, a correction map was created and a reference image was generated based on the correction map. However, in the present embodiment, a reference image and training images are generated without using a correction map. That is, the correction amount of the reference image and training images relative to the original image in the present embodiment is constant within the image. Steps S601 and S602 are the same as steps S101 and S102 in the first embodiment, respectively, and therefore will not be described here.

[0083] In step S603, the generation unit 108c generates a training image (first image) using the optical information acquired in step S602. The training image in this embodiment corresponds to, for example, an image (an image corresponding to a certain image height) obtained by extracting only a portion of an image (an image corresponding to the entire angle of view of the optical system) acquired by photographing using an optical system. In the first embodiment, a method of linearly interpolating the PSF to vary the PSF for each pixel position was described. However, in this embodiment, a uniform blur is applied without changing the PSF depending on the pixel position. Instead, in this embodiment, in order to accommodate variations in optical performance (optical information) depending on the image height, the number of original images is increased to increase the number of generated answer images and training images. If it is necessary to accommodate multiple lens states, image heights, and azimuths during estimation, pairs of answer images and training images are generated by mixing these.

[0084] Next, in step S604, the generation unit 108c generates a correct image (second image) that pairs with the training image generated in step S603. The correct image in this embodiment corresponds to, for example, an image (an image corresponding to a certain image height) that is extracted from an image (an image corresponding to the entire angle of view of the optical system) acquired by photographing using an optical system. The region of the original image extracted as the correct image may be equivalent to the paired training image. In this embodiment, the performance of the optical system is used as the peak value of the PSF to generate the correct image. The higher the performance of the optical system, the higher the peak value of the PSF. Conversely, the lower the performance, the lower the peak value. Therefore, this is suitable as an index for generating a correct image from an original image. Note that, as described in the second embodiment, the correct image may be generated based on the presence or absence of a brightness saturation area rather than the optical performance.

[0085] FIG. 15 is a simulation diagram showing the relationship between the performance of the optical system (optical performance) and the amount of blurring of the PSF applied to the original image to generate a ground truth image, and is divided into three regions (sections A, B, and C) according to the performance of the optical system. In this embodiment, the horizontal axis of FIG. 15 represents the optical performance corresponding to the peak value of the PSF. When the optical performance belongs to section A, the peak value of the PSF is high, and it can be considered that the performance of the optical system is sufficiently exhibited. When the optical performance of the region of the original image extracted as the ground truth image belongs to section A, the original image is used as is as the ground truth image. In this way, when the optical performance is high and the possibility of the above-mentioned adverse effects occurring is low, the deviation between the training image and the ground truth image is maintained. As a result, when the optical performance belongs to section A, learning is performed so that the amount of correction for the training image (input image) is relatively large.

[0086] If the optical performance of the region of the original image extracted as the correct image falls within section B, the amount of blurring of the PSF is changed according to the PSF peak value. As shown in Figure 15, the amount of blurring applied to the original image is adjusted so that it is small when the PSF peak value is high, and conversely, the amount of blurring is increased when the peak value is low. The change in region B in Figure 15 is represented by a straight line, but it may also be represented by a nonlinear function such as a quadratic function or an exponential function.

[0087] Section C is the region where the performance of the optical system is lowest. When the optical performance of the region of the original image extracted as the correct image falls within section C, the correct image is generated using the PSF used to generate the training image as is. In other words, the training image and the correct image are generated using the same method. When the optical performance is low, as in section C, the possibility of adverse effects occurring increases. Therefore, by matching the training image with the correct image, the correction effect on the training image (input image) through learning is suppressed. This makes it possible to suppress adverse effects. In this way, by adjusting the amount of blurring added when generating a correct image from an original image according to the performance of the optical system, it is possible to maximize the effect under conditions where adverse effects are unlikely to occur, and conversely, to suppress the effect under conditions where adverse effects are likely to occur, thereby prioritizing the suppression of adverse effects.

[0088] Here, the peak value of the PSF is used as an index for adjusting the amount of blurring. However, frequency characteristics can also be used as an index. In this case, the OTF is generated by Fourier transforming the PSF, and the absolute value is taken to calculate the MTF. When using the MTF as an index, it can be the MTF value at a specific frequency (e.g., the MTF value at half the Nyquist frequency), or the integrated value over a certain interval (the integrated value up to the Nyquist frequency) or the average value. Another index can be the asymmetry of the blur, which can be quantified and the amount of blur adjusted accordingly. Generally, the more asymmetric the PSF, the more likely adverse effects will occur. To quantify asymmetry, for example, the LSFs of two orthogonal cross sections, such as the meridional and sagittal directions, are calculated from the PSF, and the difference between the peak values ​​of the two LSFs is used as an index. In this case, a larger difference in peak values ​​indicates an asymmetric shape, so the amount of blurring should be increased; conversely, a smaller difference should result in a smaller amount of blurring. Instead of the LSF, evaluation may be performed using frequency characteristics, and the average value of the MTF characteristics in the meridional and sagittal directions or the difference value at a specific frequency may be used as an index of asymmetry. Alternatively, the skewness of the PSF may be used. The larger the absolute value of the skewness, the greater the asymmetry, and therefore the greater the amount of blurring. Conversely, the smaller the absolute value, the smaller the asymmetry, and therefore the less the amount of blurring. Furthermore, the image height corresponding to the correct image (the position of the correct image in the image corresponding to the full angle of view of the optical system) may be used as a simpler index for adjusting the amount of blurring. This is because, as described in the first embodiment, there is generally a correlation between higher image height and reduced optical performance.

[0089] If the high frequency components of the original image are insufficient, the original image may be reduced to generate a correct image. At this time, the training images are also reduced in the same way.

[0090] Subsequently, in step S605, the pair of the training image generated in step S603 and the correct image generated in step S604 is stored as learning data in the storage unit 108a.

[0091] The above is the method for producing the learning data in this embodiment. By learning using the thus-created ground truth image and training images, it is possible to generate a learning model that achieves highly accurate correction while reducing the possibility of adverse effects occurring.

[0092] Next, the estimation process executed by the image estimation device 103 will be described with reference to Fig. 16. Fig. 16 is a flowchart of the estimation process.

[0093] In the first embodiment, a method for acquiring network parameters based on correction intensity information was described. In the present embodiment, a method for acquiring a network model based on resource information of the image estimation device 103 is described. In this embodiment, the network model includes network parameters and a network configuration (architecture). As described in the first embodiment, the size of the input image input to the CNN may be different from that during learning. If a partial region of the captured image is used as the input image, a corrected captured image can be generated by synthesizing the output estimated image. When performing estimation processing in this manner, increasing the size of the input image reduces the number of divisions of the captured image, allowing for faster estimation processing. However, increasing the size increases memory (RAM) usage. Exceeding the memory capacity of the image estimation device can actually reduce processing speed or result in an abnormal termination. In other words, for more efficient processing, it is better to set the input size to match the image estimation device. An example of this is described here.

[0094] First, in step S701, the acquisition unit 103b acquires a captured image from the imaging device 102 or the recording medium 105. The captured image is an undeveloped RAW image. If the signal values ​​of the RAW image are encoded, the correction unit 103c executes a decoding process. The acquisition unit 103b also acquires memory information (resource information) of the image processing device. The acquired memory information may be the physical memory capacity, but since the available capacity changes depending on other processes, it is preferable to acquire the free capacity. If the impact of other processes is small, the physical memory capacity may be used, or the available memory capacity may be estimated from the physical memory and the memory capacity in use.

[0095] Next, in step S702, the correction unit 103c acquires an appropriate network model based on the memory information acquired in step S701. Here, in the case of a network model with a fixed input image size, an appropriate network model must be selected. For example, if the available memory capacity acquired in step S701 is sufficiently larger than the memory capacity used by the CNN, a network model with a large input image size is acquired. Conversely, if the available memory capacity is small, a network model with a small input image size is acquired. Since the network parameters do not need to be changed, only the network configuration may be acquired based on the memory information. While various network model formats exist, switching between them in this manner can efficiently execute correction processing when the input image size is fixed. For a network model with a variable input image size, there is no need to change the network model; only the input image size needs to be set appropriately. While the selection of two network models has been described here, increasing the variety of network models enables the selection of a network model more suited to the image processing device. This network model selection may not only be automatically determined based on memory information and a corresponding network model selected, but may also be based on user input information. For example, by allowing users to select between a high-performance mode and a lightweight mode and taking the mode information into account, processing more tailored to the user's needs can be achieved.

[0096] Steps S703 and after are the same as steps S303 and after in the first embodiment, and therefore detailed description thereof will be omitted.

[0097] The above is the estimation process in this embodiment. By acquiring a network model and configuration based on memory information of the image processing device, it is possible to realize estimation processing suited to the processing environment.

[0098] In this embodiment, the training images and the correct image correspond to a portion of the image height in an image corresponding to the entire angle of view of the optical system. In this case, during learning, information about the image height of the training images may be input to the neural network in addition to the training images for learning. This makes it possible to more accurately learn the amount of correction corresponding to the position in the image corresponding to the entire angle of view of the optical system. Furthermore, when information about the image height of the training images is input to the neural network for learning, information about the image height of the input image is also input to the neural network in the estimation step in the same manner as during learning.

[0099] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0100] According to each embodiment, it is possible to provide a method for producing training data that can achieve high-precision correction while reducing the possibility of adverse effects occurring, even when the image input to the neural network includes a highly bright subject or a subject that is significantly blurred due to aberrations in the optical system.

[0101] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and various modifications and changes are possible within the scope of the invention. [Explanation of symbols]

[0102] 108 Learning Data Production Device 108c Generation unit (acquisition means, generation means)

Claims

1. A method for generating training data used in training a neural network, comprising: acquiring an original image; generating training images based on the original image and information related to a first blur; generating a ground truth image based on the original image and information related to the second blur; A generating method, characterized in that the information about the second blur is generated based on the information about the first blur.

2. The generation method according to claim 1, characterized in that the information regarding the second blur indicates a smaller amount of blur in an area of ​​a second luminance value lower than the first luminance value in the original image than in an area of ​​the first luminance value.

3. 3. The method according to claim 1, wherein the information about the second blur is determined based on the information about the first blur.

4. 4. The method according to claim 1, wherein the information about the first blur is a plurality of values ​​that differ depending on the image height.

5. The method of claim 4 , wherein the plurality of values ​​are values ​​based on an optical transfer function or a point spread function.

6. 6. The generation method according to claim 1, wherein the second blur information indicates a different amount of blur depending on a position in the original image.

7. The generation method according to claim 6, characterized in that the information regarding the second blur indicates a greater amount of blur at a position corresponding to a second image height higher than a first image height of the original image than at a position corresponding to the first image height.

8. 8. The generation method according to claim 1, wherein the information about the first blur indicates a different amount of blur depending on a position in the original image.

9. 9. The generation method according to claim 8, wherein the information about the first blur indicates a smaller amount of blur at a position corresponding to a second image height higher than the first image height of the original image than at a position corresponding to the first image height of the original image.

10. A training method for training a neural network using training data generated by the generation method according to any one of claims 1 to 9, comprising: generating processed images by inputting the training images into the neural network; A learning method comprising a step of updating parameters of the neural network based on the correct image and the processed image.

11. A generation device that generates training data used for training a neural network, comprising: a first generating means for acquiring an original image; a second generating means for generating a training image based on the original image and information related to the first blur; a third generating means for generating a ground truth image based on the original image and the second blur information; A generating device, characterized in that the information about the second blur is generated based on the information about the first blur.

12. A learning device that performs neural network learning using training data generated by the generation method according to any one of claims 1 to 9, a generating means for generating a processed image by inputting the training image into the neural network; A learning device comprising: an update means for updating parameters of the neural network based on the correct image and the processed image.

13. A program causing a computer to execute the generation method according to any one of claims 1 to 9.

14. A storage medium storing the program according to claim 13.

15. A generation method for generating a trained model by training a neural network using training data generated by the generation method according to any one of claims 1 to 9, generating processed images by inputting the training images into the neural network; and updating parameters of the neural network based on the ground truth image and the processed image.

16. An image processing method for generating an estimated image using a trained model obtained by the generation method according to claim 15, An image processing method comprising a step of generating an estimated image by inputting an input image into the trained model.

17. the input image is an image obtained by imaging, 17. The image processing method according to claim 16, wherein the estimated image is an image in which blurring in the input image that occurs during imaging has been corrected by processing using the neural network.

18. A program causing a computer to execute the image processing method according to claim 16 or 17.

19. An image processing device that generates an estimated image using a trained model obtained by the generation method according to claim 15, An image processing device comprising: a means for generating an estimated image by inputting an input image to the trained model.

20. obtaining an input image; inputting the input image into a neural network to generate an estimated image; The neural network is generated by training using training images and ground truth images; the training images are generated based on the original image and information related to the first blur; the ground truth image is generated based on the original image and information related to the second blur; 10. An image processing method, wherein the information regarding the second blur is generated based on the information regarding the first blur.

Images