Learning data manufacturing method, learning method, and learning data manufacturing device

Abstract

The present disclosure relates to a learning data manufacturing method, a learning method, and a learning data manufacturing apparatus. The learning data manufacturing method is used for causing a neural network to perform learning. The learning data manufacturing method includes a first acquisition step configured to acquire an original image, a second acquisition step configured to acquire a first image generated by adding blur to the original image as a training image, and a third acquisition step configured to acquire a second image generated by adding blur to the original image as a ground truth image. An amount of blur added to the second image is smaller than an amount of blur added to the first image.

Description

Technical Field

[0001] This invention relates to a method for generating learning data for deep learning and a learning method using deep learning. Background Technology

[0002] Japanese Patent Publication No. (“JP”) 2019-121252 discloses a method for suppressing undershoot and ringing by taking into account the effect of gamma correction when enabling a multilayer neural network to perform learning using RAW images as input, where each of undershoot and ringing is caused by increasing resolution or contrast (sharpening). X. Mao, C. Shen, Y. Yang, “Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections”, https: / / arxiv.org / abs / 1606.08921 (hereinafter referred to as Mao et al.) discloses a network configuration that is generally applicable to various regression problems. Mao et al. further disclose methods for performing upsampling, JPEG deblocking (removal of compressed noise), denoising, non-blind deblurring, or restoration on input images using a network.

[0003] However, the methods disclosed in JP 2019-121252 and by Mao et al. may fail to suppress undershoot, ringing, etc., depending on the input image, and these side effects may occur in the corrected image (estimated image). Specifically, these side effects may occur when the input image includes high-brightness objects, when the input image includes objects that are significantly blurred due to aberrations of the optical system, or especially when the input image includes objects with high brightness values ​​and brightness saturation. When such images are used as training data, side effects such as undershoot and ringing may occur depending on the amount of correction. Summary of the Invention

[0004] The present invention provides a method for manufacturing learning data, etc., which can achieve highly accurate correction for each of the learning data, etc., even when the image input to the neural network includes a high-brightness object or an object that is greatly blurred due to the aberration of the optical system, while reducing the possibility of side effects.

[0005] A method for generating learning data, as one aspect of the present invention, is used to enable a neural network to perform learning. The method includes: a first acquisition step configured to acquire an original image; a second acquisition step configured to acquire a first image, generated by adding blur to the original image, as a training image; and a third acquisition step configured to acquire a second image, generated by adding blur to the original image, as a ground truth image. The amount of blur added to the second image is less than the amount of blur added to the first image.

[0006] As another aspect of the present invention, the learning method enables the neural network to perform learning using learning data generated by the aforementioned learning data manufacturing method. This learning method includes the following steps: generating a processed image by inputting a first image as a training image into the neural network, and updating the neural network based on a second image and the processed image.

[0007] As another aspect of the present invention, a learning data manufacturing apparatus manufactures learning data for learning neural networks. The learning data manufacturing apparatus includes: a first acquisition unit configured to acquire an original image; a second acquisition unit configured to acquire a first image generated by adding blur to the original image; and a third acquisition unit configured to acquire a second image generated by adding blur to the original image. The amount of blur added to the second image is less than the amount of blur added to the first image.

[0008] As another aspect of the present invention, the learning apparatus includes the above-described learning data generation apparatus, a generation unit configured to generate a processed image by inputting a first image as a training image into a neural network, and an update unit configured to update the neural network based on a second image and the processed image.

[0009] Another aspect of the present invention is a method for manufacturing a learned model by having a neural network perform learning using learning data generated by the aforementioned method for manufacturing learning data. The method for manufacturing a learned model includes the following steps: generating a processed image by inputting a first image as a training image into a neural network, and having the neural network perform learning based on a second image and the processed image.

[0010] Other features of the invention will become clear from the following description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating a method for generating learning data according to the first embodiment.

[0012] Figure 2 This is a block diagram illustrating an image processing system according to the first and third embodiments.

[0013] Figure 3 This is an example of the appearance of an image processing system according to the first and third embodiments.

[0014] Figure 4 These are explanatory diagrams relating to optical information according to each embodiment.

[0015] Figure 5 These are explanatory diagrams related to the calibration diagrams according to the first and third embodiments.

[0016] Figure 6 This is a flowchart illustrating the learning phase according to each embodiment.

[0017] Figure 7 This is a diagram illustrating a convolutional neural network according to each embodiment.

[0018] Figure 8A and Figure 8B These are explanatory diagrams relating to the color components of the images according to each embodiment.

[0019] Figure 9 This is a flowchart illustrating the estimation stage according to the first and second embodiments.

[0020] Figure 10 This is a block diagram illustrating an image processing system according to a second embodiment.

[0021] Figure 11 This is an example of the appearance of an image processing system according to a second embodiment.

[0022] Figure 12 This is a flowchart illustrating a method for generating learning data according to the second embodiment.

[0023] Figure 13A and Figure 13B This is an explanatory diagram related to the calibration diagram according to the second embodiment.

[0024] Figure 14 This is a flowchart illustrating a method for generating learning data according to the third embodiment.

[0025] Figure 15 This is an explanatory diagram illustrating the optical performance and blur amount according to the third embodiment.

[0026] Figure 16 This is a flowchart illustrating the estimation process according to the third embodiment. Detailed Implementation

[0027] A description of embodiments of the invention will now be given with reference to the accompanying drawings. Corresponding elements in the corresponding drawings will be indicated by the same reference numerals, and their descriptions will be omitted.

[0028] First, a definition of each term used in each embodiment will be given. Each embodiment relates to a method for solving regression problems and estimating various output images from input images using deep learning. Deep learning is machine learning that uses multi-layered neural networks. When learning network parameters (weights and biases) from pairs of large numbers of training images and corresponding ground truth images (the desired outputs), it is possible to perform highly accurate estimations even for unknown input images.

[0029] Image processing using multilayer neural networks comprises two stages: a processing stage for updating network parameters (weights and biases) and a processing stage for performing estimation on unknown inputs using the updated parameters. In the following text, the former will be referred to as the learning stage, and the latter as the estimation stage.

[0030] Next, the names of the images used in the learning and estimation phases are defined. The image input to the network will be called the input image, and in particular, the input image whose ground truth image is known and used in the learning phase will be called the training image. The image output from the network will be called the output image, and in particular, the output image during the estimation phase will be called the estimated image. The network's input image and ground truth image are RAW images. A RAW image is undeveloped image data output from an image sensor, and the light intensity and signal value of each pixel have a substantially linear relationship. The RAW image is developed before the user views the image, and gamma correction is performed during development. Gamma correction is, for example, a process of raising the input signal value to a power, and using 1 / 2.2, etc., as the exponent. In the method of generating the learning data (learning image) in each embodiment, an adjusted ground truth image is generated, and this adjusted ground truth image is used in the learning process to achieve highly accurate correction while suppressing side effects. In each embodiment, the image that is substantially undegraded and is the basis for generating the ground truth image or training image is called the original image.

[0031] Before describing each embodiment in detail, the key points of the invention will be described. The invention implements a neural network and provides the necessary learning data for it, which can correct images including high-brightness objects or objects significantly blurred due to optical aberrations, while preventing side effects. Instead of using the original image as ground truth image for learning data, the invention uses an adjusted ground truth image. An adjusted ground truth image is an image in which blur is added to potentially side-effect parts of the original image (such as high-brightness parts, high-contrast parts, and areas with large aberrations). This adjustment makes the sharpness vary between the ground truth image and the training image depending on the location of the image. Alternatively, the training image can be generated by sharpening. Any of these methods can adjust the amount of blur (sharpness) in the ground truth image to an intermediate amount of blur between the original image and the training image. Thus, in areas where side effects may occur, the difference between the ground truth image and the training image can be made smaller than the difference between the original image and the training image. When learning is performed using ground truth images and training images both generated in this way, it is possible to generate a learning model that achieves highly accurate correction while reducing the likelihood of side effects.

[0032] First Embodiment

[0033] A description of an image processing system according to a first embodiment of the present invention will be given. In this embodiment, firstly, learning data for learning via a multi-layer neural network is generated. Subsequently, learning is performed using the generated learning data, and correction processing is performed using the learned model.

[0034] Figure 2 This is a block diagram illustrating the image processing system 100 in this embodiment. Figure 3 This is an external view illustrating an image processing system 100. The image processing system 100 includes a learning device (image processing device) 101, an image acquisition device 102, an image estimation device (image processing device) 103, a display device 104, a recording medium 105, an output device 106, a network 107, and a learning data generation device (image processing device) 108.

[0035] The learning data generation apparatus 108 is an apparatus configured to generate learning data used in the learning phase, and includes a memory 108a, an acquirer 108b, and a generator 108c. The acquirer 108b is configured to acquire an original image (a third image) and characteristic information about the optical system, such as the image height used to generate the learning data. The generator 108c is configured to generate a training image (a first image) and a ground truth image (a second image) based on the original image. That is, the generator 108c functions as an acquirer unit for acquiring the training image and a generator unit for generating a ground truth image having optical characteristics different from those of the training image. The training image and the ground truth image generated by the generator 108c are stored in the memory 108a. (Referring later...) Figure 1 The flowchart provides a detailed description of the generation of learning data.

[0036] The learning device 101 is an image processing apparatus configured to perform a learning phase, and includes a memory 101a, an acquirer 101b, a generator (generation unit) 101c, and an updater (update unit) 101d. The acquirer 101b is configured to acquire training images and ground truth images. The generator 101c is configured to input the training images (first images) into a multi-layer neural network and generate an output image (processed image). The updater 101d is configured to update the network parameters of the neural network based on the difference (error) between the output image generated by the generator 101c and the ground truth image. A detailed description of the learning phase will be given later with reference to a flowchart. The learned network parameters are stored in the memory 101a.

[0037] Image acquisition device 102 includes an optical system 102a and an image sensor 102b. The optical system 102a is configured to collect light entering the image acquisition device 102 from object space. The image sensor 102b is configured to acquire a captured image by receiving light from an optical image (object image) formed via the optical system 102a (i.e., by photoelectric conversion of the optical image). The image sensor 102b is, for example, a CCD (charge-coupled device) sensor, a CMOS (complementary metal-oxide-semiconductor) sensor, etc. The captured image acquired by the image acquisition device 102 includes blurring caused by aberrations or diffraction of the optical system 102a and noise caused by the image sensor 102b.

[0038] Image estimation device 103 is a device configured to perform an estimation phase and includes a memory 103a, an acquirer 103b, and a correction unit (estimation unit) 103c. Image estimation device 103 is configured to acquire a captured image, perform deblurring (correcting blur) while suppressing side effects, and generate an estimated image. A multi-layer neural network is used for blurring, and information about network parameters is read from memory 103a. The network parameters are learned by learning device 101. Image estimation device 103 is configured to read network parameters from memory 101a in advance via network 107 and store the network parameters in memory 103a. The stored network parameters can be numerical values ​​or encoded forms. A detailed description of the learning of network parameters and the deblurring process using the network parameters will be given later.

[0039] The output image is output to at least one of the display device 104, the recording medium 105, and the output device 106. The display device 104 is, for example, a liquid crystal display or a projector. The user can perform editing tasks while viewing the image being processed via the display device 104. The recording medium 105 is, for example, a semiconductor memory, a hard disk, or a server on a network. The output device 106 is, for example, a printer. The image estimation device 103 has the function of performing developing processes and other image processing as needed.

[0040] Next, we will refer to Figure 1 A description is given of a method for manufacturing learning data executed by the learning data manufacturing apparatus 108. Figure 1 This is a flowchart illustrating the method for generating learning data. Figure 1 Each step is performed by each part of the learning data manufacturing apparatus 108 (memory 108a, acquirer 108b, and generator 108c).

[0041] First of all, Figure 1 In step S101, the acquirer 108b acquires multiple raw images (third images) (first acquisition step). The raw images can be pre-stored in and retrieved from memory 108a, or they can be retrieved from an external device (not shown). In this embodiment, the raw images are undeveloped RAW images. However, this embodiment is not limited to this and can also be developed images. The multiple raw images are images that include various objects (i.e., edges of various intensities and directions, various textures, various gray levels, various flat areas, etc.). The raw images can be captured images or images generated by CG (computer graphics). The raw images can have signal values ​​higher than the brightness saturation value of the image sensor 102b because when image acquisition is performed by the image acquisition device 102 under specific exposure conditions, the actual object can be an object with a signal value greater than the brightness saturation value.

[0042] Subsequently, in step S102, the acquirer 108b acquires the optical information of the optical system 102a used to blur (add blur) the original image acquired in step S101 (fourth acquisition step). The optical system 102a causes different aberrations and diffractions depending on multiple lens states (zoom, aperture, and focus distance states), image height, and azimuth. Therefore, it is necessary to add different blurs to each original image caused by aberrations or diffractions depending on the lens state, image height, and azimuth. In step S102, the PSF (Point Spread Function) of the optical system 102a is acquired, which is used for blurring in the next step S103. At this time, the optical information acquired by the acquirer 108b can be two-dimensional data corresponding to the distribution of the PSF, or it can be a set of one-dimensional vectors obtained by decomposing the PSF using singular value decomposition or the like. The PSF can be fitted and approximated by a specific function, and multiple coefficient data obtained during approximation can be acquired as optical information. In this case, the PSF can be obtained by reconstructing using the acquired coefficient data and the approximation function. In this embodiment, the use of PSF for blurring will be described, but OTF (Optical Transfer Function) can be used instead of PSF. When acquiring OTF as optical information, as with acquiring PSF, OTF can be acquired as two-dimensional data, a one-dimensional vector, or coefficient data.

[0043] Figure 4 This illustrates the positional relationship between the original image and the PSF. Figure 4 In the diagram, the rectangle represents the original image, while the white circle indicates the location of the PSF to be acquired or generated. When the PSF changes within the original image, the acquisition or generation process is adjusted accordingly. Figure 4 The PSF corresponding to each position is shown. Figure 4 In this model, the PSF position is 9×9, but it can be reduced for weight reduction or increased for higher accuracy.

[0044] Subsequently, in step S103, generator 108c generates a training image (first image) (second acquisition step) using the optical information acquired in step S102. A PSF is generated when the acquired optical information is not a PSF but coefficient data, etc., which forms the basis of the PSF. Afterward, generator 108c blurs the original image using the PSF. In this embodiment, blurring is performed by convolving the PSF as a blur filter into the original image. When changing the degree of blurring based on pixel height, the PSF can be changed for each pixel of the original image, and a linear sum of the PSF and the corresponding region of the original image can be calculated for each pixel in all pixels. For example, when using... Figure 4To obtain the pixel value after blurring the pixels located between the white circles (the pixels of interest) at the positions shown in the diagram, the PSF corresponding to the four white circles surrounding the neighboring pixels is first obtained. Then, the four obtained PSFs are interpolated to calculate the PSF corresponding to the pixel of interest. A linear sum of the PSFs and the original image is calculated, and this value is assumed to be the value of the pixel of interest after blurring. Known interpolation methods such as bilinear, bicubic, and nearest neighbor interpolation can be used.

[0045] Next, the blurred original image is reduced. When the signal value of the original image is higher than the brightness saturation value of the image sensor 102b, the signal is clipped to the brightness saturation value of the image sensor 102b. In particular, when the captured image is used as the original image, it is already blurred due to aberrations or diffraction. The effect of blurring can be reduced by reducing the original image, thereby obtaining an image with high resolution (high quality). When the original image contains a sufficient amount of high-frequency components, reduction may not be performed.

[0046] Subsequently, in step S104, generator 108c generates a calibration map for generating a ground truth image. Figure 5 This is a diagram illustrating the correction diagram in this embodiment. Figure 5 In the correction map, the original image is divided into four regions, A, B, C, and D, based on image height. Figure 5 In the diagram, the area inside the solid line represents region A, the area enclosed by the solid and dashed lines represents region B, the area enclosed by the dashed line and the double-dotted line represents region C, and the area outside the double-dotted line represents region D. In this embodiment, a ground truth image is generated by blurring the original image based on the performance of the optical system. Regarding the performance of the optical system used to generate the calibration map, any performance-related metrics can be used; however, this embodiment uses the OTF of the optical system to generate the calibration map.

[0047] First, through the comparison with Figure 4 The PSF corresponding to the white circle in the image is subjected to a Fourier transform to generate the OTF at the location corresponding to the white circle. Next, for each OTF, the absolute value is obtained to calculate the MTF (Modulation Transfer Function). Subsequently, the average value of each MTF up to the Nyquist frequency of the image sensor is calculated, and this value is used as an index to generate the calibration map. The index values ​​at locations other than the white circles are obtained by interpolation with the index values ​​corresponding to nearby white circles. Although the index is calculated here using the average MTF up to the Nyquist frequency, the average MTF can be calculated in different frequency bands, or a specific frequency can be used as the index. In this embodiment, the calibration map is used to blur the ground truth image, so the calibration map corresponds to the amount of blur in each pixel.

[0048] exist Figure 5 In this system, the calibration map is divided into four regions based on image height, and further divided based on the average MTF value used as an index. For example, region A has an average MTF of 0.8 or higher, region B has an average MTF of 0.6 or higher but less than 0.8, region C has an average MTF of 0.4 or higher but less than 0.6, and region D has an average MTF less than 0.4. When the optical system 102a is a coaxial system, the optical performance is symmetrical with respect to the optical axis, therefore the calibration map is also rotationally symmetrical, as shown below. Figure 5 As shown in the diagram. Generally, in optical systems, performance tends to be higher as the position is closer to the center of the image, i.e., closer to the optical axis (as image height decreases), and lower as the position is closer to the periphery of the image, i.e., farther from the optical axis (as image height increases). Therefore, the correction pattern may have the following characteristics: Figure 5 The distribution is shown in the figure. In other words, in the ground truth image, when the image height is the first image height, the sharpness is the first sharpness (the first blur amount), and when the image height is the second image height which is higher than the first image height, the sharpness is the second sharpness which is lower than the first sharpness (the second blur amount which is greater than the first blur amount).

[0049] This symmetry is lost when considering factors such as the optical low-pass filter, the pixel aperture of the image sensor 102b, and variations during the manufacturing of the optical system 102a; therefore, the calibration map may not necessarily possess circular symmetry. This step calculates the OTF to compute the index, but as long as the index reflects the optical performance, another value can be used as the index. For example, the peak value of the PSF can be used as the index. The peak value of the PSF is the integral of the real part of the OTF, and therefore this peak value is an index related to the MTF. Alternatively, the sum of squares of the PSF can be calculated using Parseval's theorem, and the average of the sum of squares of the MTF can be calculated. When the index is calculated directly from the PSF without calculating the OTF and the calibration map is generated in this way, the Fourier transform becomes unnecessary, and it is possible to reduce the computational load. Figure 5 As shown, the calibration graph in this embodiment is divided into four regions, but the number of regions can be increased or decreased. The index is divided by stage, but the index values ​​can be used as is or can be used as continuous values.

[0050] Subsequently, in step S105, generator 108c generates a ground truth image (second image) based on the correction map generated in step S104 (third acquisition step). When blur is corrected by deep learning (performing deblurring), the greater the difference between the training image in the learning data used for learning and the ground truth image, the more likely side effects will occur during estimation. The lower the performance of the optical system 102a in the region, the greater the degree to which the training image is blurred in step S103. Therefore, the lower the performance of the region when the original image is used as is, the more different the training image and the ground truth image are from each other. By reducing this difference, this embodiment reduces the possibility of side effects occurring. Figure 5 In the calibration image, since region A is a high-performance region, the original image is used as is without being blurred to the ground truth image corresponding to this region. Thus, the calibration amount can be maintained by preserving the difference between the training image and the ground truth image for high-performance regions that are unlikely to cause side effects.

[0051] Next, regarding region B, whose performance is lower than that of region A but higher than that of region C, the original image is slightly blurred. A ground truth image is generated, for example, by generating a PSF reduced to 30% of the PSF corresponding to each pixel and adding the reduced PSF to the original image. Since region C's performance is lower than that of region B, the blurring amount is further increased. For example, a ground truth image is generated by generating a PSF reduced to 60% of the PSF added to the training image and adding the reduced PSF to the original image. Since region D has the lowest performance, a ground truth image is generated by adding the PSF added to the training image to the original image without reduction. The reduction in PSF used for the correction image is an example and is not limited to this. Region A is described as an unblurred region, but the entire region can be blurred. However, the amount of blur added to the ground truth image needs to be equal to or less than the amount of blur added when generating the training image. That is, the training image has lower sharpness (greater blurring) than the ground truth image. Therefore, when blurring the ground truth image, the PSF used is equal to or less than the PSF used when generating the training image.

[0052] When the training image is blurred and then reduced in step S103 during the generation of the training image, the ground truth image is also reduced by the same factor. At this time, when the signal value of the original image is higher than the brightness saturation value of the image sensor 102b, the signal is clipped to the brightness saturation value of the image sensor 102b.

[0053] Subsequently, in step S106, memory 108a stores the pair of training images generated in step S103 and ground truth images generated in step S105 as learning data. The training images and ground truth images to be generated can be divided and output as multiple patches. A patch is an image with a predetermined number of pixels (e.g., 64×64 pixels). The number of pixels in the ground truth patch and the training patch may not necessarily match. When images are output as patches, they can be divided by... Figure 1 The flowchart outputs the training image and the ground truth image, or the original image can be decomposed into blocks in step S101, and then the processing after step S102 can be performed.

[0054] As described above, when generating a ground truth image, this embodiment does not simply reduce and output the original image, but rather blurs areas of low optical performance of the optical system, i.e., reduces sharpness. Blurring can be performed using rotationally symmetric blurring (such as Gaussian distribution) or blurring based on optical system degradation. When the optical system's PSF is used as in this embodiment, the image can be deblurred with greater accuracy.

[0055] Next, we will refer to Figure 6 A description is given of a method for learning network parameters (i.e., a method for manufacturing a learning model) performed by the learning device 101 in the embodiment. Figure 6 It is a flowchart related to the learning of network parameters. Figure 6 Each step is primarily performed by the acquirer 101b, generator 101c, or updater 101d of the learning device 101.

[0056] First of all, Figure 6 In step S201, the acquirer 101b acquires ground truth blocks (first ground truth images) and training blocks (first training images). Ground truth blocks are images with relatively less blur, while training blocks are images with relatively more blur. This embodiment uses mini-batch learning to learn network parameters through a multi-layer neural network. Therefore, in step S101, multiple sets of ground truth blocks and training blocks are acquired. However, this embodiment is not limited to this, and online learning or batch learning can be used.

[0057] Subsequently, in step S202, the acquirer 101b acquires information about the correction intensity used in the learning phase, which is also referred to below as correction intensity information. In this embodiment, the correction intensity information is information associated with the learning data and makes it possible to control the amount of correction during estimation. For example, when two pieces of information, “strong” and “weak”, can be set as correction intensity information during estimation, it is necessary to use the learning data corresponding to each piece of information (i.e., training images and ground truth images) to perform learning and prepare two types of network parameters. In this embodiment, a description of the use of these two pieces of information will be given, but the number of settable pieces of information can be increased, or the information can be set as continuous information instead of discrete information.

[0058] Subsequently, in step S203, generator 101c selects at least one training image from the plurality of training images obtained in step S201, inputs the selected training image into the network, and calculates (generates) an output image. Batch learning is the case where all of the plurality of training images are selected, i.e., all training images are input into the network and the network parameters are updated using all outputs. In this method, the computational load increases significantly as the number of training images increases. Online learning is the case where a single training image is selected, i.e., a single training image is used to update the network parameters, and a different training image is used for each update. In this method, the computational load does not increase even if the total number of training images increases, but it is susceptible to noise present in a single training image. Therefore, a mini-batch method can be used, where a small number (mini-batch) of training images are selected from the plurality of training images and those are used to update the network parameters. In the next update, a different small number of training images are selected and used. By repeating this process, the weaknesses of batch learning and online learning can be reduced.

[0059] Here, for reference Figure 7 This will provide a description of the processing performed by a multi-layer neural network. Figure 7 This is a diagram illustrating a convolutional neural network (CNN). However, this embodiment is not limited to this, and for example, a residual network can be used as a CNN, or a GAN (Generative Adversarial Network) can be used, etc. Figure 7 For simplicity, a single training image 201 is shown as the input, but in practice, an output image is generated for each of the selected training images. The training image 201 is an image of a RAW image arranged in three dimensions for each color component.

[0060] Figure 8A and Figure 8BThis is an explanatory diagram related to the color components of an image. In this embodiment, the training image is as follows: Figure 8A The image shown is of a Bayer array. Here, R, G, and B represent red, green, and blue, respectively. Figure 8B An example configuration is shown, where for those from Figure 8A The components of each color in the Bayer array are rearranged. G includes two types, G1 and G2, and they are extracted and arranged separately. Arrangement in three dimensions. Figure 8B The four-channel images of the four images are Figure 7 The training image 201 is shown. This processing is not always necessary, but aberrations and diffraction vary depending on the wavelength, so it is easier to perform deblurring when each color component has the same level of blurring. Additionally, when R, G, and B are arranged in the same dimension, pixels with different brightness levels are locally mixed, which can reduce estimation accuracy. Therefore, the training image can be separated for each color component. The Bayer arrangement is described here, but the same applies to other arrangements, such as cellular structures. In the case of monochrome, rearranging the color components may not be necessary. This embodiment describes an example of jointly learning and estimating multiple color components, but each color can be learned and estimated individually.

[0061] In this embodiment, each of the training image and the ground truth image may include multiple color components arranged periodically, and a step of generating a color component image consisting only of each color component of the training image or the ground truth image may be provided. In this case, the step of generating the color component image is performed on the training image before it is input into the neural network, and the step of generating the color component image is performed on the ground truth image before calculating the error. The CNN has a structure comprising multiple layers, and linear and nonlinear transformations are performed in each layer. The linear transformation is formed by the convolution of the input image (or feature map) with a filter and a bias (e.g., ...). Figure 7 The sum of (as shown) represents the sum. The learning phase updates the network parameters (filter weights and biases) for each layer. The nonlinear transformation is achieved through the activation function (which is a nonlinear function). Figure 7 The transformation is performed by AF in the example. The activation function is, for example, a sigmoid function or a hyperbolic tangent function, and this embodiment uses ReLU (rectified linear unit) as expressed by the following expression (1).

[0062] f(x)=max(x,0) (1)

[0063] In expression (1), max represents the MAX function that outputs the maximum value of the independent variable.

[0064] In the first convolutional layer, the sum of convolutions and biases of the training image 201 input to the input layer and each of the plurality of filters 202 is obtained. The number of channels is the same between each filter 202 and the training image 201. When the number of channels of the training image 201 is two or more, the filter 202 becomes a three-dimensional filter (the third dimension represents the number of channels). The vertical and horizontal dimensions of the filter are arbitrary. A non-linear transformation of the activation function is performed on the result of each convolution and bias sum, and the first feature map 203 is output to the first intermediate layer. The number of channels (the number of arrangements in the three-dimensional direction) of the first feature map 203 is the same as the number of filters 202. Subsequently, the first feature map 203 is input to the second convolutional layer in the same manner as above, and the sum of convolutions and biases of the first feature map 203 and each of the plurality of filters 204 is obtained. The result is non-linearly transformed, and the same processing is repeated for each number of convolutional layers. In general, deep learning is performed by a CNN with three or more convolutional layers. The result output from the last convolutional layer is the output image 211 of the CNN. In the final convolutional layer, it may not be necessary to perform a nonlinear transformation through an activation function.

[0065] Subsequently, in step S204, generator 101c calculates the error between output image 211 and ground truth image 221. In this embodiment, the error is calculated after performing gamma correction on output image 211 and ground truth image 221. Gamma correction is, for example, a process of raising the input signal value to a power, using 1 / 2.2, etc., as the exponent. As in training image 201, ground truth image 221 is arranged for each color component and stacked in the channel direction. In this embodiment, generator 101c uses the following expression (2) to calculate the error L.

[0066]

[0067] In expression (2), t represents the signal value of the ground truth image 221, y represents the signal value of the output image 211, j represents the number of pixels, N represents the total number of pixels, and g represents gamma correction. The Euclidean norm is used in expression (2), but another index can be used, as long as it represents the difference between the ground truth image and the output image. In this embodiment, the error is calculated after gamma correction is performed on the output image 211 and the ground truth image 221. However, this process is not necessary, and the error can be calculated without gamma correction.

[0068] Subsequently, in step S205, updater 101d uses the error calculated in step S204 to calculate the update amount of the network parameters and updates the network parameters. Backpropagation is used here. In backpropagation, the update amount is calculated based on the derivative of the error. However, this embodiment is not limited to this.

[0069] Subsequently, in step S206, the updater 101d determines whether a predetermined termination condition has been met, i.e., whether the optimization of the network parameters has been completed. The predetermined termination condition is, for example, when the learning phase reaches a predetermined time, when the parameters have been updated a predetermined number of times, or when training images and ground truth images that will not be used for parameter updates are prepared and the error between the output image and the prepared ground truth images is equal to or less than a predetermined value. Alternatively, the user can indicate the end of the optimization. When the predetermined termination condition is not met, the process returns to step S203, and the updater 101d obtains a new mini-batch and updates the network parameters. On the other hand, when the predetermined termination condition is met, the process proceeds to step S207.

[0070] In step S207, the network parameters updated by updater 101d are output to recording medium 105. In this embodiment, since the network parameters are learned for each different correction intensity information, the network parameters and the corresponding correction intensity information are stored together in recording medium 105. Based on the above learning phase, even when performing correction processing on an image including regions with low optical performance, a multi-layer neural network capable of reducing the possibility of side effects can be obtained.

[0071] Next, we will refer to Figure 9 A description is given of the estimation stage performed by the image estimation device 103. Figure 9 This is a flowchart illustrating the estimation phase.

[0072] First, in step S301, the acquirer 103b acquires a captured image from the image acquisition device 102 or the recording medium 105. The captured image is an undeveloped RAW image. When the signal values ​​of the RAW image are encoded, the correction unit 103c performs decoding processing. The acquirer 103b acquires correction intensity information from the image acquisition device 102 or the recording medium 105. As described above, the correction intensity information is a parameter associated with correction intensity such as "strong" and "weak," and the user can freely select it via settings in the image acquisition device. The correction intensity information can be retained in the captured image as title information of the captured image. In the following description, title information indicates additional information about the image and can be footnote information. The correction intensity information may not be selected by the user, but may be automatically determined by the image acquisition device 102 based on the image acquisition scenario.

[0073] Subsequently, in step S302, the correction unit 103c obtains the network parameters corresponding to the correction intensity information obtained in step S301. The network parameters are read from the memory 101a of the learning device 101. Alternatively, the memory 103a of the image estimation device 103 may pre-store multiple network parameters, and these network parameters can be read from the memory 103a. The network parameters are then obtained that best match or closely approximate the correction intensity information obtained in step S301 with the correction intensity information used in the learning phase.

[0074] Subsequently, in step S303, the correction unit 103c obtains the input image to be fed into the CNN from the captured image. As in the training images, the input images are arranged for each color component and stacked in a three-dimensional direction. The size of the input image in the estimation stage does not necessarily have to be the same as the size of the training image in the learning stage.

[0075] Subsequently, in step S304, the correction unit 103c generates an estimated image based on the input image and network parameters. Similar to the learning phase, it uses... Figure 7 The CNN shown generates an estimated image. However, Figure 7 The output image 211 is the estimated image and no subsequent processing, such as error calculation with the ground truth image, is performed.

[0076] Subsequently, in step S305, the correction unit 103c determines whether the estimation of a predetermined region of the captured image has been completed. If the estimation is not complete, the process returns to step S303, and the correction unit 103c obtains a new input image from the predetermined region of the captured image. In the CNN used for estimation, when the size of the output image is smaller than the size of the input image, it is necessary to obtain a new input image that overlaps with the previous image from the predetermined region. The predetermined region is all or part of the captured image. Because the captured image is a RAW image, in addition to the image acquired by receiving light, the captured image may also include header information (i.e., information such as the number of pixels in the image and the image acquisition time), as well as optical black information of the image sensor. Since the header information and optical black are unrelated to blurring caused by aberrations or diffraction, they can be excluded from the predetermined region.

[0077] Subsequently, in step S306, the correction unit 103c combines the generated multiple estimated images and outputs a deblurred captured image in which blurring caused by aberrations or diffraction has been corrected. If necessary, the correction unit 103c also outputs header information and optical black information.

[0078] Based on the estimation stage described above, even when performing estimation processing on captured images including images severely degraded due to the performance of the optical system, blurring caused by aberrations or diffraction can be corrected while suppressing undershoot and ringing. After the estimation stage, the user can optionally perform editing work (such as exposure correction) and obtain the final developed image through development processing. This embodiment has described a method of switching network parameters to perform correction based on correction intensity information. However, multiple output images can be generated by acquiring multiple network parameters and inputting the input image into each network. In this case, it is possible to generate multiple output images with different correction intensities. Thus, an output image with an intermediate correction intensity can be generated, for example, by interpolating the output image. Alternatively, a single piece of correction intensity information can be used, and only specific network parameters can be stored in the image pickup device 102 or the recording medium 105.

[0079] The second image can have different sharpness depending on the image height. In the second image, when the image height is the first image height, the sharpness can be the first sharpness, and when the image height is the second image height which is higher than the first image height, the sharpness can be the second sharpness which is lower than the first sharpness.

[0080] The second image can have different sharpness depending on the brightness value. In the second image, when the brightness value is a first brightness value, the sharpness can be a third sharpness, and when the brightness value is a second brightness value lower than the first brightness value, the sharpness can be a fourth sharpness higher than the third sharpness.

[0081] The first image may have lower sharpness than the second image.

[0082] The method for generating learning data may also include a step of acquiring a third image (the original image). In the image generation step, a second image is generated by blurring the third image. In the image generation step, the second image can be generated by blurring the third image, where the amount of blurring varies depending on the optical characteristics of the optical system.

[0083] Second Embodiment

[0084] The image processing system of the second embodiment of the present invention will be described below.

[0085] Figure 10 This is a block diagram illustrating the image processing system 300 in this embodiment. Figure 11 This is an example of the external appearance of an image processing system 300. The image processing system 300 includes a server 301 and an image acquisition device 302 connected via a network 303.

[0086] Server 301 includes a learning unit 310 and a learning data generator 330, which serves as a learning data generation apparatus. The learning unit 310 includes a memory 311, an acquirer 312, a generator 313, and an updater 314, and is configured to enable a neural network to learn network parameters for correcting blur caused by aberrations or diffraction. The learning data generator 330 includes a memory 331, an acquirer 332, and a generator 333. The acquirer 332 is configured to acquire the original image and characteristic information of the optical system used to generate the learning data. The generator 333 is configured to generate training images and ground truth images based on the original image. That is, the generator 333 functions as an acquirer unit for acquiring training images and a generator unit for generating ground truth images with optical characteristics different from those of the training images. The training images and ground truth images generated by the generator 333 are stored in the memory 331.

[0087] Image acquisition device 302 is configured to capture an image of the object space to obtain a captured image, and to use read network parameters to correct blurring in the captured image caused by aberrations or diffraction. Image acquisition device 302 includes an optical system 321 and an image sensor 322. Image estimation unit 323 includes an acquirer 323a and an estimation unit 323b, and is configured to perform correction on the captured image using network parameters stored in memory 324. The network parameters are pre-learned by learning unit 310 and stored in memory 311. Image acquisition device 302 is configured to read network parameters from memory 311 via network 303 and store them in memory 324. Recording medium 325 is configured to store a deblurred captured image (output image) in which blurring caused by aberrations or diffraction has been corrected. When a user gives an instruction to display the output image, the stored output image is read out and displayed on display unit 326. The captured image pre-stored in recording medium 325 can be read out and deblurred by image estimation unit 323. The system controller 327 is configured to perform the above series of controls.

[0088] Next, we will refer to Figure 12 A description is given of the learning data generation process performed by the learning data generator 330. Figure 12 This is a flowchart illustrating the generation and processing of learning data. Figure 12 Each step is performed by each part of the learning data generator 330 (memory 331, acquirer 332, and generator 333).

[0089] In the first embodiment, for regions with low optical performance of the optical system, the original image is blurred when generating the ground truth image. Thus, in regions with low optical performance, the difference between the training image and the ground truth image is reduced, while in regions with high optical performance, the difference is maintained. Therefore, learning data that reduces the likelihood of side effects occurring is generated. On the other hand, this embodiment focuses on the brightness values ​​of the image, and a method for generating learning data to reduce the likelihood of side effects such as undershoot and ringing, which each occur near high-brightness regions, will be described.

[0090] Figure 12 Steps S401, S402, and S403 are the same as Figure 1 The processes in steps S101, S102, and S103 are identical, and therefore their description will be omitted. When information about the brightness and saturation values ​​of the image sensor 322 is used in step S404, the acquirer 332 can acquire the information about the brightness and saturation values ​​from the image pickup device 302 in step S401 or step S402. Alternatively, the information about the brightness and saturation values ​​can be acquired from the title information of the original image acquired in step S401.

[0091] Next, in step S404, a calibration map is generated for generating the ground truth image. Figure 13A This is a diagram illustrating the correction map in this embodiment. In the correction map, the original image is divided into four regions, A, B, C, and D, based on brightness values. Figure 13A In the diagram, the area inside the solid line represents region D, the area inside the dashed line excluding region D is region C, the area inside the double-dotted line excluding both regions D and C is region B, and the area outside the double-dotted line is region A. Figure 13B It is a cross-sectional view of the original image and is related to... Figure 13A The thick solid line corresponds to this. Figure 13B The solid lines, dashed lines, and double-dotted lines in the text are respectively with Figure 13A The solid lines, dashed lines, and double-dotted lines correspond to each other. In this embodiment, region D is the region with a brightness value equal to or higher than the brightness saturation value of image sensor 322, region C is the region with a brightness value greater than 60% and less than the brightness saturation value, region B is the region with a brightness value greater than 20% and less than 60% of the brightness saturation value, and region A is the region with a brightness value less than 20% of the brightness saturation value. In this embodiment, as... Figure 13A As shown, the calibration map is divided into four regions based on the brightness values, but the number of regions can be increased or decreased. In this embodiment, the index is divided into steps, but the index values ​​can be used as is in the calibration map or as continuous values.

[0092] Subsequently, in step S405, a ground truth image is generated based on the correction map generated in step S404. Figure 13A In the calibration map, region A has low brightness values ​​and is unlikely to cause side effects. Therefore, the ground truth image corresponding to this region is used as is, without adding blur. Thus, by maintaining the difference between the training image and the ground truth image for high-performance regions where the possibility of side effects is inherently low, the calibration amount can be maintained.

[0093] Regarding region B, its brightness is higher than that of region A but lower than that of region C, so a slight blur is added to the original image. For example, a ground truth image is generated by adding a reduced PSF to the original image, which is achieved by reducing the PSF corresponding to each pixel to 20%. Subsequently, region C's brightness is higher than that of region B, further increasing the blur amount. For example, a ground truth image is generated by adding a reduced PSF to the original image, which is achieved by reducing the PSF given to the training image to 50%. Region D has the highest brightness, so a ground truth image is generated by blurring the original image with the PSF added to the training image, which is not reduced. In other words, in the ground truth image, when the brightness value is the first brightness value, the sharpness is the third sharpness (the third blur amount), while when the brightness value is the second brightness value, which is lower than the first brightness value, the sharpness is the fourth sharpness, which is higher than the third sharpness (the fourth blur amount, less than the third blur amount).

[0094] The reduction in PSF for the calibration image described above is an example and is not limited to. Region A is described as a region where no blur is added, but blur can be added to the entire region. However, the amount of blur added to the ground truth image needs to be equal to or less than the amount of blur used when generating the training image. That is, the training image has lower sharpness (greater blur amount) than the ground truth image. Therefore, when blur is added to the ground truth image, a PSF equal to or less than the PSF used when generating the training image is used. When the training image is reduced after blurring during the generation of the training image in step S403, the ground truth image is also reduced by the same factor. At this time, when the signal value of the original image is higher than the brightness saturation value of the image sensor 322, the signal is clipped to the brightness saturation value of the image sensor 322.

[0095] Subsequently, in step S406, the pair of training images generated in step S403 and ground truth images generated in step S405 are stored as learning data in memory 331. Step S406 is the same as step S106, so its detailed description will be omitted.

[0096] In this embodiment, the learning unit 310 performs a learning phase, and the image estimation unit 323 performs an estimation phase. For the learning phase in this embodiment, the image is used in… Figure 12 The learning data (ground truth images and training images) generated during the process. Other content is the same as in the first embodiment. Figure 6 The flowchart shown is the same, therefore its details will be omitted. The estimation stage is the same as in the first embodiment. Figure 9 The process shown in the flowchart is the same, so its description will be omitted.

[0097] As described above, a calibration map is generated based on brightness values, and a ground truth image is generated based on the calibration map. This allows for the creation of learning data capable of reducing the likelihood of side effects such as undershoot and ringing, which occur near high-brightness areas. Network parameters are generated using this learning data, and estimation processing is performed using information about the network parameters. Therefore, even when the captured image includes high-brightness areas, blurring caused by aberrations or diffraction can be corrected with high accuracy while reducing the likelihood of side effects. In this embodiment, the calibration map is divided into four parts, but it can be divided into two parts: a saturated brightness area and a portion other than the saturated brightness area. For example, when a ground truth image is generated for the saturated brightness area by blurring the original image with a PSF added to the training image, the difference between the training image and the ground truth image for the saturated brightness area disappears. When the network performs learning using this learning data and performs estimation processing using the network parameters, the following processing can be achieved: the region other than the saturated brightness area is deblurred, while the saturated brightness area is not deblurred. Specifically, in the captured image, the saturated areas are regions where information is lost, and if deblurring is performed as in other areas, side effects may occur. By performing learning as described above, the network can output an image with high-quality estimates. In this embodiment, the correction map is generated using the brightness values ​​of the original image, but the correction map can be generated based on changes in brightness values. In this case, a differential image can be generated by applying a differential filter, a Prewitt filter, or a Sobel filter to the original image. Alternatively, a second-order differential image can be generated using a Laplacian filter, etc. When the correction map is generated based on changes in brightness in this way, learning data that can suppress side effects occurring near edges with high contrast can be generated.

[0098] The first embodiment described the generation of a performance-based calibration map, and this embodiment describes the generation of a brightness-based calibration map; however, both can be performed. In this case, in step S404, not only is a brightness-based calibration map generated, but also a calibration map based on the optical information of the optical system is generated, and the two calibration maps are integrated. For example, when integrating... Figure 5 and Figure 13A When creating a correction map, the integrated map is a map with increased blur added to the correction map. That is, using a method that will serve as... Figure 5 Region B in and as Figure 13A The method of setting the region of region D in the D region is as follows. Alternatively, the calibration map can be generated by integrating the average value of the reduction rate of PSF in each calibration map. The integration method is not limited to these methods, and the integrated calibration map can be generated by using another method. By using two calibration maps, learning data that takes into account the effects of both the performance of the optical system and the brightness values ​​of the image can be generated.

[0099] Third Embodiment

[0100] Next, a description of the image processing system according to a third embodiment of the present invention will be given. The configuration and learning process of the image processing system in this embodiment are the same as those in the first embodiment, but the learning data generation process (method of generating learning data) and the estimation process are different from those in the first embodiment.

[0101] Reference Figure 14 A description of the method for generating the learning data performed in this embodiment is given. Figure 14 This is a flowchart illustrating the method for generating learning data.

[0102] In the first embodiment, a calibration map is generated, and a ground truth image is generated based on the calibration map. However, in this embodiment, the ground truth image and training image are generated without using the calibration map. In this embodiment, the calibration amount of the ground truth image and training image relative to the original image is constant in the image. Since steps S601 and S602 are the same processes as steps S101 and S102 in the first embodiment, their description will be omitted.

[0103] In step S603, generator 108c uses the optical information acquired in step S602 to generate a training image (first image). In this embodiment, the training image is, for example, an image obtained by extracting only a portion of the image (corresponding to a portion of the image height) acquired using image picking with the optical system (an image corresponding to the entire field of view of the optical system). The first embodiment described a method for linearly interpolating the PSF so that the PSF is different for each pixel location; however, this embodiment adds a uniform blur without changing the PSF based on pixel location. To address fluctuations in optical performance (optical information) that depend on image height, this embodiment increases the number of ground truth images and training images to be generated by increasing the number of original images. When multiple lens states, image heights, and azimuths need to be processed during estimation, they are mixed and used to generate pairs of ground truth images and training images.

[0104] Subsequently, in step S604, generator 108c generates a ground truth image (second image) to be paired with the training image generated in step S603. In this embodiment, the ground truth image corresponds to an image (corresponding to a portion of the image height) obtained, for example, by extracting only a portion of an image acquired using image picking with an optical system (an image corresponding to the entire field of view of the optical system). The region of the original image extracted as the ground truth image can be equivalent to the training image to be paired. In this embodiment, the ground truth image is generated by using the peak value of the PSF as a measure of the optical system's performance. Higher optical system performance results in a higher peak value of the PSF, and lower performance results in a lower peak value. Therefore, this peak value is suitable as an indicator used when generating the ground truth image from the original image. As in the second embodiment, the ground truth image can be generated based on the presence or absence of brightness saturation portions rather than optical performance.

[0105] Figure 15 This is a graph illustrating the relationship between the performance of the optical system (optical performance) and the amount of blur added to the PSF (Plain Oldest Field Image) to generate the ground truth image, divided into three regions (parts A, B, and C) based on the performance of the optical system. In this embodiment, Figure 15 The horizontal axis represents the optical performance corresponding to the peak value of the PSF. When part A includes optical performance, the peak value of the PSF is considered high and the performance of the optical system is fully utilized. When part A includes the optical performance of a region of the original image extracted as the ground truth image, the original image is used as the ground truth image. In this case, the optical performance is high and the possibility of side effects is low, so the difference between the training image and the ground truth image is maintained. Thus, learning is performed such that when the optical performance is included in part A, the amount of correction used for the training image (input image) becomes relatively large.

[0106] When part B includes the optical properties of a region from the original image extracted as a ground truth image, the amount of blur in the PSF is adjusted based on the peak value of the PSF. For example... Figure 15 As shown, when the peak value of PSF is high, the amount of blur applied to the original image is small. On the other hand, when the peak value is low, the amount of blur is adjusted to be large. Figure 15 The change in part B is represented by a straight line, but the change is not limited to this and can also be represented by nonlinear functions such as quadratic functions and exponential functions.

[0107] Part C represents the region with the lowest optical system performance. When part C includes the optical performance of the region of the original image from which the ground truth image is extracted, the ground truth image is generated by using the PSF (the PSF used when generating the training image) as is. That is, the training image and the ground truth image are generated using the same method. When the optical performance is as low as in part C, side effects are more likely to occur. Therefore, by matching the training image with the ground truth image, the deblurring effect introduced into the training image (input image) by learning is suppressed. This allows side effects to be suppressed. In this way, when generating the ground truth image from the original image, the amount of blur is adjusted based on the performance of the optical system. Thus, under conditions where side effects are unlikely to occur, the deblurring effect can be increased. On the other hand, under conditions where side effects may occur, the deblurring effect is suppressed, and side effect suppression can be prioritized.

[0108] The peak value of the PSF is used as an indicator for adjusting the amount of ambiguity, but frequency characteristics can also be used as an indicator. In this case, the OTF is generated by performing a Fourier transform on the PSF, and the MTF is calculated by taking the absolute value of the OTF. When using the MTF as an indicator, the MTF value at a specific frequency (e.g., the MTF value at half the Nyquist frequency) can be used, or an integral value of a portion (up to the Nyquist frequency) or an average value can be used. Ambiguity asymmetry can be used as another indicator, or the asymmetry can be quantified, and the amount of ambiguity can be adjusted based on this value. Basically, the trend is that the more asymmetric the PSF, the more likely side effects are to occur. As a method for quantifying asymmetry, for example, the following method is used: for two cross-sections orthogonal to each other, such as the meridional and sagittal directions, the LSF is calculated from the PSF, and the difference between the peak values ​​of the two LSFs is used as an indicator. In this case, the greater the difference between the peak values, the more asymmetric the shape, and therefore the more ambiguity is increased. On the other hand, as the difference decreases, the ambiguity decreases. Frequency characteristics can be used instead of LSF for evaluation, and the average value of MTF characteristics in the meridional and sagittal directions, or the difference at a specific frequency, can be used as an indicator of asymmetry. Alternatively, the skewness of PSF can be used. The larger the absolute value of the skewness, the higher the asymmetry, and thus the more blurry the image. On the other hand, the smaller the absolute value, the smaller the asymmetry, and thus the less blurry the image. As a simpler indicator for adjusting the amount of blur, the image height (the position of the ground truth image in the image corresponding to the entire viewing angle of the optical system) can be used. As described in the first embodiment, there is generally a correlation that optical performance decreases with increasing image height.

[0109] When the high-frequency components of the original image are insufficient, the original image can be reduced in size when generating the ground truth image. In this case, the training image should also be reduced in the same way.

[0110] Subsequently, in step S605, the training image generated in step S603 and the ground truth image generated in step S604 are stored as learning data in memory 108a.

[0111] The method for generating the learning data in this embodiment has been described. When learning using ground truth images and training images generated in this way, a learning model can be generated that achieves highly accurate correction while reducing the likelihood of side effects.

[0112] Next, we will refer to Figure 16 A description is given of the estimation phase performed by the image estimation device 103. Figure 16 This is a flowchart illustrating the estimation phase.

[0113] The first embodiment described a method for obtaining network parameters based on correction intensity information. This embodiment will describe a method for obtaining a network model based on resource information of the image estimation device 103. In this embodiment, the network model includes network parameters and network configuration (architecture). As described in the first embodiment, the size of the input image to the CNN can be different from the size during learning. When a portion of a region of the captured image is used as the input image, the estimated output images can be combined to generate a deblurred captured image. When estimation processing is performed in this way, by increasing the size of the input image to reduce the number of segments of the captured image, estimation processing can be performed at a higher speed. However, as the size increases, the amount of memory (RAM) used increases accordingly. If the memory capacity of the image estimation device is exceeded, the processing speed will decrease or the processing will terminate abnormally. That is, to perform processing more efficiently, it is better to set the input size according to the image estimation device, and an example of this will be described here.

[0114] First, in step S701, the acquirer 103b acquires a captured image from the image pickup device 102 or the recording medium 105. The captured image is an undeveloped RAW image. When the signal values ​​of the RAW image are encoded, the correction unit 103c performs decoding processing. The acquirer 103b acquires memory information (resource information) of the image processing device. The memory information to be acquired may be information about the capacity of the physical memory, but since the available capacity depends on other processing, information about the free capacity may be acquired. When the influence of other processing is small, information about the capacity of the physical memory may be acquired, or the available memory capacity may be estimated based on the physical memory and the memory capacity in use.

[0115] Subsequently, in step S702, the correction unit 103c obtains a suitable network model based on the memory information acquired in step S701. In the case of a network model where the input image size is fixed, it is necessary to select a suitable network model. For example, when the free capacity of the memory acquired in step S701 is sufficiently greater than the memory capacity used by the CNN, the correction unit 103c obtains a network model with a large input image size. On the other hand, when the free capacity is small, the correction unit 103c obtains a network model with a small input image size. Since it is not necessary to change the network parameters, the correction unit 103c can obtain only the network configuration based on the memory information. Network models include various formats, but when the input image size is fixed, switching the network model in this way allows for efficient execution of the correction process. In the case of a network model where the input image size is variable, it is not necessary to change the network model, and appropriate values ​​can be set only for the input image size. Two network model selection methods have been described, but as the variety of network models increases, it becomes possible to select a network model more suitable for the image processing device. The network model can be selected not only based on automatic determination according to memory information but also based on user input information. For example, users can choose between high-performance mode and lightweight mode, and when selecting a network model, the selected mode information is also taken into account, thereby achieving the processing desired by the user.

[0116] Since step S703 and subsequent steps are the same as step S303 and subsequent steps in the first embodiment, their detailed description will be omitted.

[0117] The estimation stage in this embodiment has been described. By acquiring the network model and configuration based on the memory information of the image processing device, estimation processing suitable for the processing environment can be achieved.

[0118] In this embodiment, the training image and the ground truth image correspond to a portion of the image height in the image corresponding to the entire viewpoint of the optical system. In this case, during learning, in addition to the training image, information about the image height of the training image can also be input into the neural network and learned. This allows for more accurate learning of the correction amount for the position in the image corresponding to the entire viewpoint of the optical system. While information about the image height of the training image is input into the neural network for learning, information about the image height of the input image is input into the neural network, as in the learning phase during estimation.

[0119] Other embodiments

[0120] One or more embodiments of the present invention can also be implemented by a computer that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a "non-transitory computer-readable storage medium") to perform the functions of one or more embodiments described above and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing the functions of one or more embodiments described above, and by a method performed by a computer of a system or device by, for example, reading and executing computer-executable instructions from a storage medium to perform the functions of one or more embodiments described above and / or controlling one or more circuits to perform the functions of one or more embodiments described above. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessor unit (MPU)) and may include a network of individual computers or individual processors to read and execute computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or a storage medium. The storage medium may include, for example, a hard disk, random access memory (RAM), read-only memory (ROM), a storage device for a distributed computing system, an optical disc (such as a CD, DVD, or Blu-ray disc), or a digital versatile disc (BD). TM One or more of the following: flash memory devices, memory cards, etc.

[0121] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0122] Each embodiment can provide a method for manufacturing learning data, etc., so that even when the image input to the neural network includes a high-brightness object or an object that is greatly blurred due to the aberration of the optical system, each of the learning data, etc., can achieve highly accurate correction while reducing the possibility of side effects.

[0123] While the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. A method for manufacturing learning data for enabling a neural network to perform learning, the method comprising: The first acquisition step is configured to acquire the original image; The second acquisition step is configured to acquire a first image generated by adding a first blur to the original image as a training image; as well as The third acquisition step is configured to acquire a second image, generated by adding a second blur to the original image, as a ground truth image; The characteristic is that the second blur amount added to the second image is less than the first blur amount added to the first image, and The first and second blurs are obtained based on the same optical information.

2. The method for generating learning data according to claim 1 further includes a fourth acquisition step configured to acquire optical information. The optical information consists of multiple optical information lines, each different from the image height. The second acquisition step acquires a first image generated by adding blur using the multiple optical information, and The third acquisition step involves acquiring a second image generated by adding blur using the multiple optical information.

3. The method for generating learning data according to claim 1, The second image is generated by adding a blur of varying amounts depending on the image height to the original image.

4. The method for generating learning data according to claim 3, The second image is generated by adding a blur with a first blur amount to a first image height of the original image and by adding a blur with a second blur amount greater than the first blur amount to a second image height of the original image that is higher than the first image height.

5. The method for generating learning data according to claim 1 further includes a fourth acquisition step configured to acquire optical information. The optical information depends on multiple optical information lines, each with its own unique characteristics, and these lines vary depending on the image height. The second acquisition step acquires a first image generated by adding blur to the optical information corresponding to the image height of the first image from among the multiple optical information sources, and... The third acquisition step involves acquiring a second image generated by adding blur using the optical information corresponding to the image height of the second image from among the multiple optical information.

6. The method for generating learning data according to claim 1, in, In the third acquisition step, the blur to be added is adjusted based on the brightness value.

7. The method for generating learning data according to claim 6, in, When the brightness value is the first brightness value, a second image is generated by adding a blur with a first blur amount to the original image, and Specifically, when the brightness value is a second brightness value that is lower than the first brightness value, a second image is generated by adding a second blur amount that is greater than the first blur amount to the original image.

8. A learning method that enables a neural network to perform learning using learning data generated by a method for manufacturing learning data according to any one of claims 1 to 7, the learning method comprising the following steps: Processed images are generated by inputting a first image as a training image into a neural network; as well as The neural network is updated based on the second image and the processed image.

9. A learning data manufacturing apparatus for manufacturing learning data for learning neural networks, the learning data manufacturing apparatus comprising: The first acquisition unit is configured to acquire the original image; The second acquisition unit is configured to acquire a first image generated by adding a first blur to the original image; as well as The third acquisition unit is configured to acquire a second image generated by adding a second blur to the original image; The characteristic is that the second blur amount added to the second image is less than the first blur amount added to the first image, and The first and second blurs are obtained based on the same optical information.

10. A learning device, comprising: The learning data manufacturing apparatus according to claim 9; The generation unit is configured to generate a processed image by inputting a first image as a training image into a neural network; as well as The update unit is configured to update the neural network based on the second image and the processed image.

11. A non-transitory computer-readable storage medium storing a computer program that enables a computer to execute a method for generating learning data according to any one of claims 1 to 7.

12. A method for manufacturing a learned model, comprising the steps of: manufacturing a learned model by having a neural network perform learning using learning data generated by a method for manufacturing learning data according to any one of claims 1 to 7; the method comprising the steps of: The processed image is generated by inputting the first image as a training image into the neural network; as well as The neural network performs learning based on the second image and the processed image.

13. An image processing method, comprising the following steps: Obtain the neural network obtained by the manufacturing method according to claim 12; An estimated image is generated by feeding the input image into the neural network.

14. The image processing method according to claim 13, in, The input image is an image obtained through image picking, and The estimated image is generated by the neural network to correct blur in the input image generated during image picking.

15. A non-transitory computer-readable storage medium storing a computer program that enables a computer to execute the image processing method according to claim 13.

16. An image processing apparatus comprising at least one processor or circuitry configured to perform a plurality of tasks, the plurality of tasks including: The task is configured to acquire a neural network obtained by the manufacturing method according to claim 12; The generation task is configured to estimate an image by feeding an input image into the neural network.

Images