Training method, image processing method, method for producing learned model, program, and training device

Abstract

To provide an image processing method that can suppress generation of images with low shaping accuracy in shaping of defocus blur that uses a machine learning model.SOLUTION: An image processing method includes: a first step of generating a first image by adding a first blur to an original image, and generating a second image by adding a second blur to the original image; and a second step of generating an output image by inputting the first image to a machine learning model, and performing learning of the machine learning model on the basis of the output image and the second image. The first blur and the second blur are blurs of different shapes. Optical performance corresponding to the first blur is higher than that corresponding to the second blur at a specific frequency.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing method for correcting blur caused by defocus in a captured image to obtain an image with good blur. [Background technology]

[0002] Patent Document 1 discloses a method for shaping the shape of blur caused by defocus (defocus blur) by dividing the pupil of an optical system into multiple split pupils, capturing multiple parallax images of an object space observed from each split pupil, and adjusting the weight when combining the multiple parallax images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-220016 A Summary of the Invention [Problem to be solved by the invention]

[0004] The method disclosed in Patent Document 1 adjusts the weight of each divided pupil to synthesize multiple parallax images, and therefore cannot reproduce defocus blur corresponding to a pupil larger than the pupil of the optical system. In other words, the method disclosed in Patent Document 1 cannot fill in the gaps of defocus blur caused by vignetting. In addition, when the weight of the synthesis of multiple parallax images becomes uneven, noise increases. In addition, since double-line blur and the like have a fine structure, in order to reduce their influence, it is necessary to finely divide the pupil of the optical system. In this case, the spatial resolution of each parallax image decreases and noise increases.

[0005] On the other hand, in order to solve the above-mentioned problem, a method of shaping defocus blur using a machine learning model such as CNN (Convolutional Neural Network) is known. However, in the method of shaping defocus blur using a machine learning model, it is not always possible to shape the defocus blur into an arbitrary shape, and the shaping accuracy may decrease.

[0006] Therefore, an object of the present invention is to provide an image processing method capable of suppressing the generation of images with low shaping accuracy in shaping defocus blur using a machine learning model. [Means for solving the problem]

[0007] An image processing method as one aspect of the present invention includes a first step of generating a first image by adding a first blur to an original image, and generating a second image by adding a second blur to the original image, and a second step of generating an output image by inputting the first image to a machine learning model, and training the machine learning model based on the output image and the second image, wherein at a specific frequency, optical performance corresponding to the first blur is higher than optical performance corresponding to the second blur.

[0008] Other objects and features of the present invention are illustrated in the following examples. Effect of the Invention

[0009] According to the present invention, it is possible to provide an image processing method capable of suppressing the generation of images with low shaping accuracy in shaping defocus blur using a machine learning model. [Brief description of the drawings]

[0010] [Figure 1] 6 is a flowchart relating to a method for determining a second defocus blur in the first embodiment. [Diagram 2] 1 is a block diagram of an image processing system according to a first embodiment. [Diagram 3]1 is an external view of an image processing system according to a first embodiment. [Figure 4] 4 is a flowchart relating to a method for generating learning data in the first embodiment. [Diagram 5] 4 is a flowchart relating to weight learning in the first embodiment. [Figure 6] 4 is a flowchart related to generation of an estimated image in the first embodiment. [Figure 7] FIG. 1 is a configuration diagram of a machine learning model in a first embodiment. [Figure 8] 4A to 4C are explanatory diagrams of the optical performance of the first defocus blur in the first embodiment. [Figure 9] 6A to 6C are explanatory diagrams of the optical performance of the second defocus blur in the first embodiment. [Figure 10] FIG. 11 is a configuration diagram of a machine learning model in the second embodiment. [Figure 11] FIG. 11 is a block diagram of an image processing system according to a second embodiment. [Figure 12] FIG. 11 is an external view of an image processing system according to a second embodiment. [Figure 13] 13 is a flowchart relating to a method for generating learning data in the second embodiment. [Figure 14] 13 is a flowchart relating to weight learning in the second embodiment. [Figure 15] 13 is a flowchart related to generation of an estimated image in the second embodiment. [Figure 16] 13 is an explanatory diagram of the optical performance of the first defocus blur in the second embodiment. FIG. [Figure 17] FIG. 11 is a block diagram of an image processing system according to a third embodiment. [Figure 18] FIG. 11 is an external view of an image processing system according to a third embodiment. [Figure 19] 13 is a flowchart related to generation of an estimated image in the third embodiment. [Figure 20] 6A to 6C are diagrams illustrating point image intensity distributions at defocus distances in the respective examples. [Figure 21]FIG. 13 is a diagram showing a result of shaping defocus blur in a comparative example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0012] Before describing each embodiment in detail, the gist of the present invention will be described. The present invention shapes the defocus blur of a captured image using a machine learning model. The machine learning model includes, for example, a neural network, genetic programming, or a Bayesian network. The neural network includes a CNN, a Generative Adversarial Network (GAN), a Recurrent Neural Network (RNN), and the like. The shaping of the defocus blur includes shaping from a two-line blur to a Gaussian blur or a circular blur.

[0013] With reference to FIG. 20, double line blur, circular blur, and Gaussian blur will be described. FIG. 20 is a diagram showing a point spread function (PSF) at a defocus distance. FIG. 20(A) shows the PSF of double line blur, FIG. 20(B) shows the PSF of circular blur, and FIG. 20(C) shows the PSF of Gaussian blur. In FIG. 20(A) to (C), the horizontal axis shows the spatial coordinate (position), and the vertical axis shows the intensity. As shown in FIG. 20(A), double line blur has a PSF with a separated peak. When the PSF at the defocus distance has a shape as shown in FIG. 20(A), a subject that is originally a single line appears to be doubly blurred when defocused. As shown in FIG. 20(B), circular blur has a PSF with flat intensity. As shown in FIG. 20(C), Gaussian blur has a PSF with a Gaussian distribution. Other examples of defocus blur that can be shaped include defocus blur missing due to vignetting, ring-shaped defocus blur caused by pupil obstruction by a catadioptric lens, etc. Note that the shapes of defocus blur that can be shaped are not limited to these.

[0014] The shaping of defocus blur in the present invention does not mean adding defocus blur to a pan-focus image with a deep depth of field and reproducing an image with a shallow depth of field. It is necessary to shape an already defocus-blurred subject into a desired defocus blur, that is, to apply a defocus blur that satisfies the difference between the already applied defocus blur and the desired defocus blur, which requires more advanced processing.

[0015] In shaping defocus blur using a machine learning model, if the defocus blur (first blur) of a training image (first image) does not have a higher frequency of optical performance than the defocus blur (second blur) of a correct answer image (second image), the shaping accuracy may decrease. In other words, if the first blur does not have a higher optical performance at a specific frequency than the second blur, the shaping accuracy may decrease. The correct answer image is an image to be obtained as an output image of the machine learning model, and the training image is an input image to the machine learning model.

[0016] Fig. 21 is a diagram showing the results of defocus blur shaping in a comparative example, showing the results of shaping defocus blur using weights learned from learning images in which the training images do not have a frequency with higher optical performance than the correct image. In Fig. 21, the horizontal axis indicates spatial coordinates (position), and the vertical axis indicates intensity. A dotted line 601 indicates the defocus blur before shaping, a solid line 602 indicates the correct defocus blur, and a dashed line 603 indicates the defocus blur after shaping. There is a large difference between the correct defocus blur (solid line 602) and the defocus blur after shaping (dashed line 603), and the accuracy of shaping is low.

[0017] In addition, the defocus blur changes depending on the state (lens state) of the optical system used to capture the captured image, the image height, the azimuth, and the defocus amount. Therefore, unless an appropriate defocus blur is learned as a correct image in consideration of these changes, the learning difficulty increases and the accuracy of shaping the defocus blur may decrease. Therefore, each embodiment provides an image processing method for shaping the defocus blur of a captured image by setting an appropriate defocus blur to a correct image so that the optical performance of the defocus blur of the training image is higher than that of the correct image at a specific frequency.

[0018] In the following, the stage of learning the weights of the machine learning model will be referred to as the learning phase, and the stage of shaping defocus blur using the machine learning model using the learned weights will be referred to as the estimation phase. EXAMPLES

[0019] First, an image processing system 100 according to the first embodiment of the present invention will be described. Fig. 2 is a block diagram of the image processing system 100. Fig. 3 is an external view of the image processing system 100.

[0020] The image processing system 100 includes a learning device (image processing device) 101, an imaging device 102, an image estimation device 103, a display device 104, a recording medium 105, an output device 106, and a network 107. The learning device 101 includes a storage unit 101a, an acquisition unit 101b, a generation unit (generation means) 101c, and an update unit (learning means) 101d, and learns weights of a machine learning model used for defocus blur shaping. Details regarding learning of the weights and defocus blur shaping processing using the weights will be described later.

[0021] The imaging device 102 has an optical system 102a and an imaging element 102b, and captures an image of a subject space. The optical system 102a collects light incident from the subject space to form an optical image (subject image). The imaging element 102b photoelectrically converts the optical image to obtain the captured image. The imaging element 102b is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor.

[0022] The image estimation device 103 includes a storage unit 103a, an acquisition unit 103b, a blur shaping unit 103c, and a generation unit 103d. The image estimation device 103 generates an estimated image by shaping defocus blur for a captured image (or at least a part thereof) captured by the imaging device 102. For defocus blur shaping, a machine learning model using weights learned by the learning device 101 is used. The learning device 101 and the image estimation device 103 are connected by a network 107, and the image estimation device 103 reads out information on the learned weights from the learning device 101 during or before defocus blur shaping. The estimated image is output to at least one of a display device 104, a recording medium 105, and an output device 106.

[0023] The display device 104 is, for example, a liquid crystal display or a projector. A user can perform editing work while checking an image in the middle of processing via the display device 104. Details of the user interface during editing work will be described later. The recording medium 105 is, for example, a semiconductor memory, a hard disk, a server on a network, etc., and stores the estimated image. The output device 106 is, for example, a printer.

[0024] Next, the generation of learning data executed by the learning device 101 will be described with reference to Fig. 4. Fig. 4 is a flowchart relating to a method of generating learning data in this embodiment.

[0025] First, in step S101, the acquisition unit 101b acquires an original image. The original image may be one or more. The original image may be a real-life image or a CG (Computer Graphics) image. In the following steps, a first defocus (first blur) and a second defocus blur (second blur) are applied to the original image to generate a training image (first image) and a correct image (second image). For this reason, it is desirable that the original image has edges with various intensities and directions, textures, gradations, flat parts, etc., so that the shape of the defocus blur can be shaped correctly for various subjects.

[0026] Preferably, the original image has a signal value higher than the brightness saturation value of the image sensor 102b. This is because even in real subjects, there are subjects that do not fall within the brightness saturation value when imaged by the image capture device 102 under specific exposure conditions. The ground truth image and the training images are subjected to defocus blur and then clipped by the brightness saturation value of the image sensor 102b, thereby reproducing subjects that do not fall within the actual brightness saturation value.

[0027] Next, in step S102, the generating unit 101c generates a first defocus blur (first blur) and stores it in the storage unit 101a. The first defocus blur is the defocus blur to be shaped. The first defocus blur changes depending on the lens state and image height / azimuth of the optical system 102a, and the defocus amount. The lens state is the state of the optical system 102a, and includes, but is not limited to, the state of zoom, aperture, and focus distance. For example, the defocus blur at 5m when focused at 2.5m differs when the aperture is set to F1.4 and F2.8. Also, even with the same lens state, the defocus blur at 5m differs between 80% image height / 0 degree azimuth and 50% image height / 45 degree azimuth.

[0028] Next, in step S103, the generating unit 101c generates a second defocus blur (second blur) and stores it in the storage unit 101a. The second defocus blur is the defocus blur after shaping. A method of determining the second defocus blur for the first defocus blur in a specific lens state will be described in detail with reference to FIG. 1. FIG. 1 is a flowchart relating to the method of determining the second defocus blur in this embodiment.

[0029] In this embodiment, the second defocus blur is determined to have the same shape as the first defocus blur that changes depending on the image height and azimuth. This makes it possible to shape the defocus blur to the same shape over the entire angle of view even if a captured image contains various defocus blur shapes.

[0030] In addition, since the first defocus blur changes for each defocus amount, the second defocus blur is determined for each defocus amount. This makes it possible to determine an appropriate defocus blur for the correct image according to the defocus amount.

[0031] Moreover, since the first defocus blur changes for each lens state of the optical system 102a, the second defocus blur is determined for each lens state. This allows a defocus blur appropriate for a correct image to be determined for each lens state. Note that in this embodiment, the second defocus blur is determined in one way for each defocus amount in a specific lens state, but two or more ways may be determined. The case where the second defocus blur is determined in two or more ways will be described later in Example 2. When the second defocus blur is determined in one way, the number of learning images and the learning time can be reduced.

[0032] First, in step S1031, the acquisition unit 101b acquires the optical performance of the first defocus blur generated in step S102. In this embodiment, the optical performance includes, but is not limited to, a modulation transfer function MTF or an optical transfer function OTF.

[0033] Next, in step S1032, the acquisition unit 101b acquires the average value of the optical performance of the first defocus blur acquired in step S1031. In this embodiment, the acquisition unit 101b acquires the average value from half the Nyquist frequency of the MTF to the Nyquist frequency.

[0034] In this embodiment, the first defocus blur has a higher average value of optical performance at a plurality of frequencies than the second defocus blur. Here, the plurality of frequencies are in a range from half the Nyquist frequency of the training image to the Nyquist frequency. Note that any interval can be selected as the calculation interval of the average value used for comparing the optical performance, and an appropriate calculation interval can be set according to the optical performance of the defocus blur. For example, the plurality of frequencies may be in a range from frequency 0 to half the Nyquist frequency of the training image.

[0035] In this embodiment, half the Nyquist frequency is f N / 2, the Nyquist frequency is f N When the number of data points is n, the average value M is calculated by the following formula (1).

[0036]

number

[0037] FIG. 8 is an explanatory diagram of the optical performance of the first defocus blur in this embodiment, and shows the MTF of the first defocus blur on the axis, 50% image height, azimuth 0 degrees, and 100% image height, azimuth 0 degrees in a specific lens state and defocus amount. In FIG. 8, the horizontal axis shows the spatial frequency, and the vertical axis shows the MTF. On the axis, 50% image height, azimuth 0 degrees, and 100% image height, azimuth 0 degrees correspond to the dotted line 1001, the dashed line 1002, and the dashed line 1003, respectively. The Nyquist frequency of the image sensor 102b used for imaging is 90 [lp / mm]. For each MTF, the average value from half the Nyquist frequency to the Nyquist frequency is obtained. Similarly, the average value is obtained when the defocus amount and the image height and azimuth are changed.

[0038] Next, in step S1033, the generation unit 101c generates a plurality of patterns of defocus blur as candidates for the second defocus blur. The second defocus blur to be generated has a different shape. The shape refers to the type and size of the defocus blur. The type refers to differences such as Gaussian blur, circular blur, and two-line blur due to differences in the intensity distribution of the PSF. The size refers to the range in which the PSF has intensity. There is no restriction on the second defocus blur to be generated as a candidate.

[0039] FIG. 9 is an explanatory diagram of the optical performance of the second defocus blur in this embodiment. In FIG. 9, the horizontal axis indicates the spatial frequency, and the vertical axis indicates the MTF. In FIG. 9, a solid line 1004, a dotted line 1005, a dashed line 1006, and a dashed line 1007 indicate MTFs of Gaussian blur with different magnitudes, respectively. As in step S1032, the acquisition unit 101b acquires an average value from half the Nyquist frequency to the Nyquist frequency for each MTF. Note that a plurality of candidates for the second defocus blur may be generated in advance, and an average value from half the Nyquist frequency to the Nyquist frequency for each MTF may be acquired and stored in the storage unit 101a.

[0040] Next, in step S1034, the acquisition unit 101b compares the average value of the first defocus blur acquired in step S1032 with the average value of the second defocus blur acquired in step S1033, and determines and acquires the second defocus blur. The average values ​​of the dotted line 1001, the dashed line 1002, and the dashed line 1003 in FIG. 8 are approximately 0.11, 0.1, and 0.07, respectively. The average values ​​of the solid line 1004, the dotted line 1005, the dashed line 1006, and the dashed line 1007 in FIG. 9 are approximately 0.25, 0.03, 0, and 0, respectively. Therefore, one of the dotted line 1005, the dashed line 1006, and the dashed line 1007 is determined as the second defocus blur.

[0041] In this embodiment, a plurality of training images and a plurality of answer images are generated. At least two types of first defocus blur are added to generate a plurality of training images. The same second defocus blur is added to generate a plurality of answer images. However, this embodiment is not limited to this, and the plurality of second defocus blur may be at least two types.

[0042] The above is the method for determining the second defocus blur in this embodiment. In this embodiment, as described above, the second defocus blur is determined for the first defocus blur in a specific lens state. Similarly, for other lens states, an average value is obtained from the optical performance of the first defocus blur to determine the second defocus blur.

[0043] In this embodiment, it is preferable that the second defocus blur is determined according to the size of the image sensor 102b. The maximum image height of the captured image is determined according to the size of the image sensor 102b. For example, even if the same optical system 102a is used, the maximum image height that can be captured is smaller for the APS-C size than for the full size. In other words, the maximum image height of the first defocus blur to be compared is smaller.

[0044] Next, in step S104 of FIG. 4, the generating unit 101c generates a training image and stores it in the storage unit 101a. The training image is an image (first image) obtained by performing imaging simulation by applying a first defocus blur to be shaped to an original image. In order to accommodate various captured images, it is preferable to apply defocus blur corresponding to various defocus amounts. The application of defocus blur can be performed by convolving the original image with a PSF or by taking the product of the frequency characteristic of the original image and the OTF. In addition, since it is preferable that the image does not change before and after defocus blur shaping on the focus surface, a training image and a correct answer image without defocus blur are also generated.

[0045] Next, in step S105, the generating unit 101c generates a correct image and stores it in the storage unit 101a. The correct image is an image (second image) obtained by performing imaging simulation by applying a second defocus blur after shaping to the original image. The correct image and training image may be undeveloped RAW images or developed images. The order of generating the training images and the correct image may be reversed.

[0046] The above is an explanation of how to generate learning data. Note that partial regions with a predetermined number of pixels may be extracted from the training images and the correct image and used for learning.

[0047] Next, weight learning (learning phase) will be described with reference to Fig. 5. Fig. 5 is a flowchart related to weight learning in this embodiment. In this embodiment, CNN is used as the machine learning model, but other models can be similarly applied.

[0048] First, in step S111, the acquisition unit 101b acquires one or more pairs of a correct answer image (second image) and a training image (first image) from the storage unit 101a. The training image is input data in the learning phase of CNN. Next, in step S112, the generation unit 101c inputs the training image to the machine learning model (CNN) and generates an output image.

[0049] Here, the generation of an output image in this embodiment will be described with reference to FIG. 7. FIG. 7 is a configuration diagram of a machine learning model in this embodiment. A training image 201 may be grayscale or may have multiple channel components. In this embodiment, the CNN 202 has one or more convolution layers or full connection layers. In the first learning, the weights of the CNN 202 (each element of the filter and the bias value) are generated by random numbers.

[0050] Next, in step S113 of FIG. 5, the update unit 101d compares the output image with a correct answer image (second image) and updates the weights of the CNN based on the difference (error) between the output image and the correct answer image. In this embodiment, the Euclidean norm of the difference in signal values ​​between the output image and the correct answer image is used as the loss function. However, the loss function is not limited to this. If multiple pairs of training images and correct answer images are obtained in step S111, the value of the loss function is calculated for each pair. The weights are updated from the calculated loss function value by backpropagation or the like.

[0051] Next, in step S114, the update unit 101d determines whether or not learning of the weights is complete. Completion of learning can be determined by whether the number of iterations of learning (updating the weights) has reached a specified number, whether the amount of change in the weights at the time of updating is smaller than a specified value, etc. If it is determined that learning is not complete, the process returns to step S111, and the acquisition unit 101b acquires one or more pairs of new training images and correct images. On the other hand, if it is determined that learning is complete, the update unit 101d ends learning and saves the weight information in the storage unit 101a.

[0052] Next, defocus blur shaping (estimation phase) of a captured image executed by the image estimation device 103 will be described with reference to Fig. 6. Fig. 6 is a flowchart relating to generation of an estimated image in this embodiment.

[0053] First, in step S121, the acquisition unit 103b acquires a captured image and weight information. The captured image to be acquired may be a part of the entire captured image. The weight information is read out from the storage unit 101a in advance and stored in the storage unit 103a.

[0054] Next, in step S122, the blur shaping unit 103c inputs the captured image to the CNN to generate an estimated image. The estimated image is an image in which the defocus blur of the captured image has been shaped. As in the learning process, the estimated image is generated using the machine learning model (CNN) shown in FIG. 7. The acquired trained weights are used in the CNN. The generated estimated images are stored in the storage unit 103a.

[0055] According to the present embodiment, in the shaping of defocus blur using a machine learning model, learning can be performed that can suppress the generation of an image with low shaping accuracy, and defocus blur shaping can be performed.

[0056] Next, a preferred condition for enhancing the effect of this embodiment will be described. The input data preferably further includes a luminance saturation map. The luminance saturation map indicates the luminance saturated pixel area of ​​the image and has the same size as the image. In the learning phase, the luminance saturation map is generated from the training image. In the estimation phase, the luminance saturation map is generated from the captured image. Since false edges different from the structure of the subject due to luminance saturation exist in the luminance saturated part, it is difficult for the machine learning model to distinguish this from defocus blur having high frequency components and focus position, which have edges. The luminance saturation map enables the machine learning model to distinguish between the luminance saturated part, defocus blur having high frequency components, and focus position, enabling high-precision shaping. Note that defocus blur having high frequency components is likely to occur when a PSF with a sharp peak such as double line blur acts.

[0057] The input data preferably further includes an image (A image) obtained by imaging the subject space based on a light beam passing through a partial pupil (second pupil) that is a part of the pupil of the optical system 102a. The image (A+B image) obtained by imaging the subject space based on a light beam passing through the pupil (first pupil) of the optical system 102a and the A image have different defocus blurs, and since the second pupil is smaller than the first pupil, the defocus blur of the A image is smaller than that of the A+B image. By using both the A+B image and the A image, it is possible to distinguish between the defocus blur in the image and the structure of the subject. That is, when there is a blurred area in the image that does not have high-frequency information, it is possible to distinguish whether this area is blurred because it is defocused, or whether it is in focus but appears blurred because the subject does not have high-frequency information. Note that an image (B image) obtained by imaging the subject space based on a light beam passing through a partial pupil (third pupil) that is a part of the pupil of the optical system 102a may be used instead of the A image.

[0058] The input data preferably further includes a state map. The state map is information that represents the state of the optical system 102a during image capture as (Z, F, D). In (Z, F, D), Z corresponds to the zoom, F corresponds to the aperture, and D corresponds to the focus distance.

[0059] The input data preferably further includes a position map, which is a map indicating image plane coordinates for each pixel of the image, which may be in a Cartesian coordinate system or a polar coordinate system (corresponding to image height and azimuth).

[0060] Defocus blur varies depending on the lens state and image height / azimuth. CNN is trained to averagely shape all defocus blur included in the learning data, so the accuracy of shaping for each defocus blur of different shapes decreases. Therefore, by inputting the state map and position map into the machine learning model, the machine learning model can identify the PSF acting on the captured image. As a result, in the learning phase, even if the training image for learning contains defocus blur of various shapes, the machine learning model learns weights that perform different shaping for each shape of defocus blur, rather than weights that averagely shape the defocus blur. As a result, in the estimation phase, high-precision shaping is possible for each defocus blur. Therefore, it is possible to learn learning data that can shape defocus blur of various shapes collectively while suppressing the deterioration of shaping accuracy. EXAMPLES

[0061] Next, an image processing system according to a second embodiment of the present invention will be described. Fig. 11 is a block diagram of an image processing system 300. Fig. 12 is an external view of the image processing system 300.

[0062] The image processing system 300 includes a learning device 301, an imaging device 302, an image estimation device 303, and networks 304 and 305. The learning device 301 includes a storage unit 301a, an acquisition unit 301b, a generation unit 301c, and an update unit 301d, and learns weights of a machine learning model used for defocus blur shaping. Details regarding learning the weights and defocus blur shaping using the weights will be described later.

[0063] The imaging device 302 includes an optical system 302a, an imaging element 302b, an acquisition unit 302c, a recording medium 302d, and a system controller 302e. The optical system 302a collects light incident from a subject space and forms an optical image (subject image). The imaging element 302b converts the optical image into an electrical signal by photoelectric conversion to generate a captured image.

[0064] The image estimation device 303 includes a storage unit 303a, a blur shaping unit 303b, an acquisition unit 303c, and a generation unit 303d. The image estimation device 303 generates an estimated image by shaping defocus blur for a captured image (or at least a part thereof) captured by the imaging device 302. Information on learned weights learned by the learning device 301 is used to generate the estimated image. The weight information is stored in the storage unit 303a. The acquisition unit 302c acquires the estimated image, and the recording medium 302d stores the estimated image. The system controller 302e controls a series of operations of the imaging device 302.

[0065] Next, the generation of learning data executed by the learning device 301 will be described with reference to Fig. 13. Fig. 13 is a flowchart relating to a method of generating learning data in this embodiment.

[0066] First, in step S201, the acquisition unit 301b acquires an original image. Then, in step S202, the generation unit 301c generates a first defocus blur (first blur) and stores it in the storage unit 301a. Then, in step S203, the generation unit 301c generates a second defocus blur (second blur) and stores it in the storage unit 301a.

[0067] Fig. 16 is an explanatory diagram of the optical performance of the first defocus blur in this embodiment, and shows the MTF of the first defocus blur on the axis, with 50% image height and 0 degree azimuth, and with 100% image height and 0 degree azimuth, in a specific lens state and defocus amount. In Fig. 16, the horizontal axis shows the spatial frequency, and the vertical axis shows the MTF. On the axis, with 50% image height and 0 degree azimuth, and with 100% image height and 0 degree azimuth correspond to the dotted line 1008, the dashed line 1009, and the dashed line 1010, respectively. The Nyquist frequency of the image sensor 302b used for imaging is 90 [lp / mm].

[0068] The first defocus blur of this embodiment is a defocus blur with lower optical performance than that of the first embodiment. For example, the on-axis defocus blur when the focus distance is 2.5 m and the defocus amount is 5 m will be a defocus blur with low optical performance when shooting with an optical system with a long focal length or a small F-number (aperture value). Therefore, it is difficult to compare the average value from half the Nyquist frequency to the Nyquist frequency with the second defocus blur shown in FIG. 9. Therefore, in this embodiment, the average values ​​from frequency 0 to half the Nyquist frequency are compared. Frequency 0 is set to f0, half the Nyquist frequency is set to f1, and so on. N / 2 If the number of data points is n, the average value M can be calculated using the following equation (2).

[0069]

number

[0070] In this way, there is no rule for the calculation interval of the average value used for comparing the optical performance, and an appropriate calculation interval may be selected according to the optical performance of the defocus blur. Also, the second defocus blur is determined in the same manner as in the flowchart of FIG. 1, but in this embodiment, the optical performance of the determined defocus blur is set as the upper limit, and two or more types of the second defocus blur are determined. This makes it possible to shape the shape of the defocus blur of the captured image into various shapes. In this embodiment, the average value of the dashed dotted line 1007 in the second defocus blur is lower than the average value of the first defocus blur. Therefore, two or more types of the second defocus blur are determined, and the dashed dotted line 1007 is set as the upper limit of the optical performance in the second defocus blur.

[0071] Next, in step S204 of FIG. 13, the generating unit 301c generates shape designation information and stores it in the storage unit 301a. The shape designation information is information on designation of the size or type of defocus blur after shaping. In this embodiment, the input data input to the machine learning model includes a captured image and information (shape designation information) that designates the shape of the defocus blur after shaping. In learning the machine learning model, the shape designation information is input together with the training image, and a correct answer image corresponding to the shape designation information is set, so that the machine learning model can distinguish and learn a plurality of correct answer images having different defocus blur shapes for one training image. That is, even if the correct answer image contains defocus blur of various shapes, the weight that performs different shaping for each shape of the defocus blur is learned, rather than a weight that shapes the defocus blur to an average shape of the defocus blur. Therefore, learning data including defocus blur of various shapes can be learned collectively with high accuracy.

[0072] Specifying the size is equivalent to virtually changing the F-number (aperture value) of the optical system 302a. Changing the F-number changes the size of the pupil of the optical system 302a, and therefore the size of the defocus blur. By performing image processing to shape the defocus blur, it is also possible to change the F-number to one that the optical system 302a cannot physically remove from the captured image.

[0073] Specifying the type is equivalent to virtually changing the optical system 302a to a different lens configuration. The types of defocus blur, such as double line blur, circular blur, and Gaussian blur, depend on the pupil function determined by the lens configuration of the optical system 302a.

[0074] That is, specifying the size or type of defocus blur after shaping is equivalent to specifying virtual lens parameters. More specifically, specifying the F-number is equivalent to changing the spread of the pupil function. Also, specifying the type of blur, such as double line blur or circular blur, is equivalent to changing the amplitude or phase of the pupil function.

[0075] Next, in step S205, the generating unit 301c generates a training image and stores it in the storage unit 301a. Next, in step S206, the generating unit 301c generates multiple correct images corresponding to each of the multiple pieces of shape designation information for one training image and stores them in the storage unit 301a. The correct image and the training image may be undeveloped RAW images or developed images. In addition, the order in which the training image, correct image, and shape designation information are generated may be reversed.

[0076] Next, weight learning (learning phase) performed by the learning device 301 will be described with reference to FIG. 14. FIG. 4 is a flowchart related to weight learning in this embodiment. In this embodiment, GAN is used as a machine learning model, but other models are also applicable in the same manner. Note that in this embodiment, explanations of the same parts as in embodiment 1 will be omitted. GAN is a generative adversarial network composed of a generator that generates an image and a classifier that identifies the generated image.

[0077] First, in step S211, the acquisition unit 301b acquires one or more pairs of a correct image and training input data from the storage unit 301a. The generation of the correct image and training images is the same as in the first embodiment.

[0078] 10 is a configuration diagram of a machine learning model (GAN) in this embodiment. A concatenation layer 406 concatenates training images 401 and shape designation information 402 in a channel direction in a predetermined order to generate training input data 403.

[0079] Next, in step S212 of FIG. 14, the generating unit 301c inputs the training input data 403 to the generator 407 to generate an output image 404. The generator 407 is, for example, a CNN. Next, in step S213, the updating unit 301d updates the weights of the generator 407 based on the error between the output image 404 and the correct image 405. For the loss function, the Euclidean norm of the difference at each pixel is used. Next, in step S214, the updating unit 301d determines whether the first learning is completed. If the first learning is not completed, the process returns to step S211. On the other hand, if the first learning is completed, the process proceeds to step S221, where the second learning is executed.

[0080] In step S221, the acquisition unit 301b acquires one or more pairs of the correct answer image 405 and the training input data 403 from the storage unit 301a, similar to step S211. Next, in step S222, the generation unit 301c inputs the training input data 403 to the generator 407, similar to step S212, to generate the output image 404. Next, in step S223, the update unit 301d updates the weight of the classifier 408 based on the output image 404 and the correct answer image 405. The classifier 408 classifies whether the input image is a fake image generated by the generator 407 or a real image that is the correct answer image 405. The output image 404 or the correct answer image 405 is input to the classifier 408 to generate a classification label (fake or real). The weights of the classifier 408 are updated based on the error between the classified label and the correct label (the output image 404 is fake, and the correct image 405 is real). Although sigmoid cross entropy is used as the loss function, other loss functions may be used.

[0081] Next, in step S224, the update unit 301d updates the weight of the generator 407 based on the output image 404 and the correct answer image 405. The loss function is a weighted sum of the Euclidean norm and the following two terms. The first term is called Content Loss, which is a term obtained by converting the output image 404 and the correct answer image 405 into a feature map and taking the Euclidean norm of the difference for each element. By adding the difference in the feature map to the loss function, the more abstract nature of the output image 404 can be brought closer to the correct answer image 405. The second term is called Adversarial Loss, which is a sigmoid cross entropy of the classification label obtained by inputting the output image 404 to the classifier 408. By having the classifier 408 learn to classify it as real, it is possible to obtain an output image 404 that subjectively looks more like the correct answer image 405.

[0082] Next, in step S225, the update unit 301d determines whether the second learning is completed. If the second learning is not completed, the process returns to step S221. On the other hand, if the second learning is completed, the update unit 301d stores information on the weights of the learned generator 407 in the storage unit 301a.

[0083] Next, defocus blur shaping (estimation phase) executed by the image estimation device 303 will be described with reference to Fig. 15. Fig. 15 is a flowchart relating to generation of an estimated image in this embodiment.

[0084] First, in step S231, the acquisition unit 303c acquires a captured image (or at least a part thereof). Then, in step S232, the generation unit 303d generates shape designation information. Then, in step S233, the acquisition unit 303c acquires input data and learned weight information. The input data includes a captured image and shape designation information. The weight information is read out in advance from the storage unit 301a and stored in the storage unit 303a. Then, in step S234, the blur shaping unit 303b inputs the input data to the generator 407 and generates an estimated image.

[0085] According to the present embodiment, in the shaping of defocus blur using a machine learning model, learning can be performed that can suppress the generation of an image with low shaping accuracy, and defocus blur shaping can be performed. EXAMPLES

[0086] Next, an image processing system according to a third embodiment of the present invention will be described. The image processing system of this embodiment differs from the first and second embodiments in that it includes a processing device (computer) that transmits a captured image to be subjected to image processing to an image estimation device and receives a processed output image from the image estimation device.

[0087] FIG. 17 is a block diagram of the image processing system 500. FIG. 18 is an external view of the image processing system 500. The image processing system 500 includes a learning device 501, an imaging device 502, a lens device 503, a control device (first device) 504, an image estimation device (second device) 505, and networks 506 and 507. The learning device 501 and the image estimation device 505 are, for example, servers. The control device 504 is a personal computer. The learning device 501 includes a storage unit 501a, an acquisition unit 501b, a generation unit 501c, and an update unit 501d, and learns weights of a machine learning model that corrects the defocus blur of an image captured using the imaging device 502. Details regarding learning will be described later.

[0088] The imaging device 502 has an imaging element 502a, and the imaging element 502a photoelectrically converts an optical image formed by the lens device 503 to obtain a captured image. The lens device 503 and the imaging device 502 are detachable, and can be combined with a plurality of types of each other. The control device 504 has a communication unit 504a, a display unit 504b, and a storage unit 504c, and controls the processing to be executed on the captured image obtained from the imaging device 502 connected by wire or wirelessly according to a user's operation. Alternatively, the captured image captured by the imaging device 502 may be stored in advance in the storage unit 504c, and the captured image may be read out.

[0089] The image estimation device 505 includes a communication unit 505a, an acquisition unit 505b, a storage unit 505c, and a shaping unit 505d. The image estimation device 505 executes defocus blur shaping processing of a captured image in response to a request from a control device 504 connected via a network 506. The image estimation device 505 acquires information on learned weights from a learning device 501 connected via a network 507 at the time of defocus blur shaping or in advance, and uses the information for defocus blur shaping of a captured image. The estimated image after defocus blur shaping is transmitted again to the control device 504, stored in the storage unit 504c, and displayed on the display unit 504b. Note that the generation of learning data and learning of weights (learning phase) performed by the learning device 501 are the same as those in the first embodiment, and therefore their description will be omitted.

[0090] Next, defocus blur shaping (estimation phase) of a captured image executed by the control device 504 and the image estimation device 505 will be described with reference to Fig. 19. Fig. 19 is a flowchart relating to generation of an estimated image in this embodiment.

[0091] First, in step S301, the communication unit (transmission means) 504a of the control device (first device) 504 transmits a captured image and a processing request to the image estimation device (second device) 505. Then, in step S302, the communication unit (reception means) 505a receives and acquires the captured image and the processing request sent. Then, in step S303, the acquisition unit 505b acquires learned weight information from the storage unit 505c. The weight information is read out from the storage unit 501a in advance and stored in the storage unit 505c. Then, in step S304, the shaping unit 505d inputs the input data to the CNN and generates an estimated image in which the defocus blur of the captured image is shaped. Then, in step S305, the communication unit 505a transmits the estimated image to the control device 504. Next, in step S306, the communication unit 504a acquires the transmitted estimated image and stores it in the storage unit 504c.

[0092] According to the present embodiment, in the shaping of defocus blur using a machine learning model, learning can be performed that can suppress the generation of an image with low shaping accuracy, and defocus blur shaping can be performed.

[0093] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0094] The image processing method of each embodiment includes a step of generating a first image (training image) and a second image (correct image) from an original image, and a step of inputting the first image to a machine learning model and learning the machine learning model by comparing the output image with the second image. The first image is an image obtained by adding a first blur (first defocus blur) to the original image. The second image is an image obtained by adding a second blur (second defocus blur) to the original image. Moreover, the first blur has higher optical performance at a specific frequency than the second blur.

[0095] According to each embodiment, it is possible to set an appropriate defocus blur to the correct image and shape the defocus blur of the captured image so that the defocus blur of the correct image does not have a frequency with higher optical performance than the defocus blur of the training image. Therefore, according to each embodiment, it is possible to provide an image processing method, a manufacturing method for a trained model, a program, and an image processing device that can suppress the generation of an image with low shaping accuracy in shaping the defocus blur using a machine learning model.

[0096] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0097] 101 Learning device (image processing device) 101c Generation unit (generation means) 101d Update unit (learning means)

Claims

1. a first step of generating a first image by adding a first blur to an original image, and generating a second image by adding a second blur to the original image; a second step of generating an output image by inputting the first image into a machine learning model, and learning the machine learning model based on the output image and the second image; An image processing method, comprising: at a specific frequency, an optical performance corresponding to the first blur is higher than an optical performance corresponding to the second blur.

2. 2. The image processing method according to claim 1, wherein an average value of the optical performance corresponding to the first blur at a plurality of frequencies is higher than an average value of the optical performance corresponding to the second blur at the plurality of frequencies.

3. 3. The image processing method according to claim 2, wherein the plurality of frequencies are frequencies in a range from half the Nyquist frequency of the first image to the Nyquist frequency.

4. 3. The image processing method according to claim 2, wherein the plurality of frequencies are frequencies in the range from 0 to half the Nyquist frequency of the first image.

5. In the first step, a plurality of first images are generated by adding a plurality of the first blurs to the original image, and a plurality of the second images are generated by adding a plurality of the second blurs to the original image; The shapes of the first blurs are different from each other, 5. The image processing method according to claim 1, wherein the shapes of the second blurs are the same as each other.

6. In the first step, a plurality of first images are generated by adding a plurality of the first blurs to the original image, and a plurality of the second images are generated by adding a plurality of the second blurs to the original image; The shapes of the first blurs are different from each other, 5. The image processing method according to claim 1, wherein the shapes of the second blurs are different from each other.

7. A step of acquiring a machine learning model trained by the image processing method according to any one of claims 1 to 6 and an input image; and generating an estimated image by inputting the input image into the machine learning model.

2. An image processing method according to claim 1, wherein the estimated image is an image having a blur of a different shape from that of the input image.

8. 8. The image processing method according to claim 1, wherein the first blur and the second blur are defocus blurs.

9. 9. The image processing method according to claim 1, wherein the optical performance comprises a modulation transfer function or an optical transfer function.

10. A program for causing a computer to execute the image processing method according to any one of claims 1 to 9.

11. a first step of generating a first image by adding a first blur to an original image, and generating a second image by adding a second blur to the original image; a second step of generating an output image by inputting the first image into a machine learning model, and learning the machine learning model based on the output image and the second image; A method for generating a trained model, characterized in that at a specific frequency, optical performance corresponding to the first blur is higher than optical performance corresponding to the second blur.

12. a generating means for generating a first image by adding a first blur to an original image, and generating a second image by adding a second blur to the original image; A learning means for generating an output image by inputting the first image into a machine learning model, and learning the machine learning model based on the output image and the second image, 13. An image processing device, comprising: an optical performance corresponding to the first blurring is higher than an optical performance corresponding to the second blurring at a specific frequency.

Images