Learning method, program, learning device, and generation method

Abstract

To provide an image processing method that, even when parallax changes by a little amount, can convert blur caused by defocusing of a picked-up image to obtain an image with a favorable blur taste.SOLUTION: An image processing method for converting defocusing blur of a picked-up image into blur in a different shape includes the steps of: acquiring an original image; setting a defocusing amount; creating a training image and a correct answer image based on the defocusing amount and original image; and comparing an output image created by inputting the training image to a machine learning model with the correct answer image to learn the machine learning model. When a value related to the defocusing amount is smaller than a threshold, the resolution of the correct answer image is equal to or higher than the resolution of the training image, and when the value related to the defocusing amount is larger than the threshold, the resolution of the correct answer image is lower than the resolution of the training image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention converts the blur caused by defocus in a captured image to obtain an image with good blur. How to learn Regarding. [Background technology]

[0002] Patent Document 1 discloses a method for converting blur caused by defocusing an image using parallax images obtained through different pupils of an optical system and a machine learning model to obtain an image with good blur. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-36310 A Summary of the Invention [Problem to be solved by the invention]

[0004] The method disclosed in Patent Document 1 converts the blur caused by defocus in a captured image by inputting parallax images obtained through different pupils of an optical system into a machine learning model, thereby obtaining an image with good blur. By using parallax images, it is possible to distinguish between the in-focus plane and the defocus in an image. However, depending on the optical system used for capturing images, the change in the amount of parallax may be small, and the machine learning model may not be able to distinguish between the in-focus plane and the defocus, resulting in learning errors. Therefore, when the change in the amount of parallax is small, the accuracy of the conversion decreases.

[0005] Therefore, the present invention aims to provide an image processing method, a program, an image processing device, and a method for manufacturing a trained model that are capable of converting the blur caused by defocus in a captured image to obtain an image with good blur even when the change in the amount of parallax is small. [Means for solving the problem]

[0006] A learning method according to one aspect of the present invention is a learning method for a trained model that converts defocus blur in a captured image into blur of a different shape, the method comprising the steps of: acquiring an original image and a defocus amount; ,before Original image and The defocus amount, generating a training image and a correct answer image based on the above; and learning the machine learning model based on an output image generated by inputting the training image to the machine learning model and the correct answer image. And Has In the generating step, The defocus The amount If the resolution is smaller than the threshold, of The resolution of the training images is equal to or greater than the resolution of the training images. Defocus Amount If is greater than the threshold, the resolution of the correct image of Resolution of the training images Make less than .

[0007] Other objects and features of the present invention are illustrated in the following examples. Effect of the Invention

[0008] According to the present invention, it is possible to provide an image processing method, a program, an image processing device, and a method for manufacturing a trained model that are capable of converting the blur caused by defocus in a captured image to obtain an image with good blur, even when the change in the amount of parallax is small. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a configuration diagram of a machine learning model in a first embodiment. [Diagram 2] 1 is a block diagram of an image processing system according to a first embodiment. [Diagram 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4] FIG. 2 is an explanatory diagram of an imaging device in the first embodiment. [Diagram 5] 4 is a flowchart relating to a method for generating learning data in the first embodiment. [Figure 6]4 is a flowchart relating to a weight learning method in the first embodiment. [Figure 7] 1 is a flowchart relating to a method of generating an estimated image in the first and second embodiments. [Figure 8] 4 is a diagram showing a PSF formed by a first pupil and a point spread function formed by a second pupil in the first embodiment. FIG. [Figure 9] 5 is a diagram showing the relationship between the amount of parallax and the subject distance in the first embodiment. FIG. [Figure 10] 6 is a diagram showing a change in luminance value of a focused object in Example 1. FIG. [Figure 11] FIG. 11 is a configuration diagram of a machine learning model in the second embodiment. [Figure 12] FIG. 11 is a block diagram of an image processing system according to a second embodiment. [Figure 13] FIG. 11 is an external view of an image processing system according to a second embodiment. [Figure 14] 13 is a flowchart relating to a method for generating learning data in the second embodiment. [Figure 15] 13 is a flowchart showing a weight learning method according to the second embodiment. [Figure 16] FIG. 11 is a diagram illustrating the accuracy of a defocus map in the second embodiment. [Figure 17] 13 is a flowchart relating to a method of generating an estimated image in the second embodiment. [Figure 18] FIG. 11 is a block diagram of an image processing system according to a third embodiment. [Figure 19] FIG. 11 is an external view of an image processing system according to a third embodiment. [Figure 20] 13 is a flowchart related to generation of an estimated image in the third embodiment. [Figure 21] 6A to 6C are diagrams illustrating point image intensity distributions at defocus distances in the respective examples. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0011] Before describing the specific embodiments, the gist of the present invention will be described. The present invention converts the defocus blur of a captured image using a machine learning model. The machine learning model includes, for example, a neural network, genetic programming, a Bayesian network, and the like. The neural network includes a CNN (Convolutional Neural Network), a GAN (Generative Adversarial Network), an RNN (Recurrent Neural Network), and the like. The conversion of the defocus blur includes conversion from a two-line blur to a Gaussian blur or a circular blur.

[0012] With reference to Figs. 21(A) to (C), double-line blur, ball blur, and Gaussian blur will be described. Fig. 21(A) is a diagram showing the PSF (point spread function) of double-line blur. In Fig. 21(A), the horizontal axis indicates spatial coordinates (position), and the vertical axis indicates intensity. This point is the same for Figs. 21(B) and (C) described later. As shown in Fig. 21(A), double-line blur has a PSF with a separated peak. When the PSF at the defocus distance has a shape as shown in Fig. 21(A), an object that is originally a single line appears to be doubly blurred when defocused. When the object is a point image, it appears to be ring-shaped when defocused, so it is also called ring blur. Fig. 21(B) is a diagram showing the PSF of ball blur. Ball blur has a PSF with flat intensity, and is also called disk blur. Fig. 21(C) is a diagram showing the PSF of Gaussian blur. Gaussian blur has a PSF with smooth edges like a Gaussian distribution. Other defocus blurs that can be converted include, for example, defocus blur missing due to vignetting, and ring-shaped defocus blur caused by pupil obstruction by a catadioptric lens or the like. There is no restriction on the shape of the defocus blur that can be converted.

[0013] The conversion of defocus blur in each embodiment does not mean adding defocus blur to a pan-focus image with a deep depth of field to reproduce an image with a shallow depth of field. The object that is already defocus blurred is converted to a desired defocus blur. In other words, it is necessary to apply a defocus blur that satisfies the difference between the defocus blur that is already applied and the desired defocus blur, and more advanced processing is required.

[0014] In the conversion of defocus blur using a machine learning model, whether or not the in-focus plane and the defocus can be distinguished and converted is an important issue. Therefore, it is preferable to use information on the defocus amount together with the captured image. For example, a parallax image acquired through different pupils of the optical system is input to the machine learning model together with the captured image. The parallax amount of the parallax image acquired through different pupils of the optical system changes according to the amount of deviation from the in-focus plane. Therefore, the in-focus plane and the defocus can be distinguished. The image sensor that divides the pupil of the optical system to acquire information will be described in detail in Example 1.

[0015] However, depending on the optical system used for imaging, the change in the amount of parallax may be small, and the machine learning model may not be able to distinguish between the in-focus plane and the defocus, resulting in a learning error. In this case, the machine learning model cannot recognize the difference in the amount of parallax, and learns to output the average value of multiple correct images. In particular, the in-focus plane may be blurred near the in-focus plane, which may have a large impact on the image quality of the captured image. Therefore, in each embodiment, a threshold is set to generate learning images and perform learning so that the in-focus plane and the defocus can be distinguished and converted. This allows learning to be performed that can convert the defocus blur of the captured image with high accuracy, even when the change in the amount of parallax is small.

[0016] In the following, the stage of learning the weights of the machine learning model will be referred to as the learning phase, and the stage of converting defocus blur using the machine learning model using the learned weights will be referred to as the estimation phase. EXAMPLES

[0017] First, an image processing system in the first embodiment of the present invention will be described with reference to FIG. 2 and FIG. 3. FIG. 2 is a block diagram of the image processing system 100. FIG. 3 is an external view of the image processing system 100. The image processing system 100 includes a learning device (image processing device) 101, an imaging device 102, an image estimation device 103, a display device 104, a recording medium 105, an output device 106, and a network 107. The learning device 101 includes a storage unit 101a, an acquisition unit 101b, a generation unit 101c, and an update unit 101d, and learns weights of a machine learning model used in defocus blur conversion. Details of weight learning and defocus blur conversion processing using the weights will be described later. The imaging device 102 includes an optical system 102a and an image sensor 102b, and captures an image of a subject space to obtain a captured image. The optical system 102a collects light incident from the subject space to form an optical image (subject image). The image sensor 102b obtains a captured image by photoelectrically converting the optical image.

[0018] Here, the configuration of the imaging device 102 will be described with reference to Figs. 4(A) and (B). Figs. 4(A) and (B) are explanatory diagrams of the imaging device 102. Fig. 4(A) shows a cross-sectional view of the imaging unit 102, and the dashed line represents an axial light beam. Fig. 4(B) shows a top view of the imaging element 102b. The imaging element 102b has a microlens array 122 and a plurality of pixels 121. The microlens array 122 is disposed at a position conjugate with the object plane 120 via the optical system 102a. As shown in Fig. 4(B), the microlenses 122a (only the microlenses 122a are shown, and 122b and subsequent pixels are omitted) constituting the microlens array 122 correspond to each of the plurality of pixels 121 (only the pixels 121a are shown, and 121b and subsequent pixels are omitted). Here, when multiple parts are collectively designated, only numbers are given, and when one of them is designated, the number and a symbol such as a are given.

[0019] Each of the pixels 121 has a first photoelectric conversion unit 123 and a second photoelectric conversion unit 124 that photoelectrically convert an optical image formed through the optical system 102a. As a result, for example, light incident on the pixel 121a is separated and received by the first photoelectric conversion unit 123a and the second photoelectric conversion unit 124a depending on the incident angle (the first photoelectric conversion unit 123a and the second photoelectric conversion unit 124a receive light incident at different incident angles). The incident angle of light is determined by which position of the pupil in the optical system 102a the light passes through. For this reason, the pupil of the optical system 102a is divided into two partial pupils by the two photoelectric conversion units, and the two photoelectric conversion units in one pixel obtain information obtained by observing the subject space from different viewpoints (pupil positions). In this embodiment, the pupil is divided in a horizontal direction, but this is not limited to this, and may be in other directions such as a vertical direction or an oblique direction.

[0020] The imaging element 102b outputs a signal (A image) acquired by the first photoelectric conversion unit 123 and a signal (A+B image) obtained by adding this signal (A image) and a signal (B image) acquired by the second photoelectric conversion unit 124. Thus, in this embodiment, the A+B image and the A image are images obtained by simultaneously imaging the subject space via the optical system 102a. The A+B image is an image acquired via the entire pupil (first pupil) of the optical system 102a, and the A image is an image acquired via a partial pupil (second pupil) that is a part of the pupil. Also, in this embodiment, the A+B image and the A image are images captured by the same imaging element 102b.

[0021] The image estimation device 103 includes a storage unit 103a, an acquisition unit 103b, a blur conversion unit 103c, and a generation unit 103d. The image estimation device 103 generates an estimated image by converting the defocus blur of a captured image (or at least a part thereof) captured by the imaging device 102. For the defocus blur conversion, a machine learning model using weights learned by the learning device 101 is used. The learning device 101 and the image estimation device 103 are connected by a network 107, and the image estimation device 103 reads out information on the learned weights from the learning device 101 during or before the defocus blur conversion. The estimated image is output to at least one of the display device 104, the recording medium 105, and the output device 106. The display device 104 is, for example, a liquid crystal display or a projector. A user can perform editing work and the like while checking an image being processed through the display device 104. Details of a user interface during editing work will be described later. The recording medium 105 is, for example, a semiconductor memory, a hard disk, a server on a network, etc., and stores the estimated image. The output device 106 is, for example, a printer.

[0022] Next, the generation of learning data executed by the learning device 101 will be described with reference to Fig. 5. Fig. 5 is a flowchart relating to a method of generating learning data.

[0023] First, in step S101, the acquisition unit (acquisition means) 101b acquires an original image. The original image may be one or more images. The original image may be a real-life image or a CG (Computer Graphics) image. In the following steps, the original image is blurred to generate a training image and a correct answer image. Therefore, it is desirable that the original image has edges with various intensities and directions, textures, gradations, flat parts, etc., so that the shape conversion of the defocus blur can be performed correctly for various subjects.

[0024] Preferably, the original image has a signal value higher than the brightness saturation value of the image sensor 102b. This is because even in real subjects, there are subjects that do not fall within the brightness saturation value when imaged by the image sensor 102 under specific exposure conditions. The ground truth image and the training image are subjected to defocus blur and then clipped by the brightness saturation value of the image sensor 102b, thereby reproducing subjects that do not fall within the actual brightness saturation value.

[0025] Next, in step S102, the acquisition unit (setting means) 101b sets and acquires a defocus amount. It is preferable to set various defocus amounts to accommodate various captured images. Next, in step S103, the generation unit 101c generates optical characteristics formed through the first pupil based on the acquired defocus amount, and stores the optical characteristics in the storage unit 101a. The first pupil is the entire area of ​​the pupil of the optical system 102a. The optical characteristics include a PSF or an OTF (optical transfer function). Next, in step S104, the generation unit 101c generates optical characteristics formed through the second pupil based on the acquired defocus amount, and stores the optical characteristics in the storage unit 101a. The second pupil is a partial pupil that is a part of the pupil of the optical system 102a.

[0026] Next, in step S105, the acquisition unit 101b acquires the amount of parallax between the optical characteristics of the first pupil and the optical characteristics of the second pupil. When the optical characteristics are PSFs, the amount of parallax can be acquired by taking the difference between the centers of gravity. FIGS. 8A and 8B show the PSFs formed by the first pupil and the PSFs formed by the second pupil. FIG. 8A is a cross-sectional view of the PSFs of the first pupil (solid line 7001) and the PSFs of the second pupil (dotted line 7002) on the in-focus plane. FIG. 8B is the PSFs of the first pupil (solid line 7003) and the PSFs of the second pupil (dotted line 7004) in a defocused state. The amount of parallax is small on the in-focus plane, and large on the defocused plane.

[0027] 5, the update unit 101d compares the acquired amount of parallax (a value related to the amount of defocus set in step S102) with a threshold. The reason for comparing the amount of parallax with the threshold is that the change in the amount of parallax may be small depending on the imaging conditions. For example, the change in the amount of parallax becomes smaller as the focusing position of the optical system 102a becomes farther away.

[0028] FIG. 9 shows the relationship between the amount of parallax and the object distance when the optical system 102a has a focal length of 50 mm, an F value of 1.4, a focus position of 2500 mm, and a pixel pitch of the image sensor 102b of 5.5 μm. In FIG. 9, the horizontal axis indicates the object distance (m), and the vertical axis indicates the amount of parallax (pixels). At focus positions of 2500 mm and 3000 mm, the difference in the amount of parallax is about 2 pixels. Here, assume that the object at the focus position of 2500 mm is not changed, and the defocus blur at 3000 mm is learned to be converted. At this time, since the change in the amount of parallax is small, the machine learning model cannot distinguish between the focus plane and the defocus, resulting in a learning error. In this case, the machine learning model cannot recognize the difference in the amount of parallax, and is trained to output the average value of multiple correct images.

[0029] Therefore, in this embodiment, a threshold is set, and a learning image is generated based on a comparison between the amount of parallax and the threshold. Specifically, the threshold is set to 5 pixels, and when the amount of parallax is equal to or less than the threshold, the resolution of the correct image and the training image is set to be the same. On the other hand, when the amount of parallax is equal to or more than the threshold, the resolution of the correct image is set to be smaller than that of the training image. Here, the resolution refers to the frequency characteristics of the image, and high resolution means that the image contains many high-frequency components. This makes it possible to learn to distinguish and convert the in-focus plane and the defocused plane. The threshold is a value equal to or greater than the value corresponding to the focal depth of the imaging device used to capture the captured image. The focal depth is determined by the F-number (aperture value) of the optical system 102a and the pixel pitch of the imaging element 102b. When the F-number is F and the pixel pitch is δ, the focal depth f s can be calculated using the following formula (1).

[0030]

number

[0031] For example, when the F value is 1.4 and the pixel pitch is 5.5 μm, the focal depth is 7.7 μm. The parallax amount of 5 pixels, which is the threshold in this embodiment, is when the subject distance is 4000 mm. The imaging position at this time can be determined geometrically and optically to be approximately 290 μm forward from the imaging element. In other words, the threshold value (parallax amount of 5 pixels) is a value that corresponds to a focal depth of the imaging device or greater.

[0032] In addition, it is preferable that the defocus amount to be learned is separated from the threshold by a value corresponding to the focal depth or more (the defocus amount and the threshold have an interval of at least the value corresponding to the focal depth). This makes it possible to distinguish between the parallax amount to be learned as the in-focus plane and the parallax amount to be learned as the defocus, thereby improving the accuracy of conversion in the defocus amount near the threshold.

[0033] Here, the change in luminance value of the focused object will be described with reference to Fig. 10. Fig. 10 is a diagram showing the change in luminance value of the focused object when a captured image is converted using weights learned without setting a threshold (dashed line 1003), and the change in luminance value when weights learned based on a comparison between the amount of parallax and a threshold (dotted line 1002) are used. In Fig. 10, the solid line 1001 is the luminance value of the focused object in the captured image. When learning is performed without setting a threshold, edges are lost and the focused object is blurred. On the other hand, when learning is performed based on a comparison between the amount of parallax and a threshold, edges are maintained and the focused object is not blurred.

[0034] Next, in step S107 of FIG. 5, the generating unit 101c generates conversion information. The conversion information is an optical characteristic that is applied to the original image when generating the correct image. In this embodiment, when the amount of parallax is equal to or less than a threshold, the correct image and the training image have the same resolution. On the other hand, when the amount of parallax is greater than the threshold, the resolution of the correct image is made lower than the resolution of the training image. Note that when it is desired to sharpen the in-focus plane, when the amount of parallax is equal to or less than a threshold, the resolution of the correct image may be made higher than the resolution of the training image. This makes it possible to convert the in-focus plane while achieving both sharpening and defocus blur. Here, sharpening refers to a process of restoring the frequency components of the subject that have been reduced or lost due to blur.

[0035] Next, in step S108, the generating unit 101c generates training images and stores the training images in the storage unit 101a. The training images are parallax images (A+B image, A image) obtained by performing imaging simulation by applying the optical characteristics of the first pupil and the optical characteristics of the second pupil to the original images. In this embodiment, the training images are A+B image and A image, but they may be A image and B image or A+B image and B image. The application of the optical characteristics can be performed by convolving the PSF with the original images or by taking the product of the frequency characteristics of the original images and the OTF.

[0036] Next, in step S109, the generating unit 101c generates a correct image and stores the correct image in the storage unit 101a. The correct image is an image obtained by performing a shooting simulation by applying conversion information to an original image. The correct image and the training image may be undeveloped RAW images or developed images. The order in which the training images and the correct image are generated may be reversed.

[0037] The above is an explanation of how to generate learning data. Note that partial regions with a predetermined number of pixels may be extracted from the training images and the correct image and used for learning.

[0038] Next, learning of weights in a machine learning model (learning phase) will be described with reference to Fig. 6. Fig. 6 is a flowchart of a weight learning method. Note that in this embodiment, CNN is used as a machine learning model, but the present invention is not limited to this, and other models can be similarly applied.

[0039] First, in step S111, the acquisition unit 101b acquires one or more pairs of a correct answer image and a training image from the storage unit 101a. The training image is input data in the learning phase of the CNN. Then, in step S112, the generation unit 101c inputs the training image to the CNN and generates an output image.

[0040] Here, with reference to FIG. 1, the generation of an output image in this embodiment will be described. FIG. 1 is a configuration diagram of a machine learning model in this embodiment. A training image 201 may be grayscale or may have multiple channel components. In this embodiment, the CNN 202 has one or more convolution layers or full connection layers. In the first learning, the weights of the CNN 202 (each element of the filter and the bias value) are generated by random numbers. As shown in FIG. 1, the training image 201 is input to the CNN 202, and an output image 203 is generated.

[0041] Next, in step S113 of FIG. 6, the update unit (learning means) 101d updates the weights of the CNN based on the error between the output image and the correct image (using a loss function). In this embodiment, the Euclidean norm of the difference in signal values ​​between the output image and the correct image is used as the loss function. However, the loss function is not limited to this. If multiple pairs of training images and correct images are obtained in step S111, the value of the loss function is calculated for each pair. The weights are updated from the calculated loss function value by backpropagation or the like.

[0042] Next, in step S114, the update unit 101d determines whether or not weight learning is complete. Completion of weight learning can be determined by, for example, whether the number of iterations of learning (weight update) has reached a specified number, or whether the amount of change in weight at the time of update is smaller than a specified value. If it is determined that learning is not complete, the process returns to step S111, and the acquisition unit 101b acquires one or more pairs of new training images and correct images. On the other hand, if it is determined that learning is complete, the learning device 101 ends learning and stores weight information in the storage unit 101a.

[0043] Next, the defocus blur conversion (estimation phase) of the captured image executed by the image estimation device 103 will be described with reference to Fig. 7. Fig. 7 is a flowchart relating to a method of generating an estimated image. Note that the estimation does not necessarily have to be performed by the image estimation device 103, and edge processing may be performed within the imaging device 102.

[0044] First, in step S121, the acquisition unit 103b acquires information on the captured image and the weights. The captured image includes an A+B image and an A image. Alternatively, if an A+B image and a B image are used as training images during learning, the image may be an A+B image and a B image, or if an A image and a B image are used as training images, the image may be an A image and a B image. The captured image to be acquired may be a part of the entire captured image. The weight information is read out from the storage unit 101a in advance and stored in the storage unit 103a. Next, in step S122, the blur conversion unit 103c inputs the captured image to the CNN to generate an estimated image. The estimated image is an image in which the defocus blur of the captured image is converted. As in the case of learning, the estimated image is generated using the CNN shown in FIG. 1. The acquired learned weights are used for the CNN. The generated estimated images are stored in the storage unit 103a. With the above configuration, even if the change in the amount of parallax is small, the defocus blur can be converted to obtain an image with good blur.

[0045] Next, a description will be given of a preferable condition for enhancing the effect of this embodiment. The input data preferably further includes a luminance saturation map. The luminance saturation map indicates the luminance saturated pixel area of ​​the image and has the same size as the image. In the learning phase, the luminance saturation map is generated from the training image. In the estimation phase, the luminance saturation map is generated from the captured image. Since false edges different from the structure of the subject due to luminance saturation exist in the luminance saturated part, it is difficult for the machine learning model to distinguish this from defocus blur having high frequency components and focus position, which have edges. The luminance saturation map enables the machine learning model to distinguish between the luminance saturated part, defocus blur having high frequency components, and focus position, enabling highly accurate conversion. Note that defocus blur having high frequency components is likely to occur when a PSF with a sharp peak such as double line blur acts.

[0046] The input data preferably further includes a state map. The state map is information that represents the state of the optical system 102a during image capture as (Z, F, D). In (Z, F, D), Z corresponds to the zoom, F corresponds to the aperture, and D corresponds to the focus distance.

[0047] The input data preferably further includes a position map, which is a map indicating image plane coordinates for each pixel of the image, which may be in a Cartesian coordinate system or a polar coordinate system (corresponding to image height and azimuth).

[0048] Defocus blur changes depending on the lens state and image height / azimuth. CNN is trained to convert all defocus blur included in the learning data on an average basis, so the accuracy of conversion for each defocus blur of different shapes decreases. Therefore, by inputting the state map and position map into the machine learning model, the machine learning model can identify the PSF acting on the captured image. As a result, in the learning phase, even if the training image for learning contains defocus blur of various shapes, the machine learning model learns weights that perform different conversions for each shape of defocus blur, rather than weights that convert those defocus blurs on an average basis. As a result, in the estimation phase, highly accurate conversion is possible for each defocus blur. Therefore, it is possible to learn learning data that can convert defocus blur of various shapes collectively while suppressing the decrease in conversion accuracy. EXAMPLES

[0049] Next, an image processing system according to a second embodiment of the present invention will be described with reference to Figs. 12 and 13. In this embodiment, a defocus map is used as information related to the defocus amount of a captured image. A small change in the amount of parallax described in the first embodiment corresponds to a low resolution of the defocus map in this embodiment. Fig. 12 is a block diagram of an image processing system 300. Fig. 13 is an external view of the image processing system 300. The image processing system 300 includes a learning device (image processing device) 301, an imaging device 302, an image estimation device 303, and networks 304 and 305.

[0050] The learning device 301 has a storage unit 301a, an acquisition unit 301b, a generation unit 301c, and an update unit 301d, and learns weights of a machine learning model used in defocus blur conversion. Details regarding weight learning and defocus blur conversion using the weights will be described later. The imaging device 302 has an optical system 302a, an imaging element 302b, an acquisition unit 302c, a recording medium 302d, a display unit 302e, and a system controller 302f. The optical system 302a collects light incident from the subject space and forms an optical image (subject image). The imaging element 302b converts the optical image into an electrical signal by photoelectric conversion, and generates a captured image.

[0051] The image estimation device 303 includes a storage unit 303a, a blur conversion unit 303b, an acquisition unit 303c, and a generation unit 303d. The image estimation device 303 generates an estimated image by converting defocus blur of a captured image (or at least a part thereof) captured by the imaging device 302. Information on learned weights learned by the learning device 301 is used to generate the estimated image. The weight information is stored in the storage unit 303a. The acquisition unit 302c acquires the estimated image, and the recording medium 302d stores the estimated image. The system controller 302f controls a series of operations of the imaging device 302.

[0052] Next, the generation of learning data executed by the learning device 301 will be described with reference to Fig. 14. Fig. 14 is a flowchart relating to a method of generating learning data.

[0053] First, in step S201, the acquisition unit 301b acquires an original image. Then, in step S202, the acquisition unit 301b sets and acquires a defocus amount. Then, in step S203, the generation unit 301c generates optical characteristics formed through the first pupil based on the acquired defocus amount, and stores the optical characteristics in the storage unit 301a. The first pupil is the entire area of ​​the pupil of the optical system 302a.

[0054] Next, in step S204, the generation unit 301c generates a defocus map based on the acquired defocus amount. The defocus map has the same number of elements (number of pixels) as one channel component of the training image, and indicates the defocus amount of the subject as a numerical value. In this embodiment, this numerical value is the defocus value. During shooting, the defocus map can be acquired by using a ToF (Time Of Flight) sensor, a parallax image, a DFD (Depth from Defocus), or the like. As an example, a case will be shown in which the defocus map has numerical values ​​normalized based on the possible range of subject distance. Let L be the subject distance, and the minimum value of the subject distance be L. min , the maximum value is L max Then, the normalized object distance L n is calculated using the following formula (2).

[0055]

number

[0056] There is no restriction on how the values ​​are taken, and the closest point may be 1, and the farthest distance from the imaging device may be 0. In this embodiment, the defocus value of the in-focus plane is 0, the closest point is -1, and the farthest distance from the imaging device is 1.

[0057] Next, in step S205, the acquisition unit 301b acquires the absolute value of the defocus value from the defocus map. Next, in step S206, the update unit 301d compares the acquired absolute value of the defocus value with a threshold value.

[0058] Here, the accuracy of the defocus map will be described with reference to Figs. 16(A) and (B). Figs. 16(A) and (B) are diagrams showing the accuracy of the defocus map. In Figs. 16(A) and (B), the horizontal axis indicates the defocus amount, and the vertical axis indicates the defocus value. In an ideal defocus map, as shown in Fig. 16(A), the defocus value changes continuously according to the defocus amount. However, the defocus value that can actually be acquired during shooting is discrete as shown in Fig. 16(B) because of a limit in resolution. Therefore, the defocus amount is different but the defocus value is the same. For example, when the resolution is 1m, the in-focus plane has a common defocus value 1m behind. In this case, if the in-focus plane is not changed and the defocus blur up to 1m behind is learned to be converted, the machine learning model will learn multiple correct images for one type of training image, and the accuracy of the conversion will decrease. Therefore, when the defocus value of the in-focus plane is set to 0, the absolute value of the defocus value is learned as being less than a threshold as the in-focus plane, and as being greater than or equal to the threshold as the defocus. As described above, when the resolution is 1 m, it is preferable to set the absolute value of the defocus value 1 m behind as the threshold. In other words, the threshold is determined by the resolution of the defocus map acquired during imaging. This allows the conversion of defocus blur without significantly changing the focal plane, even if the resolution of the defocus map is low.

[0059] 14, the generating unit 301c generates conversion information. If the absolute value of the defocus value (the value related to the defocus amount set in step S202) is smaller than the threshold, the correct image and the training image have the same resolution. On the other hand, if the absolute value of the defocus value is equal to or greater than the threshold, the resolution of the correct image is made lower than that of the training image.

[0060] Next, in step S208, the generation unit 301c generates a training image and stores the training image in the storage unit 301a. The training image is generated by applying the optical characteristics of the first pupil to the original image. Next, in step S209, the generation unit 301c generates a correct image and stores the correct image in the storage unit 301a. The correct image is generated by applying conversion information to the original image. The correct image and the training image may be undeveloped RAW images or developed images. The order in which the training images and the correct images are generated may be reversed.

[0061] Next, weight learning (learning phase) performed by the learning device 301 will be described with reference to FIG. 15. FIG. 15 is a flowchart of a weight learning method. In this embodiment, GAN is used as a machine learning model, but the present invention is not limited to this, and other models can be similarly applied. Note that the description of the same parts as in the first embodiment will be omitted. GAN is a generative adversarial network that is composed of a generator that generates an image and a classifier that identifies the generated image.

[0062] First, in step S211, the acquisition unit 301b acquires one or more pairs of a correct answer image and training input data from the storage unit 301a. The generation of the correct answer image and the training image is the same as in the first embodiment. Fig. 11 is a configuration diagram of the machine learning model (GAN) of this embodiment. The concatenation layer 406 concatenates the training image 401 and the defocus map 402 in a predetermined order in the channel direction to generate the training input data 403.

[0063] Next, in step S212, the generating unit 301c inputs the training input data 403 to the generator 407 to generate an output image 404. The generator 407 is, for example, a CNN. Next, in step S213, the updating unit 301d updates the weights of the generator 407 based on the error between the output image 404 and the correct image 405 (using a loss function). The loss function uses the Euclidean norm of the difference at each pixel.

[0064] Next, in step S214, the update unit 301d determines whether the first learning is complete. If the first learning is not complete, the process returns to step S211. On the other hand, if the first learning is complete, the process proceeds to step S221, where the learning device 301 executes the second learning.

[0065] In step S221, similar to step S211, the acquisition unit 301b acquires one or more pairs of a ground truth image 405 and training input data 403 from the storage unit 301a. Then, in step S222, the generation unit 301c inputs the training input data 403 to the generator 407 to generate an output image 404, similar to step S212.

[0066] Next, in step S223, the update unit 301d updates the weights of the classifier 408 from the output image 404 and the correct answer image 405. The classifier 408 classifies whether the input image is a fake image generated by the generator 407 or a real image that is the correct answer image 405. The output image 404 or the correct answer image 405 is input to the classifier 408 to generate a classification label (fake or real). The weights of the classifier 408 are updated based on the error between the classification label and the correct answer label (the output image 404 is fake, and the correct answer image 405 is real) (using a loss function). Although sigmoid cross entropy is used as the loss function, this is not limited thereto, and other loss functions may be used.

[0067] Next, in step S224, the update unit 301d updates the weight of the generator 407 based on the output image 404 and the correct answer image 405. The loss function is a weighted sum of the Euclidean norm of step S213 and the following two terms. The first term is called Content Loss, which is a term obtained by converting the output image 404 and the correct answer image 405 into a feature map and taking the Euclidean norm of the difference for each element. By adding the difference in the feature map to the loss function, the more abstract nature of the output image 404 can be brought closer to the correct answer image 405. The second term is called Adversarial Loss, which is a sigmoid cross entropy of the classification label obtained by inputting the output image 404 to the classifier 408. By learning to classify the output image 404 as real, the classifier 408 can obtain the output image 404 that subjectively looks more like the correct answer image 405.

[0068] Next, in step S225, the update unit 301d determines whether the second learning is completed. As in step S214, if the second learning is not completed, the process returns to step S221. On the other hand, if the second learning is completed, the learning device 301 stores information on the weights of the learned generator 407 in the storage unit 301a.

[0069] Next, the defocus blur transformation (estimation phase) executed by the image estimation device 303 will be described with reference to Fig. 17. Fig. 17 is a flowchart relating to a method for generating an estimated image. First, in step S231, the acquisition unit 303c acquires information on a captured image (or at least a part thereof), a defocus map, and learned weights. The weight information is read out in advance from the storage unit 301a and stored in the storage unit 303a. Next, in step S232, the blur transformation unit 303b inputs input data to the generator 407 and generates an estimated image.

[0070] With the above configuration, even if the resolution of the defocus map is low, it is possible to convert the blur caused by defocus and obtain an image with good blur. EXAMPLES

[0071] Next, an image processing system according to a third embodiment of the present invention will be described with reference to Figs. 18 and 19. The image processing system of this embodiment differs from the first and second embodiments in that it has a processing device (computer) that transmits a captured image to be subjected to image processing to an image estimation device and receives a processed output image from the image estimation device. Fig. 18 is a block diagram of an image processing system 500. Fig. 19 is an external view of the image processing system 500. The image processing system 500 has a learning device (image processing device) 501, an imaging device 502, a lens device 503, a control device (first device) 504, an image estimation device (second device) 505, and networks 506 and 507.

[0072] The learning device 501 and the image estimation device 505 are, for example, a server. The control device 504 is a personal computer. The learning device 501 has a storage unit 501a, an acquisition unit 501b, a generation unit 501c, and an update unit 501d, and learns weights of a machine learning model that converts defocus blur of an image captured using the imaging device 502. Details of learning will be described later. The imaging device 502 has an imaging element 502a, which photoelectrically converts an optical image formed by the lens device 503 to obtain a captured image. The lens device 503 and the imaging device 502 are detachable and can be combined with a plurality of types of each other.

[0073] The control device 504 has a communication unit 504a, a display unit 504b, and a storage unit 504c, and controls, in accordance with a user's operation, a process to be executed on a captured image acquired from the imaging device 502 connected by wire or wirelessly. Alternatively, the captured image captured by the imaging device 502 may be stored in advance in the storage unit 504c, and the captured image may be read out.

[0074] The image estimation device 505 includes a communication unit 505a, an acquisition unit 505b, a storage unit 505c, and a conversion unit 505d. The image estimation device 505 executes a defocus blur conversion process of a captured image upon request from a control device 504 connected via a network 506. The image estimation device 505 acquires information on learned weights from a learning device 501 connected via a network 507 at the time of defocus blur conversion or in advance, and uses the information for defocus blur conversion of a captured image. The estimated image after defocus blur conversion is transmitted again to the control device 504, stored in the storage unit 504c, and displayed on the display unit 504b. Note that the generation of learning data and learning of weights (learning phase) performed by the learning device 501 are the same as those in the first embodiment, and therefore will not be described.

[0075] Next, the defocus blur conversion (estimation phase) of the captured image executed by the control device 504 and the image estimation device 505 will be described with reference to Fig. 20. Fig. 20 is a flowchart relating to the generation of an estimated image.

[0076] First, in step S301, the communication unit 504a of the control device 504 transmits a captured image and a processing request to the image estimation device 505. Then, in step S302, the communication unit 505a of the image estimation device 505 receives and acquires the captured image and the processing request from the control device 504. Then, in step S303, the acquisition unit 505b acquires learned weight information from the storage unit 505c. The weight information is read out in advance from the storage unit 501a and stored in the storage unit 505c. Then, in step S304, the conversion unit 505d inputs the input data to the CNN and generates an estimated image in which the defocus blur of the captured image is converted. Then, in step S305, the communication unit 505a transmits the estimated image to the control device 504. Then, in step S306, the communication unit 504a acquires the estimated image transmitted from the image estimation device 505 and stores it in the storage unit 504c.

[0077] With the above configuration, even if the change in the amount of parallax is small, it is possible to convert the blur caused by defocus and obtain an image with good blur.

[0078] In each embodiment, when the value related to the defocus amount is smaller than the threshold, the resolution of the correct image is equal to or higher than the resolution of the training image. On the other hand, when the value related to the defocus amount is larger than the threshold, the resolution of the correct image is lower than the resolution of the training image. Preferably, the threshold is not 0. Also preferably, the value related to the defocus amount is the parallax amount corresponding to the set defocus amount, or the absolute value of the defocus value obtained from a defocus map based on the set defocus amount.

[0079] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0080] According to each embodiment, it is possible to provide an image processing method, a program, an image processing device, and a method for manufacturing a trained model that can convert the blur caused by defocus in a captured image to obtain an image with good blur, even when the change in the amount of parallax is small.

[0081] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0082] The comparison of the value related to the defocus amount with a threshold value is not limited to a comparison of the parallax amount with a threshold value, or a comparison of the absolute value of the defocus value with a threshold value, but may be, for example, a comparison of the set defocus amount itself with a threshold value. [Explanation of symbols]

[0083] 101 Learning device (image processing device) 101b Acquisition unit (acquisition means) 101c Generation unit (generation means) 101d Update unit (learning means)

Claims

1. A method for learning a learned model that converts defocus blur of a captured image into blur of a different shape, comprising: acquiring an original image and a defocus amount; generating a training image and a correct answer image based on the original image and the defocus amount; A step of learning the machine learning model based on an output image generated by inputting the training image into the machine learning model and the correct answer image, A learning method characterized in that, in the generating process, if the defocus amount is smaller than a threshold, the resolution of the correct image is set to be equal to or greater than the resolution of the training image, and if the defocus amount is greater than the threshold, the resolution of the correct image is set to be less than the resolution of the training image.

2. A method for learning a trained model that converts defocus blur in a captured image into blur of a different shape, comprising: acquiring an original image and a defocus amount; generating a training image and a correct answer image based on the original image and the defocus amount; A step of learning the machine learning model based on an output image generated by inputting the training image into the machine learning model and the correct answer image; acquiring a parallax amount corresponding to the defocus amount, A learning method characterized in that, in the generating process, if the amount of disparity is smaller than a threshold, the resolution of the correct image is made equal to or greater than the resolution of the training image, and if the amount of disparity is greater than the threshold, the resolution of the correct image is made less than the resolution of the training image.

3. A method for learning a trained model that converts defocus blur in a captured image into blur of a different shape, comprising: acquiring an original image and a defocus amount; generating a training image and a correct answer image based on the original image and the defocus amount; A step of learning the machine learning model based on an output image generated by inputting the training image into the machine learning model and the correct answer image; acquiring an absolute value of a defocus value in a defocus map based on the defocus amount, A learning method characterized in that, in the generating process, if the absolute value is smaller than a threshold, the resolution of the correct image is made equal to or greater than the resolution of the training image, and if the absolute value is greater than the threshold, the resolution of the correct image is made less than the resolution of the training image.

4. 4. The method according to claim 1, wherein the threshold value is not zero.

5. 5. The learning method according to claim 1, wherein the threshold value is determined based on a focal depth of an image capturing device used to capture the captured image.

6. The learning method according to claim 1 , wherein the defocus amount is acquired based on a focal depth of an image capturing device used to capture the captured image.

7. 7. The learning method according to claim 1, wherein when the defocus amount is smaller than the threshold, the answer image has the same resolution as the training image.

8. 7. The learning method according to claim 5, wherein the focal depth is based on an F-number of an optical system used to capture the captured image and a pixel pitch of the captured image.

9. 9. An image processing method, comprising: converting defocus blur of a captured image by inputting the captured image to the machine learning model generated by the learning method according to claim 1.

10. A program for causing a computer to execute the image processing method according to claim 9.

11. An acquisition means for acquiring an original image and a defocus amount; A generating means for generating a training image and a correct answer image based on the original image and the defocus amount; A learning means for learning the machine learning model based on an output image generated by inputting the training image into the machine learning model and the correct answer image, A learning device characterized in that, in the generation means, when the defocus amount is smaller than a threshold value, the resolution of the correct image is made equal to or greater than the resolution of the training image, and when the defocus amount is greater than the threshold value, the resolution of the correct image is made less than the resolution of the training image.

12. acquiring an original image and a defocus amount; generating a training image and a correct answer image based on the original image and the defocus amount; A step of learning the machine learning model based on an output image generated by inputting the training image into the machine learning model and the correct answer image, A method for generating a trained machine learning model, characterized in that, in the generating process, if the defocus amount is smaller than a threshold, the resolution of the correct image is made equal to or greater than the resolution of the training image, and if the defocus amount is greater than the threshold, the resolution of the correct image is made less than the resolution of the training image.

Images