Learning method, image processing method, and program

Abstract

To provide a learning method which can acquire a correction image in which blur is excellently corrected on the basis of intermediate correction data in which blur due to optical characteristics is corrected.SOLUTION: A learning method of a machine learning model for outputting a correction image from intermediate correction data in which blur due to optical characteristics of an optical system is corrected comprises the steps of: acquiring a blur image by applying blur to an original image; acquiring the intermediate correction data obtained by executing sharpening processing based on the optical characteristics to the blur image as plural pieces of training data; acquiring a plurality of correct answer images on the basis of the original image in correspondence with each training data; and learning a first machine learning model by using the learning data consisting of the training data and correct answer image. The blur applied to the blur image is different for each training data with the optical characteristics as a reference.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a learning method and an image processing method for obtaining a corrected image from intermediate correction data in which blur caused by the optical characteristics of a photographed image is corrected.

Background Art

[0002] Patent Document 1 discloses a method of using a neural network to correct blur due to aberration and diffraction from a photographed image to obtain a high-resolution image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In image processing for sharpening an image in which blur such as aberration and diffraction caused by the optical characteristics of an imaging device has occurred, high-precision aberration correction can be achieved by correcting based on known optical characteristics. On the other hand, when the optical characteristics used for aberration correction are different from the blur characteristics in the actual photographed image, overcorrection or undercorrection may occur. In particular, in the case of overcorrection, adverse effects occur. Here, the adverse effects refer to structures that do not exist in the original subject that occur in the corrected image, such as overshoot, undershoot, ringing, etc. In the aberration correction process using deep learning, while correction can be achieved with higher precision than image processing using a conventional linear filter, overcorrection and undercorrection are also more likely to become more prominent. Since the method disclosed in Patent Document 1 performs aberration correction using shooting conditions, overcorrection or undercorrection may occur when the blur in the actual photographed image is different from the assumed performance.

[0005] Therefore, an object of the present invention is to provide a learning method, an image processing method, etc. that can obtain a corrected image with better blur correction based on intermediate correction data obtained by correcting blur caused by optical characteristics.

Means for Solving the Problems

[0006] A learning method as one aspect of the present invention is a learning method of a machine learning model for outputting a corrected image from intermediate correction data obtained by correcting blur caused by the optical characteristics of an optical system, including a step of applying blur to an original image to obtain a blurred image, a step of obtaining, as a plurality of training data, the intermediate correction data obtained by performing a sharpening process based on the optical characteristics on the blurred image, a step of obtaining a plurality of correct images corresponding to each of the training data based on the original image, and a step of learning a first machine learning model using learning data composed of the training data and the correct images, wherein the blur applied to the blurred image is different for each of the training data based on the optical characteristics.

[0007] Other objects and features of the present invention will be described in the following examples.

Effects of the Invention

[0008] According to the present invention, it is possible to provide a learning method, an image processing method, etc. that can obtain a corrected image with better blur correction based on intermediate correction data obtained by correcting blur caused by optical characteristics.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and duplicate descriptions are omitted.

[0011] First, before giving a specific description of the embodiments, the gist of the present invention will be described. The present invention relates to image processing for improving overcorrection and undercorrection that occur in a sharpening process for correcting blur caused by the optical characteristics of an imaging device. Here, the optical characteristics include aberration, diffraction, low-pass filter, pixel aperture effect, and the like. In particular, it relates to improving overcorrection and undercorrection that occur when the blur characteristics in an actual captured image are different from the assumed optical characteristics in a sharpening process based on known optical characteristics. Here, blur refers to a point spread function (PSF) or an optical transfer function (OTF).

[0012] In the sharpening process, the optical characteristics can be determined from the shooting conditions of the optical system (zoom, aperture, state of the focal length) during imaging and the screen position of the captured image. When design values are used for the optical characteristics, the optical characteristics may be better or worse than expected due to manufacturing errors of the imaging device. Also, the optical characteristics change when the subject distance deviates from the assumption. If it is assumed that the subject exists on the focal plane when determining the optical characteristics used for the sharpening process, in a scene with depth, a subject on a non-focal plane (defocused) blurs with different optical characteristics. On the other hand, even when considering the optical characteristics on the non-focal plane, the optical characteristics are different from the assumption due to the deviation of the subject distance. Even for a subject on a non-focal plane, depending on the relationship with the field curvature of the imaging optical system, the optical characteristics may be better or worse than the assumed optical characteristics. Thus, when the optical characteristics in the captured image are different from the optical characteristics assumed from the shooting conditions and the screen position, overcorrection or undercorrection occurs by performing the sharpening process based on the incorrect optical characteristics.

[0013] In the present invention, in order to correct overcorrection or undercorrection when the optical characteristics are different from the assumption, a multi-layer neural network (first machine learning model) is used. By training the first machine learning model using training data having the features described later, a corrected image with better blur correction can be obtained based on the intermediate correction data acquired by the sharpening process.

[0014] In the learning of weights (filters, biases, etc.) used in a multi-layer neural network, when the optical performance deviates from the assumption, the output (intermediate correction data) obtained by the sharpening process is used as training data. As the correct image, an image with appropriately corrected aberration is used. The training data is input into the neural network, and the weights are optimized so that the error between the output and the correct image becomes small. At this time, the neural network is trained to improve (correct) overcorrection and undercorrection. Due to manufacturing errors and deviations in the subject distance, the optical characteristics vary in various ways. Therefore, by generating training data assuming a plurality of different optical characteristics, it is possible to train a neural network that can improve overcorrection and undercorrection for all cases. Hereinafter, the case where the gloss characteristic deviates from the assumption is called performance deviation, and the correction process for improving overcorrection and undercorrection caused by the influence of performance deviation is called performance deviation correction.

Example

[0015] First, with reference to FIGS. 2 and 3, the image processing system in Example 1 of the present invention will be described. In this example, a multi-layer neural network is made to learn and execute correction of overcorrection and undercorrection. FIG. 2 is a block diagram of the image processing system 100 in this example. FIG. 3 is an external view of the image processing system 100.

[0016] The image processing system 100 includes a learning device (image processing device) 101, an imaging device 102, an image estimation device (image processing device) 103, a display device 104, a recording medium 105, an output device 106, and a network 107. The learning device 101 includes a storage unit (storage means) 101a, an acquisition unit (acquisition means) 101b, a generation unit (generation means) 101c, and an update unit (learning means) 101d.

[0017] The imaging device 102 includes an optical system (imaging optical system) 102a and an imaging element 102b. The optical system 102a condenses the light incident from the subject space to the imaging device 102. The imaging element 102b receives (photoelectrically converts) the optical image (subject image) formed through the optical system 102a to acquire an imaging image. The imaging element 102b is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor. The imaging image acquired by the imaging device 102 includes blurring due to aberrations and diffraction of the optical system 102a and noise due to the imaging element 102b.

[0018] The image estimation device 103 includes a storage unit 103a, an acquisition unit 103b, and a correction unit 103c. The image estimation device 103 acquires an imaging image, performs a sharpening process, and then performs performance deviation correction to generate an estimated image (corrected image) by a first machine learning model described later. The sharpening process uses a multi-layer neural network (second machine learning model), and the weight information is read from the storage unit 103a. The shooting conditions and the screen position (image height, azimuth) of the imaging image are input to the second machine learning model to acquire an intermediate corrected image (intermediate correction data) as an output. Here, the sharpening process is executed by the image estimation device 103, but an intermediate corrected image sharpened by a different image processing device may be acquired. A multi-layer neural network (first machine learning model) is used for the performance deviation correction, and the weight information is read from the storage unit 103a.

[0019] The weights (weight information) of the first machine learning model and the second machine learning model are those learned by the learning device 101. The image estimation device 103 reads the weight information from the storage unit 101a via the network 107 in advance and stores it in the storage unit 103a. The weight information to be stored may be the numerical values of the weights themselves or in an encoded form. The learning of weights and the sharpening process using the weights are well known, and details regarding the performance deviation correction process will be described later. The image estimation device 103 performs performance deviation correction on the intermediate corrected image to generate a corrected image. In this embodiment, since an image is used as the intermediate correction data, it is called an intermediate corrected image, but a feature map described later may also be used as the intermediate correction data.

[0020] The corrected image is output to at least one of the display device 104, the recording medium 105, and the output device 106. The display device 104 is, for example, a liquid crystal display or a projector. The user can perform an editing operation or the like while checking the image during the process via the display device 104. The recording medium 105 is, for example, a semiconductor memory, a hard disk, a server on a network, or the like. The output device 106 is a printer or the like. The image estimation device 103 has a function of performing development processing and other image processing as necessary.

[0021] Next, with reference to FIGS. 1 and 4, a learning method (manufacturing method of a learned model) of weights (weight information) executed by the learning device 101 in this embodiment will be described. FIG. 1 is a diagram showing the flow of learning the weights of a neural network (first machine learning model). FIG. 4 is a flowchart regarding the learning of weights. Each step in FIG. 4 is mainly executed by the acquisition unit 101b, the generation unit 101c, or the update unit 101d of the learning device 101.

[0022] A method for training a first machine learning model will be described. First, in step S101 of FIG. 4, the acquisition unit 101b acquires an original image (subject image). In this embodiment, the original image is a high-resolution (high-quality) image with little blur due to aberration and diffraction of the optical system 102a. A plurality of original images are acquired, which are images having various subjects, that is, edges with various strengths and directions, textures, gradations, flat portions, and the like. The original image may be a real-shot image or an image generated by CG (Computer Graphics). In particular, when a real-shot image is used as the original image, since blur has already occurred due to aberration and diffraction, the influence of the blur can be reduced by reducing the image, and a high-resolution (high-quality) image can be obtained. If the original image sufficiently contains high-frequency components, reduction may not be performed.

[0023] Preferably, the original image has a signal value higher than the luminance saturation value of the imaging device 102b. This is because, in an actual subject, when imaging is performed by the imaging device 102 under specific exposure conditions, there are subjects that do not fit within the luminance saturation value. When a real-shot image is used as the original image, it can be obtained by performing HDR imaging or imaging with an imaging device having a higher dynamic range than the imaging device 102. When an image captured by an imaging device having the same dynamic range as the imaging device 102 is used as the original image, it is also possible to obtain a high signal value by proportionally multiplying the signal value. However, it is preferably performed within a range where the reduction in gradation due to proportional multiplication does not affect the learning result. Also, the original image may have noise components. In this case, since the noise included in the original image can be regarded as a subject, the noise in the original image does not particularly pose a problem.

[0024] Subsequently, in step S102, the acquisition unit 101b acquires the blur used for performing the imaging simulation described later. First, the acquisition unit 101b acquires the shooting conditions corresponding to the lens state (zoom, aperture, focal length state) of the optical system 102a. Then, the acquisition unit 101b acquires the blur determined by the shooting conditions and the screen position of the captured image. Here, the blur is the PSF (point spread function) or OTF (optical transfer function) of the optical system 102a. The blur can be acquired by optical simulation or measurement in the optical system 102a. Note that the blur due to the lens state, image height, azimuth aberration, and diffraction that differ for each original image is acquired. Thereby, the imaging simulation corresponding to a plurality of shooting conditions, image heights, and azimuths can be performed. Further, if necessary, the components of a filter such as an optical low-pass filter included in the imaging device 102 may be added to the applied blur (the filter may be applied to the optical characteristics).

[0025] Subsequently, in step S103, the generation unit 101c generates a correct patch (correct image) and a training patch (training data). A plurality of correct patches and training patches are generated respectively, and one or more patches are generated corresponding to one original image. In the present embodiment, the correct patch and the training patch are images in which the same subject is captured. In the present embodiment, a combination of a plurality of correct patches and training patches is used as learning data. The learning data is used to train a machine learning model that corrects the influence of the deviation of the optical characteristics (performance deviation) that occurs when the aberration and diffraction of the optical system 102a are sharpened based on the optical characteristics. Therefore, the correct patch is an image with less adverse effects due to overcorrection or improved undercorrection compared to the training patch. However, as will be described later, depending on the conditions, the machine learning model may not correct in some cases, so the learning data may include cases where the correct patch and the training patch are the same image.

[0026] Note that a patch refers to an image having a predetermined number of pixels (e.g., 64×64 pixels, etc.). Also, the number of pixels of the correct patch and the training patch do not necessarily have to match. In this embodiment, mini-batch learning is used for learning the weights of the multi-layer neural network. Therefore, in step S103, a plurality of sets of correct patches and training patches are generated. However, this embodiment is not limited thereto, and online learning or batch learning may be used.

[0027] In this embodiment, the correct patch and the training patch are obtained by the following method, but it is not limited thereto. In this embodiment, a plurality of blurred images with the effects of blurring due to aberration and diffraction are generated by performing imaging simulation using a plurality of original images stored in the storage unit 101a as subjects. Then, an intermediate corrected image sharpened using a second machine learning model based on the optical characteristics is generated for each blurred image. In this embodiment, the intermediate corrected image is used as training data. Also, for each training data, a correct image is obtained based on the original image. At this time, in terms of points, the correct image and the training data have the number of pixels that is the same as or larger than the number of pixels of the patch used as learning data. Then, a plurality of correct patches and training patches are obtained by extracting partial regions of a specified pixel size at the same position from a plurality of pairs of the correct image and the training data. In this embodiment, the original image is an undeveloped RAW image, and the correct patch and the training patch are also RAW images in the same way. However, this embodiment is not limited thereto, and it may be a developed image, or a feature map obtained by converting an image as described later. Also, the position of the partial region refers to the center of the partial region.

[0028] The training data used for training the first machine learning model is generated by the following method. The generation unit 101c performs the following processing for a predetermined condition (such as shooting conditions and screen position). First, the generation unit 101c generates blurring with a smaller amount of blurring based on the blurring due to aberration and diffraction of the optical system 102a, that is, based on the blurring obtained in step S102. Also, the generation unit 101c applies (convolves) the generated blurring to the original image. Thereafter, the generation unit 101c obtains a blurred image by extracting a partial region from the image clipped by the luminance saturation value of the imaging device 102b.

[0029] Next, the generation unit 101c corrects the blurred image using the second machine learning model based on the blurring (optical characteristics) obtained in step S102 under a predetermined condition, and obtains an intermediate corrected image. By extracting the same partial region from the intermediate corrected image and the original image, a set of a training patch and a correct patch can be generated. The generation unit 101c generates a plurality of sets of training patches and correct patches for each size of blurring based on blurred images to which blurrings of different sizes are applied using a plurality of amounts of blurring of the generated blurring. That is, the generation unit 101c generates a set of a training patch and a correct patch using blurred images to which blurrings of different sizes are applied based on the blurring obtained in step S102. In this embodiment, all of the plurality of blurring sizes are smaller than the blurring obtained in step S102, that is, the optical characteristics under a predetermined condition, but the present invention is not limited to this. By performing the above-described processing while changing the predetermined condition, blurred images corresponding to a plurality of performance deviations are generated for a plurality of conditions (such as shooting conditions and screen position) of the imaging device, and training data having training patches and correct patches based on the blurred images can be generated.

[0030] Here, the defocus deformation will be described. In this embodiment, the defocus applied to the defocused image is obtained by reducing the spatial spread of the PSF acquired in step S102. The reduction ratio of the PSF is randomly set between 0.1 times and 0.9 times, and the PSF is reduced at the set reduction ratio. The reduction ratio may be different values in the vertical direction and the horizontal direction. The reduction process is performed using interpolation processes such as downsampling and bicubic interpolation. As a result, based on the defocus (optical characteristics) acquired in step S102, the defocus applied to the defocused image is small. By performing a sharpening process on the defocused image thus generated using the second machine learning model, the intermediate corrected image becomes overcorrected and adverse effects occur.

[0031] Since the neural network is trained to bring the training patch closer to the correct patch, it is trained to bring the intermediate corrected image closer to the original image. That is, the first machine learning model of this embodiment suppresses overcorrection to an appropriate correction effect and also suppresses adverse effects (reduces the influence of overcorrection). Here, the appropriate correction effect is a correction effect that brings the intermediate corrected image closer to the original image or the intermediate corrected image when there is no performance deviation. Even without adverse effects such as ringing, if it is sharpened too much, it may not become a natural image. Therefore, in order to match the correction effect with the case where there is no performance deviation or other image regions where there is no performance deviation, it is advisable to suppress it to an appropriate correction effect.

[0032] Note that the blur applied to the blurred image is not limited to the aforementioned blur. A blur filter such as Gaussian blur may also be used. Instead of downsampling the PSF obtained in step S102, it may be reduced by convolution with a sharpening filter, existing reduction processing, or the like. When blur is obtained as the OTF in step S102, the blur can be reduced by performing an enlargement process such as upsampling, a process of improving the OTF by applying a frequency gain, or a dilation process. The magnitude of the blur amount in this embodiment can be represented by a predetermined index. For example, when the maximum value of the PSF is large, the half-value width in the main cross-section of the PSF is small, the MTF of a specified spatial frequency is large, the integral value of the MTF with respect to the spatial frequency is large, etc., it can be said that the blur amount is small. Further, the blur amount can be increased by convolving the PSF with a blur filter (low-pass filter) or applying the frequency characteristics of the low-pass filter to the OTF. The magnitude of the blur relative to a predetermined blur (the magnitude of the blur relative to the predetermined blur) may be represented by the difference or ratio of the aforementioned indices, or the aforementioned indices may be used for the difference or ratio of the OTF.

[0033] Learning of the sharpening process based on optical characteristics (second machine learning model) can be performed by using the blurred image to which the blur obtained in step S102 is applied as a training patch and the original image as the correct patch. When the real-world image is reduced to obtain the original image, the order of reduction and application of blur may be reversed. When applying blur first, it is necessary to make the sampling rate of the blur finer in consideration of the reduction. If it is the PSF (point spread intensity distribution), the spatial sampling points should be made finer, and if it is the OTF (optical transfer function), the maximum frequency should be increased.

[0034] It is preferable that the added blur does not include distortion aberration. In the sharpening process (aberration correction process), if the distortion aberration is large, the position of the subject changes, and the subject may be different between the correct patch and the training patch. For this reason, the sharpening process (second machine learning model) used in this embodiment does not correct the distortion aberration. Accordingly, the performance deviation correction by the first machine learning model also does not correct the distortion aberration. The distortion aberration is corrected individually after the blur correction using bilinear interpolation, bicubic interpolation, or the like.

[0035] In actual imaging, since noise is generated in the imaging element 102b, it is preferable to add noise to the training data as well. Random numbers corresponding to the noise characteristics of the imaging element 102b may be generated and added, and imaging conditions such as ISO sensitivity may be considered. When noise is added only to the training image or when noise having no correlation between the training image and the correct image is added, denoising is also learned simultaneously with the blur correction during learning. When the same noise is added to the training image and the correct image, blur correction with suppressed noise change is learned. When using the feature map described later as training data, noise is added to the blurred image instead of the training image.

[0036] Subsequently, in step S104, the generation unit 101c inputs the training patch (training data) 212 in FIG. 1 to a multi-layer neural network and generates an estimated patch (estimated image) 213. For mini-batch learning, estimated patches 213 corresponding to a plurality of training patches 212 are generated. FIG. 1 shows the flow from step S104 to step S105.

[0037] The estimated patch 213 is a training patch 212 with corrected performance deviation, and ideally coincides with the correct patch (correct image) 211. In this embodiment, the configuration of the neural network shown in FIG. 1 is used, but the present invention is not limited to this. In FIG. 1, CN represents a convolutional layer and DC represents a deconvolutional layer. For both CN and DC, the convolution of the input and the filter, and the sum with the bias are calculated, and the result is non-linearly transformed by the activation function. The initial values of each component of the filter and the bias are arbitrary and are determined by random numbers in this embodiment. As the activation function, for example, ReLU (Rectified Linear Unit) or sigmoid function can be used. The output of each layer except the final layer is called a feature map. The skip connections 222 and 223 synthesize the feature maps output from non-consecutive layers. The synthesis of the feature maps may be by taking the sum for each element or by concatenation in the channel direction. In this embodiment, the sum for each element is adopted. The skip connection 221 takes the sum of the residual estimated from the training patch 212 and the correct patch 211 and the training patch 212 to generate the estimated patch 213. The estimated patch 213 is generated for each of the plurality of training patches 212.

[0038] Subsequently, in step S105, the update unit 101d updates the weights (weight information) of the neural network from the error between the estimated patch 213 and the correct patch (correct image) 211. Here, the weights include the components of the filters and the biases of each layer. The error backpropagation method is used for updating the weights, but this embodiment is not limited to this. For mini-batch learning, the errors between a plurality of correct patches 211 and the corresponding estimated patches 213 are obtained, and the weights are updated. As the error function (Loss function), for example, L2 norm or L1 norm may be used.

[0039] Subsequently, in step S106, the update unit 101d determines whether the learning of the weights is completed. Completion can be determined by whether the number of iterations of learning (weight update) has reached a specified value, or whether the amount of change in the weights during update is smaller than the specified value, etc. If it is determined that the learning is not completed, the process returns to step S104 to acquire a plurality of new correct patches and training patches. On the other hand, if it is determined that the learning is completed, the learning device 101 (update unit 101d) ends the learning and stores the weight information in the storage unit 101a.

[0040] Next, with reference to FIG. 5, the generation of the corrected image (correction process, estimation method) executed by the image estimation device 103 in this embodiment will be described. FIG. 5 is a flowchart regarding the generation of the corrected image. Each step in FIG. 5 is mainly executed by the acquisition unit 103b or the correction unit 103c of the image estimation device 103.

[0041] First, in step S201, the acquisition unit 103b acquires the captured image and the weight information. The captured image is an undeveloped RAW image similar to the learning, and in this embodiment, it is transmitted from the imaging device 102. The weight information is the weights of the first machine learning model and the second machine learning model transmitted from the learning device 101 and stored in the storage unit 103a.

[0042] Subsequently, in step S202, the acquisition unit 103b performs a sharpening process based on the weights of the second machine learning model acquired in step S201 and acquires an intermediate corrected image.

[0043] Subsequently, in step S203, the correction unit 103c inputs the intermediate corrected image acquired in step S202 as the input image of the acquired first machine learning model and generates a corrected image. The corrected image is an image obtained by performing performance deviation correction on the intermediate corrected image. That is, the portion over-corrected more than expected due to over-correction is suppressed to an appropriate correction amount, and the adverse effects are also suppressed (the influence due to over-correction is reduced). Note that when inputting the captured image into the neural network, it is not necessary to cut it out to the same size as the training patch used during learning.

[0044] In this embodiment, the sharpening process is executed by a machine learning model. However, the sharpening process may also be other existing methods based on optical characteristics. For example, it may also be a method of convolving a sharpening filter such as a Wiener filter. By using the intermediate corrected image obtained by performing other sharpening methods on the blurred image as training data, the first machine learning model can be similarly learned. On the other hand, by executing the sharpening process with a machine learning model using a neural network such as a CNN, the deterioration of image quality due to optical characteristics can be corrected with higher precision, which is preferable. When using a neural network, due to the high correction effect, if there is a deviation from the assumed optical characteristics, the adverse effects may become prominent. However, by executing the process by the first machine learning model of this embodiment, as a result, a corrected image with a high correction effect and suppressed adverse effects can be obtained.

[0045] Also, for the sharpening process, it is preferable to use a machine learning model (a second machine learning model common to multiple optical characteristics) that is commonly learned for multiple optical characteristics. When using other sharpening methods or when the machine learning model for the sharpening process is learned for each optical characteristic, differences in correction effects are likely to occur for each optical characteristic, and differences are also likely to occur in overcorrection and the resulting adverse effects. On the other hand, by using a commonly learned machine learning model, the correction effect and the occurrence of adverse effects are common, and the first machine learning model can be learned accurately. As a result, performance deviations can be corrected better.

[0046] Also, for the machine learning model of the sharpening process, it is preferable to add information on at least one of the shooting conditions and the screen position to the input data (input and use information on the shooting conditions and the screen position) in addition to the captured image. In this case, it is necessary to input this information both during learning and during the correction process. Thereby, the blur due to aberration, diffraction, etc. can be corrected with higher precision, and by executing the process by the first machine learning model of the present invention together, as a result, a corrected image with a high correction effect and suppressed adverse effects can be obtained.

[0047] In addition, for the first machine learning model, it is preferable to add not only the intermediate correction data but also information on shooting conditions and screen position to the input data. In this case, it is necessary to input information on shooting conditions both during learning and during the correction process. For example, in a low-performance area with high image height and field curvature, overcorrection due to a three-dimensional subject or partial blurring is likely to occur. Depending on the shooting conditions and optical characteristics, the likelihood and manner of overcorrection and adverse effects vary. By inputting the shooting conditions into the first machine learning model, performance deviation can be corrected based on the shooting conditions and optical characteristics, and a good corrected image can be obtained. In particular, when information on shooting conditions is input into the machine learning model for sharpness enhancement processing, learning can be performed using the shooting condition information common to both the sharpness enhancement processing and the performance deviation correction, and a better corrected image can be obtained.

[0048] In addition, in the first machine learning model, it is preferable to add manufacturing error information of the imaging device as input data. By adding the manufacturing error information obtained by measurement after manufacturing, etc., performance deviation can be corrected based on the influence of manufacturing errors on optical characteristics. For example, if it is known that the imaging optical system has partial blurring, the screen positions likely to result in overcorrection and the screen positions likely to result in undercorrection can be determined.

[0049] In this embodiment, the intermediate correction data and the training data are used as intermediate correction images. However, when performing sharpness enhancement processing using a machine learning model, the feature maps of the intermediate layer may be used as the intermediate correction data and the training data. When the intermediate correction data is an image, it has the advantage that adverse effects on the image can be easily confirmed during learning, and a machine learning model using the difference from the captured image can also be used. On the other hand, when the number of channels of the feature map is large, the amount of information is reduced by converting it into an image.

[0050] Therefore, by using the feature map as the intermediate correction data and training data input to the first machine learning model, it is possible to correct the performance deviation without reducing the amount of information. Also in this case, the intermediate correction image may be estimated during the learning of the second machine learning model. For example, it is possible to perform learning using the error between the intermediate correction image estimated from the captured image and the original image. During the correction process using the captured image, the intermediate correction data is calculated using the weights up to the intermediate layer of the second machine learning model, and by inputting it to the first machine learning model, a sharpened correction image with the performance deviation also corrected can be obtained. Therefore, the weight information of the second machine learning model acquired in step S201 only needs to include the weights of the layers before the intermediate layer that serves as the intermediate correction data. Note that the number of pixels and the number of channels of the training data may be different from those of the correct image. Also, as the training data, a blurred image may be used, and during learning, the machine learning model for sharpening processing and the machine learning model for performance deviation correction may be connected and learned. In this case, the machine learning model for sharpened processing may be learned so as to update only the weights of the layers corresponding to the model that corrects the performance deviation without updating the weights that have been learned in advance.

[0051] Also, when performing the sharpened processing with a machine learning model, it is not necessary to output the intermediate correction data or the intermediate correction image once during the correction process from the captured image. That is, it may be executed together as one model that performs the sharpened processing and the performance deviation correction.

[0052] Also, the original image is used as the correct image when learning the first machine learning model, and the correct patch is extracted from the original image, but this embodiment is not limited to this. For example, for a blurred image given the same blur as the optical characteristics, an image obtained by performing sharpened processing based on the optical characteristics may be used as the correct image. In addition, any image that is not affected by the performance deviation and has little blur due to the optical characteristics can be used as the correct image.

[0053] Also, the correct image may be the original image only when there are adverse effects such as undershoot or ringing in the intermediate correction data and training data when training the first machine learning model. That is, even if there is overcorrection, if there are no unnatural-looking adverse effects such as undershoot or ringing, the correct image may be the intermediate correction data and the training data themselves. In this case, the first machine learning model is trained to suppress only unnatural-looking adverse effects, so that an output image with sharpness can be obtained although there is overcorrection. Also, if one tries to suppress the correction effect of the intermediate correction data or training data that is overcorrected but has no unnatural-looking adverse effects, it will be necessary to learn a suppression effect that is difficult to directly judge from the image. By avoiding this, learning becomes easier. Note that it is not necessarily required to distinguish whether adverse effects actually occur. For example, for high-contrast images or images including edge portions where adverse effects are likely to occur, the correct image may be the original image.

[0054] Also, in this embodiment, the blurry image is made to have a blur smaller than the optical characteristics to correct the performance deviation resulting in overcorrection, but this embodiment is not limited to this. By generating training data with a blur given to the blurry image that is larger than the optical characteristics, a first machine learning model that corrects the performance deviation due to undercorrection can be learned. As the correct image in this case, similar to the case of correcting the performance deviation due to overcorrection, the original image may be used, or an image obtained by performing a sharpening process based on the optical characteristics on the blurry image given the same blur as the optical characteristics may be used.

[0055] In addition, training data may be generated such that the blurring applied to the blurred image includes both blurring smaller than the optical characteristics and blurring larger than the optical characteristics. As the correct image, the original image may be used, or an image obtained by performing a sharpening process based on the optical characteristics on the blurred image to which the same blurring as the optical characteristics is applied may be used. Thereby, a first machine learning model that corrects both performance deviation due to overcorrection and performance deviation due to undercorrection can be learned. If only the correction of overcorrection is learned, the correction effect of the image region with undercorrection may be further suppressed. However, by learning both, the first machine learning model can learn both cases of overcorrection and undercorrection, so both overcorrection and undercorrection can be corrected.

[0056] In addition, when training data is generated such that the blurring applied to the blurred image includes both blurring smaller than the optical characteristics and blurring larger than the optical characteristics, only one of the performance deviations of overcorrection and undercorrection may be corrected. By changing the correct image in the case of not correcting to an image obtained by performing a sharpening process on the corresponding blurred image, a machine learning model that performs such correction can be learned. The correction of overcorrection is a process that reduces the frequency characteristics, and the correction of undercorrection is a process that improves the frequency characteristics. Since both have opposite functions, by limiting to the correction of only one of the performance deviations, a lighter model can be obtained, or the correction effect of the performance deviation can be improved. For example, as the correct image corresponding to the training data in which the blurring applied to the blurred image is smaller than the optical characteristics, an image obtained by applying blurring based on the optical characteristics to the original image and then performing a sharpening process based on the optical characteristics or the original image is used. Also, as the correct image corresponding to the training data in which the blurring applied to the blurred image is larger than the optical characteristics, the blurred image corresponding to the training data or the intermediate correction data (intermediate correction image) corresponding to the training data is used. Thereby, the adverse effects of overcorrection are removed (the influence of overcorrection is reduced), but the undercorrection due to performance deviation cannot be corrected. Similar to the above case, by learning including both, the first machine learning model can learn to distinguish between the case of not correcting and the case of correcting.

[0057] In addition, training data may be generated including the case where the blur applied to the blurred image is the same (substantially the same) as the optical characteristics. As the correct image, the original image may be used, or an image obtained by performing sharpness enhancement processing based on the optical characteristics on the blurred image to which the same blur as the optical characteristics is applied may be used. That is, as the correct image corresponding to the training data in which the blur applied to the blurred image is the same as the optical characteristics, the original image or the intermediate correction data (intermediate correction image) corresponding to the training data can be used. Thereby, the first machine learning model can learn both correction in the case of performance deviation and non-correction in the case of no performance deviation.

[0058] In addition, when performing correction using the first machine learning model, not only the intermediate correction data but also the captured image may be added to the input data. During learning, the blurred image may be added. In this case, by knowing the image before the sharpness enhancement processing, performance deviation correction can be performed with higher accuracy. For example, when adverse effects such as undershoot or ringing occur due to overcorrection, the captured image can be used as a judgment material for whether it is a real structure or a structure generated by the sharpness enhancement processing. Also, by using the difference from the captured image, it is possible to correct the performance deviation by positively using the change due to the sharpness enhancement processing.

[0059] In addition, the blur applied to the original image when acquiring the training data may be based on at least one of the shooting conditions of the optical system and the screen position. For example, when the sensitivity of the optical characteristics is high with respect to the manufacturing tolerance or the subject distance, the performance deviation from the optical characteristics can be large. Therefore, by greatly changing the blur applied to the blurred image in such a case, a first machine learning model corresponding to a larger performance deviation can be learned. To greatly change the blur, for example, the upper limit value of the magnification rate or the lower limit value of the reduction rate may be set small among a plurality of magnification rates and reduction rates used when generating the applied blur from the PSF.

[0060] In this embodiment, the case where the learning device 101 and the image estimation device 103 are separate entities has been described as an example. However, this embodiment is not limited to this. The learning device 101 and the image estimation device 103 may be integrally configured. That is, learning (the process shown in FIG. 4) and estimation (the process shown in FIG. 5) may be performed within an integrated device.

[0061] Also, the image estimation device that performs the sharpening process and the performance deviation correction process may be separate, and instead of acquiring the captured image, an image after the sharpening process may be acquired. In this case, the execution of the sharpening process is skipped.

[0062] As described above, the image processing device (learning device 101) of this embodiment executes learning of a machine learning model for outputting a corrected image from an intermediate corrected image in which blur caused by the optical characteristics of the optical system is corrected. The image processing device has a generation unit 101c having functions as a first acquisition means, a second acquisition means, and a third acquisition means, and an update unit 101d as a learning means. The first acquisition means acquires a blurred image by imparting blur to the original image. The second acquisition means acquires intermediate correction data obtained by performing a sharpening process based on optical characteristics on the blurred image as a plurality of training data. The third acquisition means acquires a plurality of correct images corresponding to each training data based on the original image. The learning means learns the first machine learning model using learning data composed of the training data and the correct images. Also, the blur imparted to the blurred image differs for each training data based on the optical characteristics.

[0063] According to this embodiment, it is possible to provide an image processing method or the like capable of acquiring a corrected image in which blur is better corrected based on intermediate correction data in which blur caused by optical characteristics is corrected.

Embodiment

[0064] Next, with reference to FIGS. 6 and 7, the image processing system in Embodiment 2 of the present invention will be described. In this embodiment, generation of the corrected image is executed by an image estimation unit 323 in the imaging device. Also, in this embodiment, the generation flow of the corrected image is different from that in Embodiment 1.

[0065] FIG. 6 is a block diagram of the image processing system 300 in this embodiment. FIG. 7 is an external view of the image processing system 300. The image processing system 300 includes a learning device (image processing device) 301 and an imaging device 302 connected via a network 303. The learning device 301 has a storage unit (storage means) 311, an acquisition unit (acquisition means) 312, a generation unit (generation means) 313, and an update unit (learning means) 314, and learns weights (weight information) for performing blur correction with a neural network. The imaging device 302 captures an object space to obtain a captured image, sharpens the captured image using the read weight information, corrects performance deviation, and generates a corrected image. The imaging device 302 has an optical system 321 and an image sensor 322. The image estimation unit 323 has an acquisition unit 323a and a correction unit 323b, and corrects the performance deviation of the captured image using the weight information stored in the storage unit 324.

[0066] The weight information is pre-learned by the learning device 301 and stored in the storage unit 311. The imaging device 302 reads the weight information from the storage unit 311 via the network 303 and stores it in the storage unit 324. The captured image (corrected image) with corrected performance deviation is stored in the recording medium 325. When an instruction regarding display of the corrected image is issued by the user, the stored corrected image is read out and displayed on the display unit 326. Note that a captured image already stored in the recording medium 325 may be read out and performance deviation correction may be performed by the image estimation unit 323. The above series of controls is performed by the system controller 327. Note that the learning of the first model executed by the learning device 301 in this embodiment is the same as that in Embodiment 1.

[0067] Next, with reference to FIG. 8, the generation of the corrected image executed by the image estimation unit 323 in this embodiment will be described. FIG. 8 is a flowchart regarding the generation of the corrected image. Each step in FIG. 8 is mainly executed by the acquisition unit 323a or the correction unit 323b of the image estimation unit 323. Steps S401 to S403 are the same as steps S201 to S203 executed by the acquisition unit 103b or the correction unit 103c in the first embodiment, respectively.

[0068] Subsequently, in step S404, the correction unit 323b performs a weighted average on the corrected image subjected to performance deviation correction and the sharpened intermediate corrected image (intermediate correction data) to generate a composite image. Thereby, the strength of the effect of performance deviation correction can be adjusted. The ratio of the weighted average may vary depending on the region of the image. In this embodiment, when the amount of luminance change due to the sharpening process, that is, the difference between the captured image and the intermediate corrected image, is smaller than a predetermined value, the ratio of the corrected image is made smaller. That is, the effect of performance deviation correction is reduced. This is because when performing performance deviation correction using the first machine learning model, the determination accuracy of the presence or absence of performance deviation may deteriorate when the amount of luminance change is small. In the texture part where the subject has a fine structure, whether to perform performance deviation correction or not varies with slight changes in luminance or structure, and it is likely to be corrected unevenly, which is not preferable. Since the influence of performance deviation is originally small in the region where the amount of luminance change due to the sharpening process is small, by reducing the change due to performance deviation correction in this region, it is possible to avoid being corrected unevenly.

[0069] In this embodiment, weighted averaging is performed between the corrected image and the intermediate corrected image, but the captured image may be used instead of the intermediate corrected image. For example, by increasing the ratio of the captured image in all or part of the image area, the strength of sharpness can be adjusted. The ratio of weighted averaging may be changed according to the user's specification, thereby realizing a sharpness effect desirable for the user. Also, the composite image is not limited to the case of using two images. By performing weighted averaging using the captured image, the intermediate corrected image, and the corrected image, the strength of sharpness and performance deviation correction may be adjusted. As for the composite method, existing methods other than weighted averaging may also be used.

[0070] As described above, in this embodiment, at least one of the captured image and the intermediate corrected data and the corrected image are synthesized based on the sharpness component (for example, the amount of luminance change by the sharpness process) obtained based on the captured image and the intermediate corrected data. Thus, in this embodiment, unlike the first embodiment, by performing the synthesis process between a plurality of images, the strength of sharpness and performance deviation correction can be adjusted. According to this embodiment, it is possible to provide an image processing method or the like capable of obtaining a corrected image in which blurring is better corrected based on the intermediate corrected data in which blurring caused by optical characteristics is corrected.

Embodiment

[0071] Next, with reference to FIG. 9, the image processing system in Embodiment 3 of the present invention will be described. The image processing system of this embodiment is different from those of Embodiments 1 and 2 in that it has a processing device (computer) that transmits a captured image that is the object of image processing to the image estimation device and receives the processed output image (corrected image) from the image estimation device.

[0072] FIG. 9 is a block diagram of the image processing system 600 in this embodiment. The image processing system 600 includes a learning device 601, an imaging device 602, an image estimation device 603, and a processing device (computer) 604. The learning device 601 and the image estimation device 603 are, for example, servers. The computer 604 is, for example, a user terminal (personal computer or smartphone). The processing device 604 is connected to the image estimation device 603 via a network 605. The image estimation device 603 is connected to the learning device 601 via a network 606. That is, the computer 604 and the image estimation device 603 are configured to be communicable, and the image estimation device 603 and the learning device 601 are configured to be communicable. Since the configuration of the learning device 601 is the same as that of the learning device 101 in the first embodiment, the description thereof is omitted. Also, since the configuration of the imaging device 602 is the same as that of the imaging device 102 in the first embodiment, the description thereof is omitted.

[0073] The image estimation device 603 includes a storage unit 603a, an acquisition unit 603b, a correction unit 603c, and a communication unit (reception means) 603d. The storage unit 603a, the acquisition unit 603b, and the correction unit 603c are respectively the same as the storage unit 103a, the acquisition unit 103b, and the correction unit 103c of the image estimation device 103 in the first embodiment. The communication unit 603d has a function of receiving a request transmitted from the computer 604 and a function of transmitting an output image generated by the image estimation device 603 to the computer 604.

[0074] The computer 604 includes a communication unit (transmission means) 604a, a display unit 604b, an image processing unit 604c, and a recording unit 604d. The communication unit 604a has a function of transmitting a request for causing the image estimation device 603 to execute processing on the captured image to the image estimation device 603, and a function of receiving the output image processed by the image estimation device 603. The display unit 604b has a function of displaying various information. The information displayed by the display unit 604b includes, for example, the captured image to be transmitted to the image estimation device 603 and the output image received from the image estimation device 603. The image processing unit 604c has a function of further performing image processing on the output image received from the image estimation device 603. The recording unit 604d records the captured image acquired from the imaging device 602, the output image received from the image estimation device 603, and the like.

[0075] Next, with reference to FIG. 10, image processing (generation of a corrected image) by the image processing system 600 will be described. FIG. 10 is a flowchart regarding the generation of the corrected image in the present embodiment. Note that the content of the image processing (correction processing) in the present embodiment is the same as the correction processing described in Embodiment 1 with reference to FIG. 5. The image processing shown in FIG. 10 is started when an instruction to start image processing is given by the user via the computer 604.

[0076] First, the operation in the computer 604 will be described. In step S701, the computer 604 transmits a request for processing the captured image to the image estimation device 603. Note that the method of transmitting the captured image to be processed to the image estimation device 603 is not limited. For example, the captured image may be uploaded to the image estimation device 603 simultaneously with step S701, or may have been uploaded to the image estimation device 603 before step S701. Further, the captured image may be an image stored on a server different from the image estimation device 603. In step S701, the computer 604 may transmit ID information for authenticating the user together with the request for processing the captured image.

[0077] Subsequently, in step S702, the computer 604 receives the output image generated within the image estimation device 603. The output image is an image in which the influence of performance deviation is corrected after the captured image is sharpened, similar to that in the first embodiment.

[0078] Next, the operation of the image estimation device 603 will be described. In step S801, the image estimation device 603 receives a request for processing the captured image transmitted from the computer 604. At this time, the image estimation device 603 determines that the processing (sharpening process and performance deviation correction process) for the captured image is instructed, and executes the processes after step S802.

[0079] In step S802, the image estimation device 603 acquires weight information. The weight information is information (trained model) learned by the same method as the method described in the first embodiment with reference to FIG. 4. The image estimation device 603 may acquire the weight information from the learning device 601, or may acquire the weight information that has been previously acquired from the learning device 601 and stored in the storage unit 603a. The subsequent steps S803 and S804 are the same as S202 and S203 in the first embodiment, respectively. Subsequently, in step S805, the image estimation device 603 transmits the output image to the computer 604. Although this embodiment has been described as performing the correction process of the first embodiment, the correction process of the second embodiment may also be performed.

[0080] When the performance deviation correction process is performed within the image estimation device 603 as in this embodiment, since the processing load due to the correction process can be borne within the image estimation device 603, it is possible to reduce the processing ability required on the computer 604 side. Also, as in this embodiment, the image estimation device 603 may be configured to be controlled using a computer 604 communicably connected to the image estimation device 603.

[0081] (Other Embodiments) The present invention can also be realized by supplying a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. It can also be realized by a circuit (for example, an ASIC) that implements one or more functions.

[0082] According to each embodiment, it is possible to provide a learning method, an image processing method, a learning apparatus, and a program capable of acquiring a corrected image in which blurring is better corrected based on intermediate correction data in which blurring caused by optical characteristics is corrected.

[0083] As described above, the preferred embodiments of the present invention have been described. However, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof.

Description of Reference Numerals

[0084] 101 Learning apparatus (image processing apparatus) 101c Generation unit (first acquisition means, second acquisition means, third acquisition means) 101d Update unit (learning means)

Claims

Claim 1 A method for training a machine learning model for outputting a corrected image from intermediate correction data obtained by correcting blur caused by the optical characteristics of an optical system, the method comprising: obtaining a blurred image by applying blur to an original image; obtaining, as a plurality of training data, the intermediate correction data obtained by performing a sharpening process based on the optical characteristics on the blurred image; obtaining a plurality of correct images corresponding to each of the training data based on the original image; training a first machine learning model using the training data consisting of the training data and the correct images; and wherein the blur applied to the blurred image is different for each of the training data based on the optical characteristics. Claim 2 The training method according to claim 1, wherein the intermediate correction data is at least one of an intermediate correction image or a feature map. Claim 3 The training method according to claim 1 or 2, wherein the sharpening process is performed by a second machine learning model common to the plurality of optical characteristics. Claim 4 The training method according to any one of claims 1 to 3, wherein the sharpening process is performed by inputting the shooting conditions of the optical system. Claim 5 The training method according to any one of claims 1 to 4, wherein the blur applied to the blurred image is generated based on the optical characteristics. Claim 6 The training method according to any one of claims 1 to 5, wherein the blur applied to the blurred image is generated by enlarging or reducing the optical characteristics. Claim 7 The training method according to any one of claims 1 to 6, wherein the blur applied to the blurred image is generated by applying a filter to the optical characteristics. Claim 8 The training method according to any one of claims 1 to 7, wherein the blur applied to the blurred image includes a blur with a small amount of blur based on the optical characteristics. Claim 9 As the correct image corresponding to the training data in which the blur applied to the blurred image is smaller than the optical characteristics, an image obtained by performing the sharpening process based on the optical characteristics after applying blur based on the optical characteristics to the original image or the original image is used. The training method according to claim 8. Claim 10 The blurring applied to the blurred image includes blurring with a large amount of blurring based on the optical characteristics, and the learning method according to any one of claims 1 to 9.

11. As the correct image corresponding to the training data in which the blurring applied to the blurred image is greater than the optical characteristics, after applying blurring to the original image based on the optical characteristics, the sharpness enhancement process is performed based on the optical characteristics. The learning method according to claim 10, characterized in that an image or the original image is used.

12. The blurring applied to the blurred image includes both blurring with a large amount of blurring and blurring with a small amount of blurring based on the optical characteristics, and the learning method according to any one of claims 1 to 11.

13. As the correct image corresponding to the training data in which the blurring applied to the blurred image is smaller than the optical characteristics, after applying blurring to the original image based on the optical characteristics, the sharpness enhancement process is performed based on the optical characteristics. An image or the original image is used, As the correct image corresponding to the training data in which the blurring applied to the blurred image is greater than the optical characteristics, the blurred image corresponding to the training data or the intermediate correction data corresponding to the training data is used. The learning method according to claim 12.

14. The blurring applied to the blurred image includes blurring identical to the optical characteristics, and the learning method according to any one of claims 1 to 13.

15. As the correct image corresponding to the training data in which the blurring applied to the blurred image is identical to the optical characteristics, the original image or the intermediate correction data corresponding to the training data is used. The learning method according to claim 14.

16. The first machine learning model inputs the blurred image in addition to the training data, and the learning method according to any one of claims 1 to 15.

17. The first machine learning model inputs at least one of the shooting conditions of the optical system and the screen position of the captured image, and the learning method according to any one of claims 1 to 16.

18. The blurring applied to the blurred image is based on at least one of the shooting conditions of the optical system and the screen position of the captured image, and the learning method according to any one of claims 1 to 17.

19. A learning device for a machine learning model that outputs a corrected image from intermediate correction data obtained by correcting blur caused by the optical characteristics of an optical system, a first acquisition means for obtaining a blurred image by applying blur to an original image, a second acquisition means for obtaining, as a plurality of training data, the intermediate correction data obtained by performing a sharpening process based on the optical characteristics on the blurred image, a third acquisition means for obtaining a plurality of correct images corresponding to each of the training data based on the original image, a learning means for learning a first machine learning model using learning data composed of the training data and the correct images, and having, the blur applied to the blurred image is different for each of the training data based on the optical characteristics, characterized in that the learning device.

20. A program characterized by causing a computer to execute the learning method according to any one of Claims 1 to 18.

21. A method for manufacturing a learned model for outputting a corrected image from intermediate correction data obtained by correcting blur caused by the optical characteristics of an optical system, a step of obtaining a blurred image by applying blur to an original image, a step of obtaining, as a plurality of training data, the intermediate correction data obtained by performing a sharpening process based on the optical characteristics on the blurred image, a step of obtaining a plurality of correct images corresponding to each of the training data based on the original image, a step of learning a first machine learning model using learning data composed of the training data and the correct images, and having, the blur applied to the blurred image is different for each of the training data based on the optical characteristics, characterized in that the method for manufacturing a learned model.

22. a step of obtaining intermediate correction data obtained by performing a sharpening process based on the optical characteristics of the optical system used for imaging the captured image on the captured image, a step of obtaining a corrected image by a first machine learning model that reduces the influence of overcorrection of the sharpening process from the intermediate correction data, characterized in that the image processing method.

23. a step of synthesizing at least one of the captured image and the intermediate correction data and the corrected image based on a sharpened component obtained based on the captured image and the intermediate correction data, characterized in that the image processing method according to Claim 22.

Images