Image processing method, image processing device, and image processing program

Abstract

To sharpen blur of an image and to suppress an artifact around a luminance saturation area.SOLUTION: An image processing method includes the steps of: generating a first blurred image by blurring a first original image having a signal value equal to or less than a first signal value; acquiring a first correct image from the first original image; and training a first machine learning model based on a first output image generated by inputting the first blurred image to the first machine learning model and the first correct image. In the image processing method, a second original image having a signal value larger than a second signal value is blurred and then clipped at a third signal value to generate a second blurred image; a second correct image having sharpness different from that of the second blurred image is generated based on the second original image; a model output from the first machine learning model to which the second blurred image is input is input to a second machine learning model to generate a third output image; and the second machine learning model is trained based on the third output image and the second correct image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing technique for sharpening blurred images using a machine learning model. [Background technology]

[0002] Patent Document 1 discloses a method for suppressing artifacts, which are spurious structures that do not actually exist around brightness saturated regions (hereinafter referred to as brightness saturated regions), in image deblurring using a neural network (NN). Example 2 of Patent Document 1 discloses a method for generating a correct image in which the brightness saturated region has a larger amount of blur (weak sharpness) than other regions, and training the NN using the correct image. A NN trained in this way can suppress artifacts that tend to occur around brightness saturated regions, at the cost of a reduced effect of sharpening blur around the brightness saturated regions. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-140758 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the method disclosed in Patent Document 1 has difficulty reducing the computational load of the neural network while maintaining the effect of sharpening the blur around the saturated brightness region. In saturated brightness regions, the original subject information is attenuated not only by blur but also by signal value clipping. Therefore, unless the neural network performs different blur sharpening around saturated brightness regions and other regions, a sharpened image with reduced artifacts cannot be generated. However, since sharpening the blur around saturated brightness regions is an extremely difficult task, the neural network must have sufficient degrees of freedom. The degrees of freedom can be increased by increasing the number of convolutional layers or the number of filters corresponding to the number of output channels of the convolutional layer. This is also true for fully connected layers. However, a greater degree of freedom in the neural network increases the computational load.

[0005] The method described in Patent Document 1 can reduce the computational load of the NN when blur sharpening is not performed around brightness saturated regions. This can be achieved by training the NN by making the amount of blur around brightness saturated regions of the correct image the same as the input image (training image in Patent Document 1) of the NN corresponding to the correct image. However, this NN does not achieve any effect of sharpening blur around brightness saturated regions.

[0006] The present invention provides an image processing method and image processing device that can suppress artifacts around brightness saturated areas while reducing the computational load when sharpening blurred images using a machine learning model. [Means for solving the problem]

[0007] An image processing method according to one aspect of the present invention is an image processing method for training a first machine learning model and a second machine learning model, the image processing method including a first step of acquiring a first original image consisting of signal values ​​equal to or less than a first signal value corresponding to a first upper limit that is the upper limit of the signal values ​​of an image input to the first machine learning model, a second step of generating a first blurred image by blurring the first original image, a third step of acquiring a first ground truth image based on the first original image and having sharper sharpness than the first blurred image, a fourth step of inputting the first blurred image to the first machine learning model to generate a first output image, a fifth step of training the first machine learning model based on the first output image and the first ground truth image, and a fifth step of training the first machine learning model based on the first upper limit. The method includes a sixth step of acquiring a second original image having a signal value greater than the corresponding second signal value, a seventh step of blurring the second original image and then clipping it at a third signal value corresponding to the first upper limit to generate a second blurred image, an eighth step of generating a second reference image based on the second original image, a ninth step of inputting the second blurred image into a first machine learning model to generate a model output and inputting the model output into a second machine learning model to generate a third output image, and a tenth step of training the second machine learning model based on the third output image and the second reference image. The second reference image is characterized by having a different sharpness from the second blurred image based on the magnitude of the signal value of the second blurred image.

[0008] Another aspect of the present invention is an image processing method, characterized by comprising an 11th step of inputting a captured image generated by an imaging device capturing an image into a first machine learning model that performs processing independent of brightness saturation of the captured image, thereby generating a first estimated image in which a sharpening component of the blur of the captured image is estimated, and a 12th step of inputting the first estimated image into a second machine learning model, thereby generating a second estimated image in which artifacts present around an area of ​​the first estimated image that corresponds to a brightness saturation area of ​​the captured image are suppressed.

[0009] An image processing device that executes processing according to the image processing method and a program that causes a computer to execute the processing also constitute other aspects of the present invention. [Effects of the Invention]

[0010] According to the present invention, in sharpening blurred images using a machine learning model, it is possible to suppress artifacts around brightness saturated areas while reducing the computational load. [Brief explanation of the drawings]

[0011] [Figure 1] 4A and 4B are diagrams showing the influence of brightness saturation on captured images in Examples 1 and 2. [Figure 2] 4A and 4B are diagrams showing blur sharpening for captured images in the first and second embodiments. [Figure 3] FIG. 1 is a block diagram showing the configuration of an image processing system according to a first embodiment. [Figure 4] FIG. 1 is an external view of an image processing system according to a first embodiment. [Figure 5] 1 is a flowchart of a process for generating a first training data set in Examples 1 and 2. [Figure 6] 4A and 4B are diagrams showing the relationship between a first original image and a first blurred image in the first and second embodiments. [Figure 7] 1 is a flowchart showing a process of training a first machine learning model in the first and second embodiments. [Figure 8] FIG. 2 is a diagram showing the structure of a first machine learning model in Examples 1 and 2. [Figure 9] 10 is a flowchart of a process for generating a second training data set in the first and second embodiments. [Figure 10] 5A and 5B are diagrams showing the relationship between a second original image, a second blurred image, and a second output image in the first and second embodiments. [Figure 11] FIG. 10 is a diagram showing the structure of a second machine learning model in Examples 1 and 2. [Figure 12] FIG. 10 is a diagram showing a function used in the second machine learning model in the first and second embodiments. [Figure 13] 10 is a flowchart showing a process of sharpening a captured image in the first and second embodiments. [Figure 14] 10A and 10B are diagrams illustrating the generation of a second weight map in the first and second embodiments. [Figure 15] FIG. 10 is a block diagram showing the configuration of an image processing system according to a second embodiment. [Figure 16] FIG. 10 is an external view of an image processing system according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] Before describing the embodiments in detail, matters common to each embodiment will be described. First, using FIG. 1, the influence of brightness saturation on the sharpening of blur in a captured image will be described. In the figure, the two-dot chain line indicates the brightness saturation value of the image sensor, and the dashed line indicates the signal distribution of the captured image captured by the image sensor. The captured image contains blur due to aberrations, diffraction, etc. of the optical system. In the figure, the one-dot chain line indicates the original signal distribution before clipping processing based on the brightness saturation value is performed. The solid line indicates the captured image when there is no blur, that is, the signal distribution of the subject.

[0014] The non-saturated blur image 211 has only signal values ​​below the brightness saturation value, and is therefore not affected by clipping due to the brightness saturation value. The same is true for the non-saturated object 201, which is the object before the non-saturated blur image 211 is blurred. The non-saturated object 201 shown here is an object with a flat structure that exhibits signal values ​​equal to the brightness saturation value. However, the object corresponding to the non-saturated blur image 211 may also have signal values ​​higher than the brightness saturation value.

[0015] On the other hand, the saturated blur image 212 has a signal value that exceeds the brightness saturation value, and is affected by clipping due to the brightness saturation value. The saturated blur image 212 is the object before it is blurred. The blur sharpening task is to bring the captured image shown in the dashed line closer to the object shown in the solid line. However, when we consider the relationship between the non-saturated blur image 211 and the non-saturated object 201, and the relationship between the saturated blur image 212 and the saturated object 201, we can see that completely different blur sharpening processes must be performed depending on whether or not the influence of brightness saturation is present. For example, the non-saturated blur image 211 has areas with low signal values ​​compared to the non-saturated object 201, while the saturated blur image 212 always has high signal values ​​compared to the saturated object 202.

[0016] However, it is an extremely difficult task to make the saturated blur image 212 resemble the saturated object 202. This is because the original signal distribution lost due to clipping caused by the brightness saturation value is unknown, and even if estimation is performed from the remaining surrounding information, the original signal distribution cannot be uniquely determined. Nevertheless, by learning a large amount of correspondences such as those between the saturated blur image 212 and the saturated object 202, a machine learning model can sharpen the brightness-saturated blur image while suppressing artifacts.

[0017] However, this requires a sufficient degree of freedom in the machine learning model, which results in a large computational load. A machine learning model with few degrees of freedom and a small computational load will produce artifacts around brightness-saturated regions even if the blur is sharpened. As in Patent Document 1, if the blur sharpening effect in brightness-saturated regions is reduced, the degree of freedom required for the machine learning model will decrease slightly, but the degree of freedom cannot be significantly reduced unless no sharpening is performed at all.

[0018] Therefore, in each embodiment, the configuration shown in Figure 2 is used to achieve blur sharpening that has a sharpening effect even on blurred images with saturated brightness, while reducing the computational load on the machine learning model and suppressing the occurrence of artifacts.

[0019] A blurred captured image 221 is input to a first machine learning model 231. The signal distribution of the captured image 221 is indicated by a dashed line. For clarity of explanation, for reference, the signal distribution of a subject without blur is indicated by a solid line, the brightness saturation value is indicated by a two-dot chain line, and the signal distribution before clipping at the brightness saturation value is indicated by a one-dot chain line.

[0020] The first machine learning model 231 has learned a large amount of only the correspondence between the non-saturated blur image 211 and the non-saturated subject 201 shown in FIG. 1. Therefore, the first machine learning model 231 sharpens the blur by assuming that the entire captured image 221 is a blur image with no brightness saturation. In other words, the first machine learning model 231 performs processing that is not dependent on the brightness saturation of the captured image 221. In this case, the first machine learning model 231 does not learn at all how to sharpen the blur for a brightness-saturated blur image, which is highly difficult, and therefore the degree of freedom can be reduced.

[0021] In the first estimated image 222 output from the first machine learning model 231, the blurred image without brightness saturation is correctly sharpened. However, the blurred image with brightness saturation is slightly sharpened (to the same extent as when the blurred image without brightness saturation is used), but artifacts occur. The artifacts referred to here are, for example, a drop in signal value that is not present in the actual subject (solid line), as in the first estimated image 222.

[0022] The first estimated image 222 is input to the second machine learning model 232. This generates a second estimated image 223 in which artifacts are suppressed. The second machine learning model 232 suppresses artifacts around brightness saturated regions of the input first estimated image 222, and hardly performs sharpening on the brightness saturated blurred image, which is difficult. Therefore, the second machine learning model 232 can also ensure task accuracy with a small degree of freedom.

[0023] In the second estimated image 223, compared to the captured image 221, blurred images that are not saturated in brightness are sharpened with high precision, and blurred images that are saturated in brightness are sharpened, although the sharpening effect is reduced, and artifacts are suppressed.

[0024] To achieve the above configuration, the first machine learning model 231 and the second machine learning model 232 are trained using different training data. The training data is a pair of a ground truth image and an input image to the machine learning model corresponding to the ground truth image.

[0025] The first machine learning model 231 is trained using a first ground truth image based on a first original image having a signal value equal to or less than a first signal value corresponding to the brightness saturation value of the captured image, and a first blurred image (described later). The second machine learning model 232 is trained using a second ground truth image based on a second original image having a signal value greater than a second signal value corresponding to the brightness saturation value, and a second blurred image. The reason why the second signal value and the first signal value may be different values ​​will be described in Example 1 (described later).

[0026] The examples will be specifically described below.

[0027] [Example 1] Fig. 3 shows the configuration of the image processing system 100 in Example 1. Fig. 4 shows the external appearance of the image processing system 100. The image processing system 100 includes a training device 101 and an imaging device 102.

[0028] The imaging device 102 has as its components an optical system 121, an image sensor 122, a memory unit 123, a communication unit 124, an acquisition unit 125, a calculation unit 126, and a display unit 127. The optical system 121 collects light incident from the subject space to form a subject image. The image sensor 122 receives the subject image and performs photoelectric conversion to generate a captured image 221 as image data. The captured image 221 undergoes predetermined image processing, such as removal of defective pixels, in the calculation unit 126 and is then stored in the memory unit 123.

[0029] The captured image 221 contains blur due to aberration and diffraction occurring in the optical system 121. The calculation unit 126 sharpens the blur contained in the captured image 221 using first and second machine learning models 231 and 232 to generate a sharpened image. The captured image 221 whose blur is to be sharpened may be an undeveloped RAW image or a developed image. In this embodiment, the blur is sharpened on the RAW image. The blur sharpening may be performed automatically when capturing an image, or may be performed on the saved captured image 221 at a timing selected by the user.

[0030] The model parameters used in the first and second machine learning models 231, 232 are trained by the training device 101, acquired in advance by the imaging device 102 via the communication units 112 and 124, and stored in the storage unit 123. Details of the training of the first and second machine learning models 231, 232 and the blur sharpening by the trained first and second machine learning models 231, 232 will be described later. The sharpened image after blur sharpening is displayed on the display unit 127 and stored in the storage unit 123.

[0031] Next, we will explain the training of the first and second machine learning models 231 and 232 executed by the training device 101, i.e., the determination of model parameters and the generation of training data used for each training. Model parameters are parameters of the machine learning models determined by training. For example, they include the weight coefficients of the fully connected layer, the filter coefficients of the convolutional layer, and the bias.

[0032] The training device 101 has, as its components, a storage unit 111, a communication unit 112, an acquisition unit 113, a calculation unit 114, a determination unit 115, and an update unit 116, and the training and training data generation processes are executed by the corresponding components.

[0033] 5 shows a process for generating a first training data set used for training the first machine learning model 231. The first training data set includes a plurality of training data. The training data is a pair of a first ground truth image and a first blurred image that is an input image of the first machine learning model 231 corresponding to the first ground truth image.

[0034] In step S101 (first step), the acquisition unit 113 acquires one or more first original images from the storage unit 111. The first original images may be real-life images or CG (Computer Graphics) images. In this embodiment, a real-life RAW image is used to sharpen the RAW image as described above. If the RAW image has only specific color components in each pixel, such as a Bayer array, demosaicing is performed so that all pixels have all color components (e.g., three colors: R: Red, G: Green, and B: Blue). In this case, by downscaling the first original image, the influence of differences in Nyquist frequencies for each color and errors due to demosaicing can be reduced. Furthermore, downscaling can also reduce the influence of blurring and noise due to aberrations and diffraction in the optical system that occur when capturing the first original image, thereby enabling the preparation of a high-quality first original image.

[0035] The first original image consists of signal values ​​equal to or less than the first signal value (has only this signal value). If the first original image has a signal value greater than the first signal value, it is clipped to the first signal value before step S103. The first signal value is a signal value corresponding to the luminance saturation value of the captured image 221 input to the trained first machine learning model 231. The luminance saturation value is the upper limit of the signal value that the captured image 221 can assume. However, the luminance saturation value of the captured image 221 and the first signal value do not necessarily have to be the same value. For example, if the signal values ​​of an image (first blurred image or captured image 221) are normalized when inputting the image to the first machine learning model 231, the upper limits of the signal values ​​of both images do not necessarily have to be the same before normalization. A specific example of the normalization method will be described later in step S202.

[0036] In step S102, the acquisition unit 113 acquires one or more PSFs (Point Spread Functions) from the storage unit 111. The PSF has RGB color components. The PSF corresponds to blurring due to aberration and diffraction occurring in the optical system 121 of the image capture device 102. The PSF of the optical system 121 varies depending on conditions including the state of the optical system 121 (focal length, F-number, and focus distance) and image plane coordinates (two-dimensional spatial coordinates in the vertical and horizontal directions on the image-side focus plane). Therefore, the acquisition unit 113 determines the above conditions and acquires a PSF corresponding to the conditions. At this time, multiple PSFs corresponding to multiple different conditions may be acquired. Alternatively, an OTF (Optical Transfer Function) may be acquired instead of a PSF, or data approximating the PSF or OTF may be acquired. Note that the focus distance is the distance between the subject focused by the optical system 121 and either the image sensor 122 or the optical system 121. Note that the order of execution of steps S101 and S102 may be reversed.

[0037] In this embodiment, the target of sharpening is blur due to aberration and diffraction of the optical system 121, but other blurs can also be sharpened. Other blurs include defocus blur, vibration, air turbulence, etc. When sharpening these blurs, the PSFs corresponding to each blur are acquired in step S102.

[0038] In step S103 (second process), the calculation unit 114 blurs the first original image with the PSF to generate a first blurred image. FIG. 6 shows an overview of the processing of step S103. In step S103, the first original image 241 is blurred with the PSF 242 to generate a first blurred image 243. The first blurred image 243 is an image input to the first machine learning model 231 when training the first machine learning model 231. The two-dot chain line in FIG. 6 indicates the first signal value.

[0039] If there are multiple first original images 241 or PSFs 242, multiple first blurred images 243 are generated for each combination. The PSFs 242 are convolved with the first original image 241 for each color. When an OTF is used, the product of the OTF and the frequency characteristics (the result of the Fourier transform) of the first original image 241 is calculated for each color. Note that in this embodiment, blurring is performed for each RGB color for simplicity's sake. However, blurring may be performed for each single wavelength component and then combined using weighting corresponding to each color of the color filter of the image sensor 122. Also, while the reduction of the first original image 241 was described in step S101, the sampling density of the PSFs 242 may be set finer than the pixels of the image sensor 122, and the first original image 241 before reduction may be blurred and then reduced to the same sampling density as the pixels of the image sensor 122.

[0040] The calculation unit 114 samples the result of blurring the first original image 241 with the PSF 242 using the color filter array (Bayer array in this embodiment) of the image sensor 122, and adds noise generated by the image sensor 122 to the sampling result to generate a first blurred image 243. Because the strength of the noise varies depending on the ISO sensitivity of the image sensor 122, the ISO sensitivity is determined and the corresponding noise is added. If noise is not added to the first blurred image 243, the trained first machine learning model 231 will amplify the noise while sharpening the blur, so it is desirable to add noise to the first blurred image 243. However, if the captured image 221 is noise-removed in advance and then input to the first machine learning model 231, adding noise to the first blurred image 243 is not essential.

[0041] In step S104 (third step), the calculation unit 114 acquires a first correct image based on the first original image 241. The first correct image is an image that contains the same subject as the first blurred image 243 and has stronger sharpness (smaller blur amount) than the first blurred image 243. The strength of sharpness can be expressed by the steepness of the edge of the image (magnitude of the absolute value of the spatial differential) or the magnitude of each frequency component in the frequency characteristics of the image (result of Fourier transform). If a plurality of first blurred images 243 has been generated in step S103, the calculation unit 114 acquires a plurality of first correct images corresponding to the plurality of first blurred images 243, respectively.

[0042] The first reference image may be the first original image 241 used as is, or may be generated by performing any processing on the first original image 241. For example, the first original image 241 may be blurred using a PSF with a blur amount smaller than the PSF 242. If the first original image 241 is reduced when generating the first blurred image 243, the first reference image is generated by similarly reducing the image. In this embodiment, the first original image 241 is reduced and sampled using a Bayer array as necessary, and then noise correlated with the noise added when generating the first blurred image 243 in step S103 is added to generate the first reference image.

[0043] If the first blurred image 243 contains noise and the first reference image does not contain noise, the first machine learning model 231 is trained to also perform noise removal. Simultaneously performing blur sharpening and noise removal is highly difficult and requires the first machine learning model 231 to have many degrees of freedom. Therefore, in this embodiment, noise is also added to the first reference image. If correlated noise exists between the first blurred image 243 and the first reference image, the trained first machine learning model 231 can perform blur sharpening while suppressing noise fluctuations. If there is no correlation between the noise between the first blurred image 243 and the first reference image, the trained first machine learning model 231 may perform noise removal. This principle is described in the following literature 1. [Reference 1] J. Lehtinen, el al., “Noise2Noise: Learning Image Restoration without Clean Data”, ICML2018. Note that if noise is added, the signal value may exceed the first signal value, so the first blurred image 243 and the first genuine image are clipped at the first signal value. If noise is not added to the first blurred image 243, it is not necessary to add noise to the first genuine image. Furthermore, the process of step S104 may be executed at any time between steps S101 and S105.

[0044] In step S105, the determination unit 115 determines whether or not the generation of the first training data set is complete. If it is complete, the determination unit 115 stores all of the generated training data as the first training data set in the storage unit 111. If it is not complete, the process returns to step S101, and the acquisition unit 113 and the calculation unit 114 acquire a new first original image 241 and a PSF 242 corresponding to the new conditions, and similarly repeats the processes of steps S102 to S104.

[0045] In this embodiment, all blur caused by aberration and diffraction occurring in the optical system 121 is sharpened using common model parameters. For this reason, the first training data set needs to include first blurred images 243 blurred with various PSFs 242 corresponding to the optical system 121. Whether generation of the first training data set is complete can be determined based on whether the PSFs 242 used to generate the first blurred images 243 included in the first training data set have sufficient variation. Note that, if noise is added to the first blurred images 243, whether noise at various ISO sensitivities that the image sensor 122 can take is also included as a criterion for determination.

[0046] Through the above processing, it is possible to generate a pair of a first blurred image 243 that is not affected by brightness saturation and a corresponding first ground truth image that has no or little blur. By training the first machine learning model 231 with this first training data set, the first machine learning model 231 learns only how to sharpen blurred images that are not affected by brightness saturation.

[0047] Next, a method (process) for training the first machine learning model 231 using the first training data set will be described with reference to the flowchart of FIG.

[0048] In step S201, the acquisition unit 113 acquires one or more sets of training data from a first training data set. The training data is training input data including a first blurred image 243 and a first ground truth image. The training input data is data input to the first machine learning model 231 during training. In this embodiment, in addition to the first blurred image 243, blur specification information (first blur specification information) that specifies the blur assigned to the first blurred image 243 is input to the first machine learning model 231. The blur specification information in this embodiment is information on five values ​​of the focal length, F-number, focus distance, and vertical and horizontal image plane coordinates of the optical system 121 that correspond to the PSF 242 assigned to the first blurred image 243. The information on the five values ​​may indicate the five values ​​themselves or may be information that can be converted into five values.

[0049] However, the blur specification information may be other information as long as it can specify the blur. For example, it may be information about the two-dimensional intensity distribution of PSF242 or OTF, or information about each coefficient, etc., obtained by approximating them using a Gaussian mixture model. The information about the intensity distribution, coefficient, etc. may be information that indicates the intensity distribution, coefficient, etc. itself, or may be information that can be converted into the intensity distribution, coefficient, etc.

[0050] If the image capture device 102 is an interchangeable lens camera, the optical system 121 may be connected to multiple image capture devices 102 with different pixel pitches. In this case, the magnitude of the PSF 242 changes depending on the pixel pitch, so information about the pixel pitch may be included in the blur specification information. The information about the pixel pitch may be information that indicates the pixel pitch itself or information that can be converted into the pixel pitch.

[0051] In the first blurred image 243, the number of pixels, which are the number of elements in the vertical direction, horizontal direction, and channel direction, respectively, is (H, W, C), and in this embodiment assuming a Bayer array, C=1. The acquisition unit 113 rearranges the first blurred image 243 to (H / 2, W / 2, 4) before inputting it to the first machine learning model 231. At this time, pixels with odd vertical and horizontal indices are set to C=1, and pixels with odd vertical indices and even horizontal indices are set to C=2. Similarly, pixels with even vertical indices and odd horizontal indices are set to C=3, and pixels with even vertical and horizontal indices are set to C=4. However, the rearrangement method is not limited to this. Because the shape of the blur differs for the three RGB colors, sharpening is made easier by dividing each color into multiple channels (G is divided into two channels).

[0052] In this embodiment, the blur specification information is a map having five values ​​for each channel: the focal length, F-number, focus distance, and vertical and horizontal image plane coordinates of the optical system 121. The number of elements in each direction of the map is (H / 2, W / 2, 5), and is concatenated with the first blurred image 243 in the channel direction.

[0053] Furthermore, in addition to the blur specification information, noise information indicating the intensity of noise added to the first blurred image 243 may be input to the first machine learning model 231. For example, a map having values ​​indicating the variance and standard deviation of noise for a certain brightness is linked to the first blurred image 243 in the channel direction and input to the first machine learning model 231. At the time of estimation after training, if the noise characteristics of the image sensor 122 are known, it is possible to identify the variance and standard deviation of noise corresponding to the captured image 221 from information about the ISO sensitivity at the time of capturing. Inputting the noise information to the first machine learning model 231 makes it easier to distinguish between noise and the subject, and makes it possible to suppress fluctuations in noise due to sharpening of the blur.

[0054] In step S202 (fourth process), the calculation unit 114 inputs the training input data to the first machine learning model 231 and generates a first output image. If the upper limit (first upper limit) of the signal values ​​of the image input to the first machine learning model 231 and the upper limit of the signal values ​​of the first blurred image 243 are not the same value, the signal values ​​of the first blurred image 243 are scaled so that these upper limits become the same, and then the first blurred image 243 is input to the first machine learning model 231. The scaling is expressed by the following equation (1).

[0055]

number

[0056] In equation (1), I nor is the signal value after scaling, I is the signal value of each pixel of the first blurred image 243, S is the upper limit of the signal value, B is the black level of optical black, L M1 is the first upper limit. The first upper limit L M1 is 1. Therefore, normalization is performed on the first blurred image 243 by equation (1) so that the upper limit of the signal value becomes 1.

[0057] On the other hand, the lower limit of the signal value may have a negative value rather than 0 due to the influence of noise. Note that when generating the first blurred image 243 from the first original image 241, unless a process of multiplying all pixels by a uniform coefficient other than 1, i.e., a process of changing brightness, is performed, the upper limit of the signal value of the first blurred image 243 will match the first signal value. If the first original image 241 and the first blurred image 243 are normalized from the beginning, scaling using equation (1) is not necessary. In this case, the first signal value, the first upper limit, and the upper limit of the signal value of the first blurred image 243 are all 1, which is the same value.

[0058] In this embodiment, a CNN (Convolutional Neural Network) having the configuration shown in Figures 8(A) and 8(B) is used as the first machine learning model 231. However, other machine learning models such as an MLP (Multi-Layer Perceptron), a ViT (Vision Transformer), a GAN (Generative Adversarial Network), or a diffusion model may also be used.

[0059] The first blurred image 243 and the blur identification information 244 included in the training input data are input to the first machine learning model 231, concatenated in the channel direction, and then input to a convolution layer (Convolution in FIG. 8 ). Note that the blur identification information 244 may be normalized before input, similar to the first blurred image 243. The input method is not limited to this, and the first blurred image 243 and the blur identification information 244 may be input to separate convolution layers, and the generated feature maps may be concatenated. Furthermore, the blur identification information 244 may be input to a fully connected layer as a five-dimensional vector instead of a map, converted into a feature map, and then concatenated with the first blurred image 243.

[0060] The number of taps in the vertical and horizontal directions of the filter in each convolution layer is (3,3). The number of filters, which corresponds to the number of channels of the feature map output from the convolution layer, is 24. However, the number of taps and filters is not limited to this. Activation is the activation function, and ReLU (Rectified Linear Unit) is used, but Leaky ReLU or a sigmoid function may also be used.

[0061] The residual block has the configuration shown in FIG. 8(B). The residual block generates a feature map 247 output from the input feature map 246 in the configuration shown in FIG. 8(B). Here, Sum represents the sum of each element. In this embodiment, after passing through 10 residual blocks, the result output from the last convolutional layer is summed with the first blurred image 243 to generate a first output image 245. Note that instead of performing the final summation, the residual image, which is a sharpening component summed with the first blurred image 243, may be used as the first output image 245. If multiple first blurred images 243 are acquired, a first output image 245 is generated for each of them. Note that the first machine learning model 231 generates a first feature map 248 when generating the first output image 245. Although the first feature map 248 is input to the last convolutional layer in FIG. 8(A), a feature map at another position may also be used.

[0062] In step S203 (fifth step), the update unit 116 updates the model parameters of the first machine learning model 231 based on the error between the first output image 245 and the first ground truth image. That is, the first machine learning model 231 is trained. In this embodiment, the model parameters are the filter coefficients and biases of each convolutional layer. Note that, in the first training, the model parameters are determined by random numbers. Also, in this embodiment, the MSE (Mean Square Error) between the first output image 245 and the first ground truth image is used as the loss function, but other functions may also be used. For example, the MAE (Mean Absolute Error) or the Perceptual Loss may also be used.

[0063] Since the number of elements in the first output image 245 is (H / 2, W / 2, 4), the updating unit 116 also rearranges the first reference image in the same way as the first blurred image 243 to match the number of elements in each direction with that of the first output image 245. Furthermore, if the first output image 245 is a residual image, the updating unit 116 adds it to the first blurred image 243 and then calculates the error between it and the first reference image, or calculates the error between the result of subtracting the first blurred image 243 from the first reference image. If there are multiple first reference images and multiple first output images 245, the updating unit 116 calculates the error for each of them. Furthermore, if the scales of the signal values ​​of the first reference image and the first output image 245 do not match—for example, if the first output image 245 is normalized but the first reference image is not—the updating unit 116 scales one of them and then calculates the error. To update the model parameters, backpropagation or the like is used.

[0064] In step S204, the determination unit 115 determines whether or not the training is complete. If not, the process returns to step S201 to acquire one or more new sets of training input data and a first correct answer image. If the training is complete, the model parameters are stored in the storage unit 111. The determination unit 115 determines whether or not the training is complete based on the number of updates, the magnitude of the update amount of the model parameters, etc.

[0065] The above process makes it possible to generate a first machine learning model 231 that has learned to sharpen only blurred images that are not affected by brightness saturation. By not learning to sharpen blurred images that are affected by brightness saturation, which is highly difficult, the first machine learning model 231 can perform highly accurate sharpening even with fewer degrees of freedom. However, because it has not learned to sharpen blurred images that are affected by brightness saturation, artifacts occur around brightness saturated areas. These artifacts are suppressed by the second machine learning model 232, which will be described later.

[0066] Next, a method (process) for generating a second training data set used for training the second machine learning model 232 will be described with reference to the flowchart of FIG.

[0067] In step S301 (sixth step), the acquisition unit 113 acquires a second original image having a signal value greater than the second signal value. The second signal value corresponds to the brightness saturation value of the captured image 221 and also corresponds to the upper limit of the signal value of the image input to the first machine learning model 231. As explained in step S101, the respective values ​​do not necessarily have to be the same. Here, all of the training data included in the second training data set does not have to be generated from second original images having signal values ​​greater than the second signal value.

[0068] In step S302, the acquisition unit 113 acquires a PSF in the same manner as in step S102.

[0069] In step S303 (seventh step), the calculation unit 114 blurs the second original image with the PSF and then clips it with a third signal value to generate a second blurred image. Fig. 10 shows an overview of step S303. The calculation unit 114 blurs the second original image 251 with the PSF 252 and then clips it with the third signal value to generate a second blurred image 253. Sampling and noise addition in the color filter array of the image sensor 122 are performed in the same manner as in step S103.

[0070] The third signal value corresponds to the brightness saturation value of the captured image 221 and also corresponds to the upper limit of the signal value of the image input to the first machine learning model 231. When generating the second blurred image 253 from the second original image 251, the second signal value and the third signal value will match unless a process is performed in which all pixels are multiplied by a uniform coefficient other than 1 (or unless the sum of the PSFs 252 is not 1). In the figure, the two-dot chain line of the second original image 251 represents the second signal value, and the two-dot chain line of the second blurred image 253 represents the third signal value. In training the second machine learning model 232, the second blurred image 253 is input to the first machine learning model 231, which will be described in detail later.

[0071] In step S304 (eighth process), the calculation unit 114 generates a second reference image 255 based on the second original image 251. The second reference image 255 has a different sharpness from the second blurred image 253 based on the magnitude of the signal value of the second blurred image 253. The difference in sharpness between a first region of the second blurred image 253 and a region of the second reference image 255 corresponding to the first region is smaller than the difference in sharpness between a second region of the second blurred image 253 and a region of the second reference image 255 corresponding to the second region. In this case, the first region of the second blurred image 253 has a larger average signal value than the second region of the second blurred image 253. Alternatively, the first region of the second blurred image 253 includes saturated pixels, and the second region of the second blurred image 253 does not include saturated pixels.

[0072] A method in this embodiment for generating the second ground truth image 255 having the above-mentioned properties will now be described in detail. First, the calculation unit 114 rearranges the second blurred image 253, which is a Bayer array, in the same manner as described in step S201 to convert it into four channels. The calculation unit 114 generates a brightness saturation map in which areas of the second blurred image 253 that have been converted into four channels and are equal to or greater than a third signal value (corresponding to the brightness saturation value of the captured image 221) are set to 1, and other areas are set to 0. The calculation unit 114 also generates a one-channel map by blurring each channel of the brightness saturation map with the PSF 252 of the corresponding color component and then adopting the maximum value in the channel direction. The calculation unit 114 upscales the map by doubling the number of elements in each of the vertical and horizontal directions using bilinear interpolation or the like to return the number of elements to the same as that of the second blurred image 253, thereby generating a first weighting map.

[0073] Furthermore, the calculation unit 114 generates a third correct image by clipping the second original image 251 with the second signal value and performing scaling to match the scale of the signal value with that of the second blurred image 253 (making the second signal value equal to the third signal value). The third correct image has stronger sharpness than the second blurred image 253. When the third correct image is G3, the first weighting map is M, and the second blurred image 253 is I2, the second correct image 255G2 is generated by the following weighted average equation (2).

[0074]

number

[0075] where c is an arbitrary coefficient greater than or equal to 0, f min is a MIN function that uses the smaller of the two arguments. Equation (2) enables the generation of a second correct answer image 255 in which areas unrelated to brightness saturation in the second blurred image 253 have the same sharpness as the third correct answer image, and areas surrounding the second blurred image 253 that are more saturated with brightness than the areas in the first correct answer image have weaker sharpness. The sharpness around the saturated brightness area can be controlled by the coefficient c, and the larger c becomes, the closer the image becomes to the second blurred image 253.

[0076] When generating the first weight map, the maximum value is taken in the channel direction (i.e., in adjacent RGGB) in order to prevent unintended coloring in the second ground truth image 255 due to the weights of the first weight map changing for each color. The brightness saturation map used to generate the first weight map may be generated from the second output image 254, which is a model output when the second blurred image 253 is input to the first machine learning model 231. The model output refers to at least one of the data generated when the second blurred image 253 is input to the first machine learning model 231. Specifically, the model output refers to at least one of the multiple feature maps and the second output image 254, which are outputs of each convolution layer.

[0077] Note that the method for generating the second ground truth image 255 is not limited to this. For example, the second ground truth image 255 may be generated by multiplying the second original image 251 by the brightness saturation map of the second blurred image 253, converting the map blurred by the PSF 252 into four channels, and repeating the same process as described above.

[0078] Furthermore, in this embodiment, similar to step S104, correlated noise is added to the second blurred image 253 and the second correct image 255. This is to suppress fluctuations in noise, similar to step S104.

[0079] In step S305, the determination unit 115 determines whether or not the generation of the second training data set is complete. If it is complete, all of the generated training data is stored in the storage unit 111 as the second training data set. If it is not complete, the process returns to step S301, a new second original image 251 and a PSF 252 under new conditions are obtained, and the same process is repeated. As in step S105, a second blurred image 253 blurred by various PSFs 252 generated in the optical system 121 is included in the second training data set.

[0080] The above processing makes it possible to generate a pair of a second blurred image 253 that includes both blurred images affected by brightness saturation and blurred images not affected by brightness saturation, and a corresponding second ground truth image 255. When the second blurred image 253 is input to the first machine learning model 231, a second output image 254 is generated. In the second output image 254, the blurred image not affected by brightness saturation is sharpened with high precision, but the blurred image affected by brightness saturation has not been learned, and therefore, although it is sharpened to some extent, artifacts occur.

[0081] The second machine learning model 232 is trained based on the error between a third output image generated by inputting the second output image 254 into the second machine learning model 232 and a second correct answer image 255. In areas of the second blurred image 253 that are not affected by brightness saturation, the blur is sharpened in both the second output image 254 and the second correct answer image 255, and therefore the second machine learning model 232 is trained to do almost nothing in those areas.

[0082] In contrast, in the region of the second blurred image 253 affected by brightness saturation, the second output image 254 has the blur sharpened to a certain extent, but artifacts still occur. In the second ground truth image 255, the blur is sharpened to a certain extent in that region, but there are no artifacts. Therefore, the second machine learning model 232 is trained to suppress artifacts while maintaining the blur sharpening effect on the blurred image affected by brightness saturation. Because it focuses only on suppressing artifacts, the second machine learning model 232 can achieve high accuracy even with a small degree of freedom.

[0083] Next, a method for training the second machine learning model 232 with the second training dataset will be described with reference to the flowchart of FIG.

[0084] In step S201, the acquiring unit 113 acquires one or more sets of training data from the second training data set. The training data is training input data including a second blurred image 253 and a second ground truth image 255. The training input data includes data input to the first machine learning model 231 during training and data input to the second machine learning model 232. The first machine learning model 231 is input with the second blurred image 253 and blur specification information (second blur specification information) that specifies the blur assigned to the second blurred image 253. Note that the second blur specification information may be the same as or different from the first blur specification information described above.

[0085] The second machine learning model 232 receives as input a brightness saturation map and a second output image 254 generated by the first machine learning model 231. The brightness saturation map is a map that indicates brightness saturated regions of either the second blurred image 253 or the second output image 254. For example, it is a map in which brightness saturated regions of the second blurred image 253 are assigned a value of 1 and other regions are assigned a value of 0. Therefore, the training input data includes the second blurred image 253, blur specification information, and the brightness saturation map.

[0086] In step S202 (ninth step), the calculation unit 114 generates a third output image from the training input data using the first machine learning model 231 and the second machine learning model 232. Here, similar to the training of the first machine learning model 231, the second blurred image 253 and blur specification information are input to the first machine learning model 231 to generate a second output image 254. Similar to the training of the first machine learning model 231, the upper limit (first upper limit) of the signal value of the image input to the first machine learning model 231 and the upper limit of the signal value of the second blurred image 253 are set to the same value. If these upper limits are not the same value, the signal value of the second blurred image 253 is scaled so that they are the same, and then input to the first machine learning model 231.

[0087] In the first embodiment, the second machine learning model 232 is a CNN having the configuration shown in FIG. 11. However, this is not limited to this. In FIG. 11, Concat. indicates concatenation in the channel direction, Conv. indicates a convolutional layer, and Activ. indicates an activation function (ReLU in the first embodiment). Res. Blocks indicates a structure in which one or more (two in the first embodiment) residual blocks shown in FIG. 8(B) are connected. Space to Depth and Depth to Space are functions that perform the sorting shown in FIG. 12. When data with the number of elements in the vertical, horizontal, and channel directions of (h, w, c) is input, Space to Depth sorts the data into (h / 2, w / 2, 4). In FIG. 12, a 2x2 range is used as one unit for the vertical and horizontal directions, but this is not limited to this. For example, 3x3 may be used as one unit, and in this case the output is (h / 3, w / 3, 9). Depth to Space is a function that performs the inverse conversion of Space to Depth.

[0088] In the first embodiment, the second output image 254 and the intensity saturation map 256 are concatenated in the channel direction and input to the convolution layer. However, the input method to the convolution layer is not limited to this. Since the second output image 254 has the number of elements in each direction of (H / 2, W / 2, 4), the intensity saturation map 256 is similarly rearranged to have the number of elements of (H / 2, W / 2, 4). As with the first machine learning model 231, noise information may be further input to the second machine learning model 232. Furthermore, a second blurred image 253 may be further input to the second machine learning model 232.

[0089] Alternatively, a second feature map may be input to the second machine learning model 232 as a model output generated by inputting the second blurred image 253 to the first machine learning model 231 instead of the second output image 254. Furthermore, the second feature map may be input to the second machine learning model 232 together with the second output image 254. The second feature map is a feature map generated at a position corresponding to the first feature map 248 when the second blurred image 253 is input to the first machine learning model 231 instead of the first blurred image 243. The second feature map includes information related to sharpening that can generate the second output image 254, and therefore a similar effect can be obtained by inputting the second feature map to the second machine learning model 232.

[0090] However, in order for the second feature map to contain sufficient information related to sharpening, it is desirable that it be generated in a layer close to the layer that outputs the second output image 254. Specifically, it is desirable that the second feature map is the feature map output in the layer after 90% or more of all product-sum operations executed in the first machine learning model 231 have been executed.

[0091] As shown in Figure 8(A), if the second feature map corresponding to the first feature map 248 is generated before the second output image 254 corresponding to the first output image 245, the calculations of the first machine learning model 231 can be interrupted up to the generation of the second feature map, and the second output image 254 does not need to be generated.

[0092] The number of filter taps in each layer of the second machine learning model 232 is (3,3), and the number of filters is 16 in the layer included in the first stage 261, 64 in the layer included in the second stage 262, and 256 in the layer included in the third stage 263. However, the number of filter taps and the number of stages (corresponding to the number of Space to Depth and Depth to Space) are not limited to these. The second machine learning model 232 generates a third output image 257.

[0093] In step S203 (tenth process), the update unit 116 updates the model parameters of the second machine learning model 232 based on the error between the third output image 257 and the second correct image 255. That is, the second machine learning model 232 is trained. Here, as in the training of the first machine learning model 231, the MSE between the third output image 257 and the second correct image 255 is used as the loss function, but this is not limiting. Also, if the third output image 257 is a residual image to be added to the second blurred image 253 or the second output image 254, the error is calculated in a similar manner.

[0094] In step S204, the determination unit 115 determines whether the training is complete. If not, the process returns to step S201 to acquire one or more new sets of training input data and a second supervised image 255. If the training is complete, the model parameters are stored in the storage unit 111.

[0095] In this embodiment, the first machine learning model 231 is trained first, and then the second machine learning model 232 is trained, but the order is not limited to this. For example, training may be performed by repeating updates alternately, such as updating the model parameters of the first machine learning model 231, then updating the model parameters of the second machine learning model 232, and then updating the first machine learning model 231.

[0096] Through the above processing, it is possible to generate a second machine learning model 232 that suppresses artifacts around brightness saturated areas that occur as a result of sharpening blur.

[0097] Next, the sharpening of blur in the captured image 221 using the first machine learning model 231 and the second machine learning model 232, which have been trained, will be described with reference to the flowchart of FIG.

[0098] In step S401, the acquisition unit 125 acquires the captured image 221. The captured image 221 may be the entire image or a partial image thereof.

[0099] In step S402, the acquisition unit 125 acquires trained model parameters of each of the first machine learning model 231 and the second machine learning model 232, and blur identification information that identifies blur in the captured image 221. If the imaging device 102 is an interchangeable lens camera and model parameters exist for each type of optical system 121, the type of optical system 121 used for capturing the image is identified from the metadata of the captured image 221, and the corresponding trained model parameters are acquired.

[0100] As the blur specifying information, the acquiring unit 125 acquires information relating to the focal length, F-number, and focus distance at the time of capturing the image from the metadata added to the captured image 221. The information relating to the focal length, F-number, and focus distance may be information indicating the focal length, F-number, and focus distance themselves, or may be information convertible into these.

[0101] Furthermore, if the captured image 221 is a partial image, the acquisition unit 125 acquires information on the position where the partial image is extracted from the entire image, and acquires the image plane coordinates. When the captured image 221 is input to the first machine learning model 231, the number of elements of the captured image 221 is rearranged from (H0, W0, 1) to (H0 / 2, W0 / 2, 4), as in training. The map representing the image plane coordinates at this time represents the spatial coordinates in the vertical and horizontal directions for every two pixels in the captured image 221 before the number of elements is rearranged, and the number of elements in the map is (H0 / 2, W0 / 2, 2).

[0102] Furthermore, when capturing an image while performing image stabilization to reduce image blur by shifting the image sensor 122, the acquisition unit 125 acquires the amount of shift (amount of deviation from the optical axis) of the image sensor 122 during capturing from the metadata and reflects this in the image plane coordinate values. The number of elements in the map representing the state of the optical system 121 (focal length, F-number, focus distance) is (H0 / 2, W0 / 2, 3), and each channel has the same value. If the blur specification information also includes the pixel pitch of the image sensor 122, the map will have a total of six channels.

[0103] In step S403, the calculation unit 126 generates a brightness saturation map that represents brightness saturation regions in the captured image 221. Specifically, the calculation unit 126 acquires a brightness saturation value from metadata of the captured image 221, and generates a brightness saturation map in which brightness saturation regions having signal values ​​equal to or greater than the brightness saturation value indicate 1 and other regions indicate 0. Note that, if the brightness saturation map is generated from the second output image 254 during training of the second machine learning model 232, the calculation unit 126 generates the brightness saturation map from the first estimated image 222 that is generated by inputting the captured image 221 to the first machine learning model 231. The order of the processing of step S403 does not matter as long as it is performed after the processing of step S401 and before the calculation using the second machine learning model 232 in step S404.

[0104] In step S404 (the eleventh and twelfth steps), the calculation unit 126 uses the first machine learning model 231 and the second machine learning model 232 to generate a second estimated image 223 in which blur due to aberration and diffraction of the optical system 121 included in the captured image 221 has been sharpened. The calculation unit 126 also inputs the captured image 221 and blur identification information to the first machine learning model 231 to generate a first estimated image 222. At this time, as in training, the captured image 221 is normalized and input to the first machine learning model 231. The first machine learning model 231 performs processing that is independent of brightness saturation of the captured image 221. In the first estimated image 222, blur is sharpened with high accuracy in areas of the captured image 221 that are not affected by brightness saturation. In contrast, blur is sharpened to a certain extent in areas of the captured image 221 that are affected by brightness saturation, but artifacts occur. If the first output image 245 during training is a residual image, the first estimated image 222 becomes a residual image whose blur is sharpened by being added to the captured image 221. Therefore, the first estimated image 222 is an image obtained by estimating the sharpening component of the blur of the captured image 221.

[0105] The calculation unit 126 inputs the first estimated image 222 and the brightness saturation map into the second machine learning model 232 to generate a second estimated image 223. The second estimated image 223 is an image in which artifacts present around regions of the first estimated image 222 corresponding to brightness-saturated regions of the captured image 221 are suppressed. If the third output image 257 is a residual image to be added to the second blurred image 253 (or the second output image 254) during training, the second estimated image 223 becomes a residual image to be added to the captured image 221 (or the first estimated image 222). By adding the residual image to the captured image 221, an image in which artifacts around brightness-saturated regions are suppressed and blur is sharpened can be obtained.

[0106] If the second feature map is input to the second machine learning model 232 during training of the second machine learning model 232, then during estimation after training, the third feature map generated by inputting the captured image 221 to the first machine learning model 231 is also input to the second machine learning model 232. The third feature map is a feature map, and is a map generated at a position corresponding to the first feature map 248 by inputting the captured image 221 to the first machine learning model 231.

[0107] In step S405, the calculation unit 126 generates a sharpened image by taking a weighted average (weighted synthesis) of the captured image 221 and the second estimated image 223. In the second estimated image 223, the sharpening effect may appear discontinuous in high-brightness gradation areas that include brightness saturation and where the signal value changes gradually. By taking a weighted average with the captured image 221, this discontinuity can be suppressed.

[0108] Furthermore, the calculation unit 126 generates a second weight map indicating a weight for each pixel of the captured image 221 based on the magnitude of the signal value of the captured image 221 (or the second estimated image 223). At this time, the second weight map is generated so that the weight of the captured image 221 increases as the signal value of the pixel of the captured image 221 increases.

[0109] Since the captured image 221 is a RAW image with a Bayer array, the calculation unit 126 performs white balance adjustment after subtracting a black level from the signal value. Furthermore, the calculation unit 126 rearranges the data into four channels using Depth to Space, and generates a one-channel map using the maximum value in the channel (color) direction. Because the number of elements in the vertical and horizontal directions of this map is half that of the captured image 221, it is upscaled to the same number of elements using bilinear interpolation or the like. The weight of each pixel of the captured image 221 is determined from the signal value of the upscaled map. The weight of each pixel of the captured image 221 is calculated, for example, using the signal value-to-weight conversion shown in FIG. 14. In FIG. 14, the two-dot chain line indicates the luminance saturation value. As indicated by the solid or dashed line, the weight increases as the signal value increases. However, the method of determining the weight is not limited to this.

[0110] The calculation unit 126 uses the second weighting map determined based on the upscaled map to perform a weighted average of the captured image 221 and the second estimated image 223 to generate a final sharpened image. If the second estimated image 223 is a residual image, a similar weighted combination can be performed by subtracting the second weighting map from an all-ones map to obtain a weighting map, multiplying the weighting map by the second estimated image 223, and adding the product to the captured image 221. Note that the second estimated image 223 may be used as the final sharpened image without performing step S405.

[0111] Through the above processing, it is possible to reduce the computational load of the machine learning model, suppress artifacts around brightness saturated regions, and sharpen blur in the captured image.

[0112] Next, a more desirable configuration will be described.

[0113] It is desirable that the blur specification information in this embodiment is not input to the second machine learning model 232. The reason for this will be explained below.

[0114] Because it is difficult to accurately separate the subject from the blur from an image that includes only the blur, it is desirable to input blur specification information when sharpening various blurs that occur in the optical system 121. When there is a gradual edge in an image, it is extremely difficult to distinguish whether the gradual edge is the result of a strong edge with a large blur caused by blurring, or whether the subject with gradation is the result of a very small blur caused by blurring. For this reason, by providing information about the blur occurring at the edge position to the first machine learning model 231 as blur specification information, the blur can be sharpened with high accuracy.

[0115] In contrast, the task of the second machine learning model 232 is to suppress artifacts around brightness saturated regions. These artifacts are relatively easy to identify from an image, so they can be suppressed with sufficient accuracy without inputting blur specification information. On the other hand, if blur specification information is input to the second machine learning model 232, the second machine learning model 232 will attempt to execute different processes depending on the value of the blur specification information. For this reason, a certain degree of freedom is required to ensure the accuracy of the task, which increases the computational load of the second machine learning model 232.

[0116] Furthermore, it is desirable that the number of elements in each of the vertical and horizontal directions in the multiple feature maps generated in the first machine learning model 231 is greater than half the number of elements in each of the vertical and horizontal directions of the first blurred image 243.

[0117] The above is the configuration during training of the first machine learning model 231, and the same content during estimation after training is as follows: The number of elements in each of the vertical and horizontal directions in the multiple feature maps generated in the first machine learning model 231 is desirably greater than half the number of elements in each of the vertical and horizontal directions in the captured image 221. Alternatively, the number of elements in each of the vertical and horizontal directions in the multiple feature maps generated in the first machine learning model 231 is desirably greater than half the number of elements in each of the vertical and horizontal directions in the first output image 245.

[0118] The above is the configuration during training of the first machine learning model 231, and the same content during estimation after training is as follows: It is desirable that the number of elements in each of the vertical and horizontal directions in the multiple feature maps generated in the first machine learning model 231 is greater than half the number of elements in each of the vertical and horizontal directions of the first estimated image 222.

[0119] The reasons why the above two configurations are desirable are explained below. The first machine learning model 231 does not perform uniform processing, but is preferably capable of performing sharpening in accordance with various shapes of blur, and therefore requires a certain degree of flexibility. On the other hand, one method for reducing the computational load of the machine learning model is to temporarily downscale the number of elements in each of the vertical and horizontal directions within the machine learning model to half or less, and then upscale it back to the original number of elements. For example, consider the configuration shown in FIG. 11.

[0120] In the figure, "Space to Depth" corresponds to downscaling, and "Depth to Space" corresponds to upscaling. Downscaling may also be performed by pooling or by increasing the stride of the convolutional layer to greater than 1. Similarly, upscaling may also be performed by bilinear interpolation, bicubic interpolation, or by increasing the stride of the deconvolutional layer to greater than 1. If a configuration in which downscaling and upscaling are performed within a model like this is adopted when the first machine learning model 231 does not have sufficient degrees of freedom, artifacts are likely to occur in the first output image 245 (or the second output image 254 or the first estimated image 222).

[0121] For example, in the configuration of Figure 11, if the degrees of freedom of the machine learning model are insufficient for the task to be performed, variations occur in each channel of the feature map, which are then rearranged in the spatial direction using Depth to Space. As a result, grid-like artifacts may occur within the same color. For this reason, it is desirable that the first machine learning model 231 not be configured to perform downscaling and upscaling within the model.

[0122] Note that, for example, if zero padding is not performed in the convolution layer, the number of elements in each of the vertical and horizontal directions changes. Therefore, the number of elements in each of the vertical and horizontal directions of the feature map in the first machine learning model 231 does not necessarily match the input and output even without downscaling or upscaling.

[0123] Furthermore, it is desirable that the number of elements in the vertical and horizontal directions in at least one of the multiple feature maps generated within the second machine learning model 232 be less than half the number of elements in the vertical and horizontal directions in the second output image 254.

[0124] The above is the configuration during training of the second machine learning model 232, and the same content during estimation after training is as follows: It is desirable that the number of elements in the vertical and horizontal directions in at least one of the multiple feature maps generated within the second machine learning model 232 is equal to or less than half the number of elements in the vertical and horizontal directions in the first estimated image 222. This is because the task performed by the second machine learning model 232 is to suppress artifacts that make it easy to distinguish from the subject in the input image, and therefore an increase in the computational load is unlikely even if downscaling and upscaling are performed within the above-mentioned model.

[0125] It is also desirable that a brightness saturation map is not input to the first machine learning model 231. This is also true when training the first machine learning model 231, because in order to reduce the computational load, it is desirable for the first machine learning model 231 to execute processing that is independent of brightness saturation. By inputting a brightness saturation map, the first machine learning model 231 executes different processing depending on whether brightness saturation is present or absent, which may increase the required degree of freedom.

[0126] With the above configuration, it is possible to provide an image processing system that can suppress artifacts around brightness saturated areas while reducing the computational load when sharpening blurred images using a machine learning model.

[0127] [Example 2] Fig. 15 shows the configuration of an image processing system 300 in Example 2. Fig. 16 shows the appearance of the image processing system 300. The image processing system 300 includes a training device 301, a sharpening device 302, an imaging device 303, and a lens device 304.

[0128] The imaging device 303 is an interchangeable lens camera, and multiple types of lens devices 304 can be connected. The subject image formed by the lens device 304 is received by an imaging element 331. The imaging element 331 generates a captured image as image data by photoelectrically converting the subject image. The captured image contains blur due to aberrations and diffraction occurring in the optical system 341. The captured image is subjected to necessary processing such as removal of defective pixels by a calculation unit 334, and then stored in a storage unit 332.

[0129] In addition, the imaging device 303 acquires information regarding the type of lens device 304 and the state of the optical system 341 (focal length, F-number, focus distance) at the time of capturing the captured image via the communication unit 343 and the communication unit 333, and writes this information into the metadata of the captured image.

[0130] The sharpening device 302 sharpens blur in the captured image using the first and second machine learning models to generate a sharpened image. The model parameters of the first and second machine learning models are trained by the training device 301, and are acquired in advance by the sharpening device 302 via the communication units 312 and 322 and stored in the storage unit 321. The sharpened image undergoes other necessary processing in the calculation unit 324, is stored in the storage unit 321 or the storage unit 332, and is displayed on the display unit 325 or the display unit 335.

[0131] In this embodiment, the training of the first and second machine learning models and the sharpening using the trained model parameters are the same as in the first embodiment.

[0132] The first training data set used to train the first machine learning model is generated by the training device 301 according to the flowchart of Fig. 5. The processing of each step in Fig. 5 is executed by a corresponding component among the components of the training device 301: the storage unit 311, the communication unit 312, the acquisition unit 313, the calculation unit 314, the determination unit 315, and the update unit 316. Since multiple types of lens devices 304 can be connected to the imaging device 303, a separate first training data set is generated for each type of lens device 304.

[0133] Similarly, the training device 301 trains the first machine learning model (determines model parameters) using the first training data set according to the flowchart of Fig. 7. The model parameters of the first machine learning model are updated for each type of lens device 304. Note that although the configurations shown in Figs. 8 and 11 are used for the first and second machine learning models, different configurations may be adopted for each type of lens device 304.

[0134] The second training data set used to train the second machine learning model is generated by the training device 301 for each type of lens device 304 according to the flowchart of Fig. 9. The training device 301 uses the second training data set corresponding to each type of lens device 304 to train the second machine learning model according to the flowchart of Fig. 7.

[0135] The sharpening device 302 uses the trained first and second machine learning models to sharpen blur caused by aberration and diffraction of the optical system 341 in a captured image. The sharpening device 302 includes, as components, a storage unit 321, a communication unit 322, an acquisition unit 323, a calculation unit 324, and a display unit 325. The sharpening of blur is performed according to the flowchart in FIG. 13, and the processing of each step is executed by the corresponding component of the sharpening device 302.

[0136] In step S402 , the acquisition unit 323 identifies the type of lens device 304 used when capturing an image from the metadata of the captured image, and acquires model parameters corresponding to the lens device 304 from the storage unit 321 .

[0137] With the above configuration, it is possible to provide an image processing system that can suppress artifacts around brightness saturated areas while reducing the computational load when sharpening blurred images using a machine learning model.

[0138] The above embodiments include the following methods.

[0139] (Method 1) 1. An image processing method for training a first machine learning model and a second machine learning model, comprising: a first step of acquiring a first original image having a signal value equal to or less than a first signal value corresponding to a first upper limit that is an upper limit of the signal values ​​of the image input to the first machine learning model; a second step of blurring the first original image to generate a first blurred image; a third step of acquiring a first target image having sharper sharpness than the first blurred image based on the first original image; a fourth step of inputting the first blurred image into the first machine learning model to generate a first output image; a fifth step of training the first machine learning model based on the first output image and the first ground truth image; a sixth step of acquiring a second original image having a signal value greater than a second signal value corresponding to the first upper limit; a seventh step of blurring the second original image and then clipping it at a third signal value corresponding to the first upper limit to generate a second blurred image; an eighth step of generating a second ground truth image based on the second original image; a ninth step of inputting the second blurred image into the first machine learning model to generate a model output, and inputting the model output into the second machine learning model to generate a third output image; and a tenth step of training the second machine learning model based on the third output image and the second ground truth image; The image processing method according to the present invention, wherein the second correct image has different sharpness relative to the second blurred image based on the magnitude of the signal value of the second blurred image. (Method 2) The image processing method according to Method 1, characterized in that in the fourth step, the first blurred image is scaled so that an upper limit of the signal value in the first blurred image is the same value as the first upper limit, and then input to the first machine learning model. (Method 3) 3. The image processing method according to method 1 or 2, wherein in the ninth step, the second blurred image is scaled so that an upper limit of the signal value in the second blurred image is the same as the first upper limit, and then input to the first machine learning model. (Method 4) a difference in sharpness of a first region of the second blurred image relative to a region corresponding to the first region of the second correct image is smaller than a difference in sharpness of a second region of the second blurred image relative to a region corresponding to the second region of the second correct image; 4. The image processing method according to any one of methods 1 to 3, wherein the first region of the second blurred image has a larger average signal value than the second region of the second blurred image. (Method 5) a difference in sharpness of a first region of the second blurred image relative to a region corresponding to the first region of the second correct image is smaller than a difference in sharpness of a second region of the second blurred image relative to a region corresponding to the second region of the second correct image; 5. An image processing method according to any one of methods 1 to 4, wherein the first region of the second blurred image includes brightness-saturated pixels, and the second region of the second blurred image does not include brightness-saturated pixels. (Method 6) In the fourth step, blur specifying information specifying the blur of the first blurred image is further input to the first machine learning model; An image processing method described in any one of methods 1 to 5, characterized in that in the ninth step, blur identification information that identifies the blur of the second blurred image is further input to the first machine learning model. (Method 7) 7. The image processing method according to Method 6, wherein the blur specification information is information including at least one of the state of the optical system corresponding to the blur and image plane coordinates. (Method 8) 8. The image processing method according to method 6 or 7, wherein the blur-specific information is not input to the second machine learning model. (Method 9) generating a plurality of feature maps within the first machine learning model; An image processing method described in any one of methods 1 to 8, characterized in that the number of elements in each of the vertical and horizontal directions of all of the multiple feature maps is greater than half the number of elements in each of the vertical and horizontal directions of the first blurred image. (Method 10) generating a plurality of feature maps within the first machine learning model; An image processing method described in any one of methods 1 to 9, characterized in that the number of elements in each of the vertical and horizontal directions of all of the multiple feature maps is greater than half the number of elements in each of the vertical and horizontal directions of the first output image. (Method 11) The model output is: a second output image; and and a second feature map corresponding to a first feature map generated together with the first output image by inputting the first blurred image into the first machine learning model. (Method 12) generating a plurality of feature maps within the second machine learning model; The image processing method described in Method 11, characterized in that the number of elements in the vertical and horizontal directions of at least one of the plurality of feature maps is less than or equal to half the number of elements in the vertical and horizontal directions of the second output image. (Method 13) 13. The image processing method according to any one of Methods 1 to 12, wherein the first blurred image and the first ground truth image are generated by adding noises that are correlated with each other. (Method 14) 14. The image processing method according to any one of Methods 1 to 13, wherein the second blurred image and the second ground truth image are generated by adding noises that are correlated with each other. (Method 15) inputting a brightness saturation map indicating a brightness saturation region of either the second blurred image or the second output image into the second machine learning model; 15. The image processing method according to any one of methods 1 to 14, wherein the second output image is generated by inputting the second blurred image into the first machine learning model. (Method 16) the second ground truth image is generated based on the second blurred image and a third ground truth image having sharper sharpness than the second blurred image, using a first weight map generated by blurring a brightness saturation map indicating a brightness saturation region of either the second blurred image or the second output image; 16. An image processing method according to any one of methods 1 to 15, wherein the second output image is generated by inputting the second blurred image into the first machine learning model. (Method 18) an eleventh step of generating a first estimated image in which a sharpening component of blur in the captured image is estimated by inputting the captured image generated by the imaging device into a first machine learning model that executes processing that is not dependent on luminance saturation of the captured image; and a twelfth step of generating a second estimated image in which artifacts present around a region of the first estimated image corresponding to a brightness saturated region of the captured image are suppressed by inputting the first estimated image to a second machine learning model. (Method 19) The image processing method described in Method 18, characterized in that in the 12th step, a brightness saturation map indicating brightness saturation regions of either the captured image or the first estimated image is further input to the second machine learning model. (Method 20) 20. The image processing method of claim 19, wherein the brightness saturation map is not input to the first machine learning model. (Method 21) An image processing method described in any one of methods 18 to 20, characterized in that in the 11th step, blur identification information that identifies blur in the captured image is further input into the first machine learning model. (Method 22) The image processing method described in Method 21, characterized in that the blur specification information includes information regarding at least one of the state of the optical system of the imaging device at the time of imaging, the spatial coordinates of multiple pixels in the captured image, and the pixel pitch of the imaging element of the imaging device. (Method 23) 23. The image processing method according to method 21 or 22, wherein the blur-specific information is not input to the second machine learning model. (Method 24) generating a plurality of feature maps within the first machine learning model; An image processing method described in any one of methods 18 to 23, characterized in that the number of elements in each of the vertical and horizontal directions of all of the multiple feature maps is greater than half the number of elements in each of the vertical and horizontal directions of the captured image. (Method 25) generating a plurality of feature maps within the first machine learning model; An image processing method described in any one of methods 18 to 24, characterized in that the number of elements in each of the vertical and horizontal directions of all of the multiple feature maps is greater than half the number of elements in each of the vertical and horizontal directions of the first estimated image. (Method 26) generating a plurality of feature maps within the second machine learning model; An image processing method described in any one of methods 18 to 25, characterized in that the number of elements in the vertical and horizontal directions of at least one of the plurality of feature maps is less than or equal to half the number of elements in the vertical and horizontal directions of the first estimated image. (Method 27) 27. The image processing method according to any one of methods 18 to 26, further comprising a step of performing weighted synthesis of the captured image and the second estimated image using a second weight map for the captured image generated based on the magnitude of the signal value of either the captured image or the second estimated image.

[0140] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0141] The embodiments described above are merely representative examples, and various modifications and alterations are possible to each embodiment when implementing the present invention. [Explanation of symbols]

[0142] 100 Image Processing System 101 Training equipment 102 Imaging device 126 Arithmetic section 221 Captured Images 222 First Estimated Image 223 Second Estimated Image 231 First Machine Learning Model 232 Second Machine Learning Model 241 First original image 242,252 PSF 243 First blurred image 244 Dementia specific information 245 First output image 251 Second Original Image 253 Second blurred image 254 Second Output Image 255 Second correct image 257 Third Output Image 300 Image Processing System 301 Training equipment 302 Sharpening device 303 Imaging Device 304 Lens device

Claims

1. 1. An image processing method for training a first machine learning model and a second machine learning model, comprising: a first step of acquiring a first original image having a signal value equal to or less than a first signal value corresponding to a first upper limit that is an upper limit of the signal values ​​of the image input to the first machine learning model; a second step of blurring the first original image to generate a first blurred image; a third step of acquiring a first target image having sharper sharpness than the first blurred image based on the first original image; a fourth step of inputting the first blurred image into the first machine learning model to generate a first output image; a fifth step of training the first machine learning model based on the first output image and the first ground truth image; a sixth step of acquiring a second original image having a signal value greater than a second signal value corresponding to the first upper limit; a seventh step of blurring the second original image and then clipping it at a third signal value corresponding to the first upper limit to generate a second blurred image; an eighth step of generating a second ground truth image based on the second original image; a ninth step of inputting the second blurred image into the first machine learning model to generate a model output, and inputting the model output into the second machine learning model to generate a third output image; a tenth step of training the second machine learning model based on the third output image and the second ground truth image; The image processing method according to claim 1, wherein the second correct image has a different sharpness relative to the second blurred image based on the magnitude of the signal value of the second blurred image.

2. In the fourth step, 2. The image processing method according to claim 1, wherein the first blurred image is scaled so that an upper limit of the signal value in the first blurred image is the same as the first upper limit, and then input to the first machine learning model.

3. In the ninth step, 2. The image processing method according to claim 1, wherein the second blurred image is scaled so that an upper limit of the signal value in the second blurred image is the same as the first upper limit, and then input to the first machine learning model.

4. a difference in sharpness between a first region of the second blurred image and a region of the second correct image corresponding to the first region is smaller than a difference in sharpness between a second region of the second blurred image and a region of the second correct image corresponding to the second region; 2. The image processing method according to claim 1, wherein the first region of the second blurred image has a larger average signal value than the second region of the second blurred image.

5. a difference in sharpness between a first region of the second blurred image and a region of the second correct image corresponding to the first region is smaller than a difference in sharpness between a second region of the second blurred image and a region of the second correct image corresponding to the second region; 2. The image processing method according to claim 1, wherein the first region of the second blurred image includes brightness-saturated pixels, and the second region of the second blurred image does not include brightness-saturated pixels.

6. In the fourth step, blur specifying information specifying the blur of the first blurred image is further input to the first machine learning model; The image processing method according to claim 1 , wherein in the ninth step, blur specification information that specifies the blur of the second blurred image is further input to the first machine learning model.

7. 7. The image processing method according to claim 6, wherein the blur specification information is information including at least one of an optical system state corresponding to the blur and image plane coordinates.

8. The image processing method according to claim 6 , wherein the blur-specific information is not input to the second machine learning model.

9. A plurality of feature maps is generated within the first machine learning model; 2. The image processing method according to claim 1, wherein the number of elements in each of the vertical and horizontal directions of all of the plurality of feature maps is greater than half the number of elements in each of the vertical and horizontal directions of the first blurred image.

10. A plurality of feature maps is generated within the first machine learning model; 2. The image processing method according to claim 1, wherein the number of elements in each of the vertical and horizontal directions of all of the plurality of feature maps is greater than half the number of elements in each of the vertical and horizontal directions of the first output image.

11. The model output is: a second output image; and and a second feature map corresponding to a first feature map generated together with the first output image by inputting the first blurred image into the first machine learning model.

12. a plurality of feature maps are generated within the second machine learning model; 12. The image processing method according to claim 11, wherein the number of elements in the vertical and horizontal directions of at least one of the plurality of feature maps is less than or equal to half the number of elements in the vertical and horizontal directions of the second output image.

13. 2. The image processing method according to claim 1, wherein the first blurred image and the first correct image are generated by adding noises that are correlated with each other.

14. 2. The image processing method according to claim 1, wherein the second blurred image and the second correct image are generated by adding noises that are correlated with each other.

15. a brightness saturation map indicating a brightness saturation region of either the second blurred image or the second output image is input to the second machine learning model; The image processing method according to claim 1 , wherein the second output image is generated by inputting the second blurred image into the first machine learning model.

16. the second ground truth image is generated based on the second blurred image and a third ground truth image having sharper sharpness than the second blurred image, using a first weight map generated by blurring a brightness saturation map indicating a brightness saturation region of either the second blurred image or the second output image; The image processing method according to claim 1 , wherein the second output image is generated by inputting the second blurred image into the first machine learning model.

17. 1. An image processing apparatus for training a first machine learning model and a second machine learning model, comprising: a means for acquiring a first original image having a signal value equal to or less than a first signal value corresponding to a first upper limit that is an upper limit of the signal value of the image input to the first machine learning model; means for generating a first blurred image by blurring the first original image; a means for acquiring a first correct image having sharper sharpness than the first blurred image based on the first original image; means for generating a first output image by inputting the first blurred image into the first machine learning model; means for training the first machine learning model based on the first output image and the first ground truth image; means for acquiring a second original image having a signal value greater than a second signal value corresponding to the first upper limit; means for generating a second blurred image by blurring the second original image and then clipping the second original image at a third signal value corresponding to the first upper limit; means for generating a second ground truth image based on the second original image; means for generating a model output by inputting the second blurred image into the first machine learning model, and for generating a third output image by inputting the model output into the second machine learning model; and means for training the second machine learning model based on the third output image and the second ground truth image; The image processing device according to claim 1, wherein the second correct image has different sharpness relative to the second blurred image based on the magnitude of the signal value of the second blurred image.

18. an eleventh step of generating a first estimated image in which a sharpening component of blur in the captured image is estimated by inputting the captured image generated by the imaging device performing imaging to a first machine learning model that executes processing that is not dependent on luminance saturation of the captured image; and a twelfth step of generating a second estimated image in which artifacts present around a region of the first estimated image corresponding to a brightness saturated region of the captured image are suppressed by inputting the first estimated image into a second machine learning model.

19. 19. The image processing method according to claim 18, wherein in the twelfth step, a brightness saturation map indicating a brightness saturation region of either the captured image or the first estimated image is further input to the second machine learning model.

20. 20. The image processing method of claim 19, wherein the intensity saturation map is not input to the first machine learning model.

21. 19. The image processing method according to claim 18, wherein in the eleventh step, blur specifying information specifying blur in the captured image is further input to the first machine learning model.

22. 22. The image processing method according to claim 21, wherein the blur specification information includes information regarding at least one of a state of the optical system of the imaging device at the time of capturing the image, spatial coordinates of a plurality of pixels in the captured image, and a pixel pitch of an imaging element of the imaging device.

23. 22. The image processing method of claim 21, wherein the blur-specific information is not input to the second machine learning model.

24. A plurality of feature maps is generated within the first machine learning model; 19. The image processing method according to claim 18, wherein the number of elements in each of the vertical and horizontal directions of all of the plurality of feature maps is greater than half the number of elements in each of the vertical and horizontal directions of the captured image.

25. A plurality of feature maps is generated within the first machine learning model; 19. The image processing method according to claim 18, wherein the number of elements in each of the vertical and horizontal directions of all of the plurality of feature maps is greater than half the number of elements in each of the vertical and horizontal directions of the first estimated image.

26. a plurality of feature maps are generated within the second machine learning model; 19. The image processing method according to claim 18, wherein the number of elements in the vertical and horizontal directions of at least one of the plurality of feature maps is less than or equal to half the number of elements in the vertical and horizontal directions of the first estimated image.

27. 19. The image processing method according to claim 18, further comprising a step of performing weighted synthesis of the captured image and the second estimated image using a second weight map for the captured image generated based on the magnitude of a signal value of either the captured image or the second estimated image.

28. a means for generating a first estimated image in which a sharpening component of blur in the captured image is estimated by inputting the captured image generated by the imaging device performing imaging to a first machine learning model that executes processing that is not dependent on luminance saturation of the captured image; and and means for generating a second estimated image in which artifacts present around a region of the first estimated image corresponding to a brightness saturated region of the captured image are suppressed by inputting the first estimated image to a second machine learning model.

29. A program causing a computer to execute a process according to the image processing method of any one of claims 1 to 16.

30. A program causing a computer to execute a process according to the image processing method of any one of claims 18 to 27.

Images