Image processing method, image processing device, image processing system, and program

Abstract

To provide an image processing method capable of correcting blur appropriately according to the brightness and scene of an image. [Solution] The image processing method includes the steps of acquiring a captured image obtained through an imaging optical system, correcting blur components in the captured image to generate a first image, and generating a second image based on the captured image, the first image, and a weight map, where the weight map is generated based on information regarding the brightness of the captured image or information regarding the scene in the captured image, and information based on saturated areas in the captured image.

Description

[Technical Field]

[0001] The present invention relates to an image processing method for reducing image blur. [Background technology]

[0002] Patent Document 1 discloses a method for sharpening blur in a captured image using a convolutional neural network (CNN), which is one of the machine learning models. A training data set is generated by blurring an image having a signal value equal to or greater than the brightness saturation value of the captured image, and the CNN is trained using the training data set, thereby suppressing adverse effects even around brightness-saturated regions and performing blur sharpening. The document also discloses a method for adjusting the strength of sharpening by performing a weighted average of the captured image and an estimated image (blur-sharpened image) based on the brightness-saturated regions.

[0003] Patent Document 2 discloses a method for reducing stripe artifacts called ringing around positions corresponding to saturated pixels by correcting blur in a captured image using deconvolution and synthesizing the resulting corrected image with the captured image. In Patent Document 2, the weighting used during synthesis is such that the synthesis ratio of the captured image is set to 1 for saturated pixels and 0 for other pixels. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2020-166628 [Patent Document 2] Japanese Patent Application Publication No. 2018-201137 Summary of the Invention [Problem to be solved by the invention]

[0005] The method disclosed in Patent Document 1 cannot suppress darkening and ringing depending on the input image, and these problems may occur in the estimated image (blur-sharpened image). Specifically, these problems are likely to occur when the subject is significantly blurred due to aberrations in the optical system. Furthermore, the degree to which darkening that occurs around brightness-saturated areas is noticeable varies depending on the brightness of the image. For example, in bright images taken outdoors during the day, darkening around saturated areas is noticeable, but in dark images such as night scenes, darkening is not noticeable.

[0006] Furthermore, in the methods disclosed in Patent Documents 1 and 2, averaging is performed according to a predetermined weight, regardless of the conspicuousness of dark spots, which differs depending on the brightness (or scene) of the image. In other words, when a weighted average of an input image and an estimated image is performed, the conspicuous dark spots around saturated areas are reduced in bright images (bright scenes), but the correction effect around saturated areas is reduced more than necessary in dark images (dark scenes).

[0007] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing method, an image processing device, an image processing system, and a program that are capable of performing appropriate blur correction according to the brightness and scene of an image. [Means for solving the problem]

[0008] An image processing method according to one aspect of the present invention includes: Statue generating a first image by correcting the blur component; 、 the first image; 、 Saturation in the captured image region and generating a second image based on information about the brightness of the captured image or information about the scene of the captured image, region The information about is generated by inputting the captured image into a machine learning model. information representing a blurred area around a brightness saturated area of ​​the captured image; .

[0009] Other objects and features of the present invention are illustrated in the following examples. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide an image processing method, an image processing device, an image processing system, and a program that are capable of performing appropriate blur correction according to the brightness and scene of an image. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a configuration diagram of a machine learning model in a first embodiment. [Figure 2] 1 is a block diagram of an image processing system according to a first embodiment. [Figure 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4] 10A to 10C are explanatory diagrams of adverse effects caused by sharpening in the first to fourth embodiments. [Figure 5] 1 is a flowchart of training a machine learning model in Examples 1, 3, and 4. [Figure 6] 10 is a flowchart of generating a model output in the first or third embodiment. [Figure 7] 10 is a flowchart of adjusting the strength of sharpening in the first to fourth embodiments. [Figure 8] FIG. 10 is an explanatory diagram of adjustment values ​​of weight maps for average signal values ​​in the first to fourth embodiments. [Figure 9] 4A and 4B are explanatory diagrams of a captured image and a saturation influence map in the first embodiment. [Figure 10] 4A and 4B are explanatory diagrams of a captured image and a saturation influence map in the first embodiment. [Figure 11] FIG. 3 is an explanatory diagram of a weight map in the first embodiment. [Figure 12] FIG. 3 is an explanatory diagram of a weight map in the first embodiment. [Figure 13] FIG. 10 is an explanatory diagram of an image restoration filter in the second embodiment. [Figure 14] 10 is an explanatory diagram (cross-sectional view) of an image restoration filter in the second embodiment. FIG. [Figure 15] FIG. 10 is an explanatory diagram of a point spread function PSF in the second embodiment. [Figure 16] 10 is an explanatory diagram of an amplitude component MTF and a phase component PTF of an optical transfer function in Example 2. FIG. [Figure 17] FIG. 10 is a block diagram of an image processing system according to a second embodiment. [Figure 18] 10 is a flowchart of generating a blur-sharpened image in the second embodiment. [Figure 19] FIG. 10 is a block diagram of an image processing system according to a third embodiment. [Figure 20] FIG. 11 is an external view of an image processing system according to a third embodiment. [Figure 21] FIG. 10 is a block diagram of an image processing system according to a fourth embodiment. [Figure 22] FIG. 10 is an external view of an image processing system according to a fourth embodiment. [Figure 23] 13 is a flowchart of model output and sharpening strength adjustment in the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and redundant explanations will be omitted.

[0013] Before describing the specific embodiments, the gist of the present invention will be described. The present invention generates an estimated image (blur-sharpened image, first image) from a captured image captured using an optical system (image capture optical system) by sharpening blur caused by the optical system. The estimated image is generated using, for example, a machine learning model. A weight map (weight information) is generated based on information about the brightness of the captured image or information about the scene of the captured image, and information (information about the saturated region) based on a saturated region (brightness saturated region) in the captured image, and a weighted average is taken of the captured image and the estimated image. Here, blur caused by the optical system includes blur due to aberration, diffraction, and defocus, the effect of an optical low-pass filter, and pixel aperture degradation of an image sensor.

[0014] Machine learning models include, for example, neural networks, genetic programming, Bayesian networks, etc. Neural networks include convolutional neural networks (CNNs), generative adversarial networks (GANs), recurrent neural networks (RNNs), etc.

[0015] Blur sharpening refers to the process of restoring frequency components of a subject that have been reduced or lost due to blur. When blur sharpening is performed, undershoot (darkness) and ringing may not be suppressed depending on the captured image, resulting in these problems in the estimated image. Specifically, these problems occur when the subject is significantly blurred due to optical system aberrations or when there are saturated brightness areas in the image. Brightness saturated areas can occur in an image depending on the dynamic range of the image sensor and the exposure during capture. In saturated brightness areas, information about the structure of the subject's space cannot be obtained, making it more likely that problems will occur. Furthermore, the degree to which undershoots that occur around saturated brightness areas are noticeable varies depending on the brightness of the image. For example, in bright images captured outdoors during the day, undershoots around saturated brightness areas are noticeable, but in dark images such as night scenes, undershoots are less noticeable.

[0016] Therefore, in each embodiment, a weighted average of the captured image and the estimated image is calculated using a weight map generated based on information about the brightness of the captured image or information about the scene of the captured image and information based on the saturated region in the captured image, thereby making it possible to maintain the blur correction effect while suppressing adverse effects around the saturated region that differ depending on the brightness and scene of the captured image.

[0017] In the following, the stage of learning the weights of the machine learning model will be referred to as the learning phase, and the stage of sharpening blur using the machine learning model using the learned weights will be referred to as the estimation phase. [Example]

[0018] First, an image processing system 100 according to a first embodiment of the present invention will be described. In this embodiment, a machine learning model is used to sharpen blur in a captured image that includes saturation in brightness. The blur to be sharpened is targeted at aberrations and diffraction occurring in the optical system, and blur caused by an optical low-pass filter. However, the effects of the present invention can be similarly achieved when sharpening blur caused by pixel aperture, defocus, or shaking. The present invention can also be similarly implemented to achieve the effects of tasks other than blur sharpening.

[0019] FIG. 2 is a block diagram of the image processing system 100. FIG. 3 is an external view of the image processing system 100. The image processing system 100 has a training device 101 and an image processing device 103, which are connected via a wired or wireless network. An imaging device 102, a display device 104, a recording medium 105, and an output device 106 are connected to the image processing device 103 via wired or wireless connections. A captured image of a subject space captured using the imaging device 102 is input to the image processing device 103. Blurring occurs in the captured image due to aberration and diffraction caused by the optical system (imaging optical system) 102a in the imaging device 102 and the optical low-pass filter of the image sensor 102b, and information about the subject is attenuated.

[0020] The image processing device 103 performs blur sharpening on the captured image using a machine learning model to generate a saturation influence map and a blur-sharpened image (model output, first image). Details of the saturation influence map will be described later. The machine learning model is trained by the training device 101, and the image processing device 103 acquires information about the machine learning model from the training device 101 in advance and stores it in the storage unit 103a. The image processing device 103 also has a function of adjusting the intensity of blur sharpening by taking a weighted average of the captured image and the blur-sharpened image. Details of training and estimation of the machine learning model and adjusting the intensity of blur sharpening will be described later. A user can adjust the intensity of blur sharpening while checking the image displayed on the display device 104. The blur-sharpened image after intensity adjustment is stored in the storage unit 103a or the recording medium 105 and, as necessary, output to an output device 106 such as a printer. The captured image may be grayscale or have multiple color components. Furthermore, it may be an undeveloped RAW image or a developed image.

[0021] Next, with reference to Figures 4(A) to (C), a description will be given of the degradation of estimation accuracy that occurs when blur sharpening is performed using a machine learning model. Figures 4(A) to (C) are explanatory diagrams of the adverse effects of sharpening, showing spatial changes in image signal values. Here, since the image is an 8-bit image, the saturation value is 255. The solid lines in Figures 4(A) to (C) are captured images (blurred images), and the dotted lines are blur-sharpened images in which the blur of the captured images has been sharpened using a machine learning model.

[0022] Figure 4(A) shows the results of sharpening a non-saturated object with significant blur due to optical system aberrations. Figure 4(B) shows a non-saturated object with minimal blur due to optical system aberrations. Figure 4(C) shows the results of sharpening a saturated object with minimal blur due to optical system aberrations. When the blur due to optical system aberrations is significant, undershoot occurs on the dark side of the edge. Furthermore, even when the blur due to optical system aberrations is minimal, sharpening a saturated object results in undershoots that did not occur in non-saturated objects and a decrease in pixel values ​​that were originally saturated. In saturated areas, information about the structure of the object space is lost, and false edges can appear at the boundaries between areas, making it impossible to extract accurate feature values ​​for the object. This reduces the estimation accuracy of the machine learning model. These results demonstrate that the adverse effects of sharpening depend on the performance of the optical system and the saturated area.

[0023] The aforementioned correction uses a machine learning model that incorporates a method of using a captured image and a brightness saturation map corresponding to the captured image as input data for the machine learning model, and a method of generating a saturation influence map. In other words, while it is possible to reduce adverse effects by using these methods, it is difficult to completely eliminate them. The method of using a brightness saturation map and the method of generating a saturation influence map will be described in detail below.

[0024] We will now explain the brightness saturation map. A brightness saturation map is a map that represents brightness saturated areas in a captured image. In areas where brightness saturation occurs (brightness saturated areas), information about the structure of the subject space is lost, and false edges may appear at the boundaries of each area, making it impossible to extract correct feature quantities for the subject. Therefore, by inputting a brightness saturation map, the neural network can identify problematic areas such as those mentioned above, thereby preventing a decrease in estimation accuracy.

[0025] Next, the saturation influence map will be described. Even when a brightness saturation map is used, the machine learning model may not make an accurate judgment. For example, if a region of interest is near a brightness saturated region, the machine learning model can determine that the region of interest is affected by brightness saturation because there is a brightness saturated region near the region of interest. However, if the region of interest is located far from the brightness saturated region, it is not easy to determine whether the region is affected by brightness saturation, and ambiguity increases. As a result, the machine learning model may make an incorrect judgment at a position far from the brightness saturated region. For this reason, when the task is blur sharpening, a sharpening process corresponding to a saturated blur image is performed on a non-saturated blur image. In this case, artifacts occur in the image with the sharpened blur, reducing the accuracy of the task. Therefore, it is preferable to use a machine learning model to generate a saturation influence map from a captured image in which blur occurs.

[0026] A saturation influence map is a map (a spatially arranged signal sequence) that represents the magnitude and range of signal values ​​that have spread due to blurring of subjects in brightness-saturated areas of a captured image. In other words, a saturation influence map is information based on saturated areas in a captured image. By having a machine learning model generate a saturation influence map, the machine learning model can accurately estimate the presence and magnitude of the influence of brightness saturation in a captured image. By generating a saturation influence map, the machine learning model can execute processing that should be executed on areas affected by brightness saturation and processing that should be executed on other areas, respectively, in the appropriate areas. Therefore, by having a machine learning model generate a saturation influence map, task accuracy is improved compared to when the generation of a saturation influence map is not involved (i.e., only recognition labels and blur-sharpened images are generated directly from the captured image).

[0027] While the above two techniques are effective, as explained with reference to FIGS. 4A to 4C, it is difficult to completely eliminate the adverse effects. Therefore, the adverse effects are suppressed by taking a weighted average of the captured image and the blur-sharpened image. The dashed-dotted line in FIG. 4A represents the signal value obtained by taking the weighted average of the captured image and the blur-sharpened image. By taking the weighted average, undershoot in dark areas is reduced while maintaining the blur-sharpening effect. In this embodiment, the weight map used when taking the weighted average of the captured image and the blur-sharpened image is generated based on information regarding the brightness of the captured image or information regarding the scene of the captured image and information based on the saturated area. This maintains the correction effect in dark images where undershoot is not noticeable, while suppressing the correction effect in bright images where undershoot is noticeable, thereby reducing the adverse effects. In other words, it is possible to maintain the blur correction effect while suppressing the adverse effects around the saturated area, which vary depending on the scene.

[0028] Next, training of the machine learning model executed by the training device 101 will be described with reference to Fig. 5. Fig. 5 is a flowchart of training of the machine learning model. The training device 101 has a storage unit 101a, an acquisition unit 101b, a calculation unit 101c, and an update unit 101d, and any of these components executes the following steps.

[0029] First, in step S101, the acquisition unit 101b acquires one or more original images from the storage unit 101a. The original image is an image having a signal value higher than the second signal value. The second signal value is a signal value corresponding to the saturation brightness value of the captured image. However, since the signal value may be normalized when input to the machine learning model, the second signal value does not necessarily have to match the saturation brightness value of the captured image. Since the machine learning model is trained based on the original image, it is desirable that the original image be an image having various frequency components (edges of different directions and intensities, gradations, flat areas, etc.). The original image may be a real-life image or CG (Computer Graphics).

[0030] Next, in step S102, the calculation unit 101c blurs the original image to generate a blurred image. The blurred image is an image input to the machine learning model during training and corresponds to the captured image during estimation. The blur to be added is the blur to be sharpened. In this embodiment, blur generated by aberration and diffraction of the optical system 102a and the optical low-pass filter of the image sensor 102b is added. The shape of the blur caused by aberration and diffraction of the optical system 102a varies depending on the image plane coordinates (image height and azimuth). It also varies depending on the magnification, aperture, and focus state of the optical system 102a. To simultaneously train a machine learning model that sharpens all of these blurs, it is recommended to generate multiple blurred images using multiple blurs generated by the optical system 102a. Furthermore, in the blurred image, signal values ​​exceeding a second signal value are clipped. This is done to reproduce brightness saturation that occurs during the capture process of the captured image. If necessary, noise generated by the image sensor 102b may be added to the blurred image.

[0031] Next, in step S103, the calculation unit 101c sets a first region based on an image based on the original image and a signal value threshold. In this embodiment, a blurred image is used as the image based on the original image, but the original image itself may also be used. The first region is set by comparing the signal value of the blurred image with the signal value threshold. More specifically, the first region is defined as a region where the signal value of the blurred image is equal to or greater than the signal value threshold. In the first embodiment, the signal value threshold is the second signal value. Therefore, the first region represents a brightness-saturated region of the blurred image. However, the signal value threshold and the second signal value do not necessarily have to match. The signal value threshold may be set to a value slightly smaller than the second signal value (for example, 0.9 times).

[0032] Next, in step S104, the calculation unit 101c generates a first region image having the signal value of the original image in the first region. The first region image has a signal value different from that of the original image in regions other than the first region. More preferably, the first region image has a first signal value in regions other than the first region. In this embodiment, the first signal value is 0, but the invention is not limited to this. In Example 1, the first region image has the signal value of the original image only in regions where the blurred image is brightness saturated, and the signal value in other regions is 0.

[0033] Next, in step S105, the calculation unit 101c blurs the first region image to generate a saturation influence correct map. The blur to be applied is the same as the blur applied to the blurred image. As a result, a saturation influence correct map is generated, which is a map (a spatially arranged signal sequence) that represents the magnitude and range of signal values ​​that have expanded due to deterioration during imaging from the subject in the brightness-saturated region of the blurred image. In the first embodiment, the saturation influence correct map is clipped with the second signal value, as with the blurred image, but clipping is not necessarily required.

[0034] Next, in step S106, the acquisition unit 101b acquires the correct model output. In this embodiment, since the task is blur sharpening, the correct model output is an image with less blur than the blurred image. In embodiment 1, the correct model output is generated by clipping the original image with the second signal value. If the original image lacks high-frequency components, an image obtained by reducing the original image may be used as the correct model output. In this case, reduction is also performed similarly when generating the blurred image in step S102. Note that step S106 may be executed any time after step S101 and before step S107.

[0035] Next, in step S107, the calculation unit 101c generates a saturation influence map and a model output based on the blurred image using a machine learning model. FIG. 1 is a configuration diagram of the machine learning model. Note that in this embodiment, the machine learning model shown in FIG. 1 is used, but is not limited to this. A blurred image 201 and a brightness saturation map 202 are input to the machine learning model. The brightness saturation map 202 is a map that indicates brightness saturated areas of the blurred image 201 (where the signal value is equal to or greater than a second signal value). For example, the brightness saturation map 202 can be generated by binarizing the blurred image 201 using the second signal value. However, the brightness saturation map 202 is not essential. The blurred image 201 and the brightness saturation map 202 are linked in the channel direction and input to the machine learning model. However, the invention is not limited to this. For example, the blurred image 201 and the brightness saturation map 202 may be converted into feature maps, and these feature maps may be linked in the channel direction. Furthermore, information other than the brightness saturation map 202 may be added to the input.

[0036] The machine learning model has multiple layers, and in each layer, a linear sum of the layer's input and weights is calculated. The initial values ​​of the weights may be determined using random numbers or the like. In the first embodiment, the machine learning model is a CNN that uses the convolution of the input and a filter as the linear sum (the value of each element of the filter corresponds to the weight, and may also include a sum with a bias). However, the invention is not limited to this. Furthermore, in each layer, nonlinear transformation is performed using an activation function such as a ReLU (Rectified Linear Unit) or a sigmoid function as necessary. Furthermore, the machine learning model may have a residual block or a skip connection (also called a shortcut connection) as necessary. As a result of passing through multiple layers (16 convolutional layers in this embodiment), a saturation influence map 203 is generated. In this embodiment, the saturation influence map 203 is generated by taking the element-by-element sum of the output of layer 211 and the intensity saturation map 202, but this is not limited to this. The saturation influence map may also be generated directly as the output of layer 211. Alternatively, the saturation influence map 203 may be the result of performing any processing on the output of the layer 211 .

[0037] Next, the saturation influence map 203 and the blurred image 201 are concatenated in the channel direction and input to a subsequent layer, and as a result of passing through multiple layers (16 convolutional layers in the first embodiment), a model output 204 is generated. The model output 204 is also generated by taking the sum of the output of layer 212 and the blurred image 201 for each element, but the configuration is not limited to this. Note that in the first embodiment, convolution is performed with 64 types of 3×3 filters in each layer (however, in the layers 211 and 212, the number of filter types is the same as the number of channels of the blurred image 201), but the configuration is not limited to this.

[0038] Next, in step S108, the update unit 101d updates the weights of the machine learning model based on the error function. In the first embodiment, the error function is a weighted sum of the error between the saturation influence map 203 and the saturation influence correct map and the error between the model output 204 and the correct model output. MSE (Mean Squared Error) is used to calculate the error. Both weights are set to 1. However, the error function and weights are not limited to this. Backpropagation or the like may be used to update the weights. The error may also be taken for the residual component. In the case of the residual component, the error between the difference component between the saturation influence map 203 and the brightness saturation map 202 and the difference component between the saturation influence correct map and the brightness saturation map 202 is used. Similarly, the error between the difference component between the model output 204 and the blurred image 201 and the difference component between the correct model output and the blurred image 201 is used.

[0039] Next, in step S109, the update unit 101d determines whether training of the machine learning model is complete. Completion can be determined by, for example, whether the number of iterations of weight update has reached a predetermined number, or whether the amount of change in weight during update is smaller than a predetermined value. If it is determined in step S109 that training is not complete, the process returns to step S101, and the acquisition unit 101b acquires one or more new original images. On the other hand, if it is determined that training is complete, the update unit 101d ends training and stores the configuration of the machine learning model and information about the weights in the storage unit 101a.

[0040] The above training method enables the machine learning model to estimate a saturation influence map that represents the magnitude and range of signal values ​​that have spread due to blurring of subjects in brightness-saturated regions of a blurred image (a captured image at the time of estimation). Explicitly estimating the saturation influence map enables the machine learning model to sharpen the blur for both saturated and non-saturated blurred images in the appropriate regions, thereby suppressing the occurrence of artifacts.

[0041] Next, blur sharpening of a captured image using a trained machine learning model executed by the image processing device 103 will be described with reference to Fig. 6. Fig. 6 is a flowchart of generating a model output. The image processing device 103 has a storage unit 103a, an acquisition unit (acquisition means) 103b, and a sharpening unit (first generation means, second generation means) 103c, any of which executes the following steps.

[0042] First, in step S201, the acquisition unit 103b acquires a captured image obtained through the optical system 102a and a machine learning model. Information on the configuration and weights of the machine learning model is acquired from the storage unit 103a. Next, in step S202, the sharpening unit (first generation means) 103c uses the machine learning model to generate a blur-sharpened image (model output, first image) from the captured image, in which the blur of the captured image has been sharpened. That is, the sharpening unit 103c corrects the blur component of the captured image to generate the first image. Note that the machine learning model has the configuration shown in FIG. 1, as in training. As in training, a brightness saturation map representing brightness-saturated regions of the captured image is generated and input, and a saturation influence map and model output are generated.

[0043] Next, the synthesis of a captured image and a model output (adjustment of the sharpening strength) executed by the image processing device 103 will be described with reference to Fig. 7. Fig. 7 is a flowchart of the sharpening strength adjustment.

[0044] First, in step S211, the sharpening unit 103c generates a weight map based on the saturation influence map when combining the captured image and the model output. The generation of the weight map will be described in detail. The weight map determines the ratio of each image when weighting the captured image and the blur-sharpened image, and has continuous signal values ​​ranging from 0 to 1. For example, if the value of the weight map determines the ratio of the blur-sharpened image, a value of 1 leaves the captured image as is. Also, if the value of the weight map is 0.5, the weighted average image is the sum of the pixel values ​​of the captured image and the blur-sharpened image, each at 50%. In this embodiment, the value of the weight map represents the weight of the captured image. When determining the weight, the saturation influence map is normalized by a set signal value, and this is used as the weight map for the captured image. It is also possible to adjust the balance between the blur-sharpening effect and adverse effects by changing the signal value by which the saturation influence map is normalized.

[0045] In the vicinity of brightness saturated regions in a captured image, subject information is attenuated due to brightness saturation compared to other regions, making it more difficult to sharpen the blur (estimate attenuated subject information). Therefore, adverse effects (such as ringing and undershoot) associated with blur sharpening are likely to occur in the vicinity of brightness saturated regions. To mitigate these adverse effects, the model output and the captured image are synthesized. In this case, by synthesizing the image based on a saturation influence map, the degradation of the blur sharpening effect of non-saturated blurred images is suppressed, while the adverse effects can be suppressed by increasing the weight of the captured image only in the vicinity of brightness saturated regions where adverse effects are likely to occur.

[0046] Next, in step S212, the acquiring unit 103b acquires information about the brightness of the captured image or information about the scene of the captured image. Here, the information about the brightness of the captured image is a statistic about the signal values ​​of the captured image, and is information based on at least one of the average, median, variance, or histogram of the signal values ​​of the captured image. Furthermore, the statistic about the signal values ​​of the captured image, which is information about the brightness of the captured image, may be information about the entire captured image, or may be a statistic for each divided area of ​​the captured image (or a third image, which will be described later).

[0047] The information about the scene of the captured image is information about the type of scene of the captured image and the imaging mode when the captured image was captured. However, it needs to be information about the scene type such as "daytime" or "night view" or the imaging mode such as "night view imaging mode" so that differences in brightness of the captured image can be distinguished. The information about the scene of the captured image may be obtained by performing scene determination on the captured image (determining the type of scene), or may be obtained from the scene type and imaging mode information written in the captured image.

[0048] In this embodiment, the average signal value of the captured image is acquired as information regarding the brightness of the captured image. However, when acquiring the average signal value of the captured image, if there are many saturated areas in the captured image, a large average signal value may be acquired even for a dark image such as a night scene, and the image may be determined to be bright. Therefore, it is preferable to acquire the average signal value of a third image obtained by removing the saturation influence map from the captured image. By acquiring the average signal value of the non-saturated areas in the third image, it is possible to appropriately determine whether the captured image is a bright scene or a dark scene.

[0049] Next, in step S213, the sharpening unit 103c adjusts the weight map of the captured image based on information about the brightness of the captured image or information about the scene of the captured image. That is, the weight map is generated based on information about the brightness of the captured image or information about the scene of the captured image and information based on saturated areas in the captured image.

[0050] When the average signal value of the captured image (or the third image) is used as information relating to the brightness of the captured image, the weight map is adjusted so that the larger the average signal value (i.e., the brighter the captured image (or the third image)), the larger the weight of the captured image. Specifically, the relationship between the average signal value and the adjustment value of the weight map is held as a linear function, and an adjustment value according to the average signal value of the captured image is obtained and applied (multiplied) to the weight map.

[0051] FIG. 8 is an explanatory diagram of weight map adjustment values ​​relative to average signal values. In FIG. 8, the horizontal axis represents the average signal value, and the vertical axis represents the weight map adjustment value. For example, an adjustment value corresponding to the average signal value obtained from relationship 121 as shown in FIG. 8 is used. However, the relationship between the average signal value and the weight map adjustment value is not limited to this. Furthermore, when the average signal value is obtained for each region obtained by dividing the captured image, the distribution of the average signal values ​​for each region may be converted into an average signal value map representing the average signal value corresponding to each pixel of the captured image, and an adjustment value corresponding to the average signal value may be obtained for each pixel to adjust the weight map.

[0052] When information about the scene of the captured image is acquired in step S203, the weight map is adjusted based on whether the captured image is a dark image, for example. For example, if the captured image is a bright image, the weight map is used as is, but if the image is a dark image such as a night scene, a weight map with the weight map values ​​adjusted to half is acquired.

[0053] When the average, median, variance, or histogram of the signal values ​​of the captured image is acquired as a statistical quantity related to the signal values ​​of the captured image, the weight map is adjusted so that the larger the statistical quantity, the greater the weight of the captured image. For example, when a histogram of the signal values ​​of the captured image is used, the weight map is adjusted so that the higher the center of gravity or peak of the histogram, the greater the weight of the captured image.

[0054] FIG. 9(A) shows a daytime image, which is a bright scene, and FIG. 9(B) shows a saturation influence map for the image captured in FIG. 9(A). FIG. 10(A) shows a nighttime image, which is a dark scene, and FIG. 10(B) shows a saturation influence map for the image captured in FIG. 10(A). FIG. 11 shows a weight map obtained by adjusting weights based on the saturation influence map for the daytime image, which is a bright scene, shown in FIG. 9(B), according to the average signal value of the image. The weight map is adjusted to be larger according to the average signal value of the image. By adjusting the weight map to be larger, adverse effects that are noticeable in bright scenes can be suppressed. FIG. 12 shows a weight map obtained by adjusting weights based on the saturation influence map for the nighttime image, which is a dark scene, shown in FIG. 10(B), according to the average signal value of the image. The weight map is adjusted to be smaller according to the average signal value of the image. Because adverse effects are less noticeable in dark scenes, the effect can be maintained by adjusting the weight map to be smaller.

[0055] In this embodiment, the weight map is generated based on the saturation influence map and is therefore applied to the saturated region and its surrounding areas. However, a second weight map for the non-saturated region, different from the weight map (first weight map) based on the saturation influence map, may be used. Furthermore, a third weight map for the saturated region may be applied to the weight map (first weight map) calculated from the saturation influence map. By using the second and third weight maps, it is possible to adjust the intensity in each of the saturated and non-saturated regions. In this case, the weight map can be calculated as (1 - first weight map) x second weight map + first weight map x adjustment value x third weight map. Furthermore, the acquired weight map may be adjusted according to user instructions. For example, the intensity can be adjusted by multiplying the entire weight map by a coefficient.

[0056] Next, in step S214, the sharpening unit (second generating means) 103c uses the weight map adjusted in step S213 to perform a weighted average of the captured image and the blur-sharpened image (model output, first image) to generate an intensity-adjusted image 205 (second image). That is, the sharpening unit 103c generates the second image based on the captured image, the first image, and the weight map. In this embodiment, a weight map obtained by subtracting the weight map of the captured image from a map of all ones is used for the blur-sharpened image.

[0057] With the above configuration, it is possible to provide an image processing system that can generate an image with an appropriate correction effect depending on the brightness and scene around the saturated region when sharpening blur using a machine learning model. [Example]

[0058] Next, an image processing system according to a second embodiment of the present invention will be described. In this embodiment, a method for sharpening blur by image restoration processing, which is a method different from machine learning, will be described.

[0059] First, an overview of the image restoration process will be explained. When the captured image (degraded image) is g(x, y), the original image is f(x, y), and the point spread function PSF, which is a Fourier pair of the optical transfer function OTF, is h(x, y), the following equation (1) holds.

[0060] g(x,y)=h(x,y)*f(x,y) … (1) Here, * denotes convolution (convolution integral, product sum), and (x, y) are coordinates on the captured image.

[0061] Furthermore, when equation (1) is Fourier transformed and converted into a frequency-plane display format, equation (2) expressed as a product for each frequency is obtained.

[0062] G(u,v)=H(u,v)·F(u,v) … (2) Here, H is the optical transfer function OTF obtained by Fourier transforming the point spread function PSF(h), G and F are functions obtained by Fourier transforming the degraded image g and the original image f, respectively. (u,v) are coordinates on the two-dimensional frequency plane, i.e., frequency.

[0063] To obtain the original image f from the captured degraded image g, it is sufficient to divide both sides of the following equation (3) by the optical transfer function H.

[0064] G(u,v) / H(u,v)=F(u,v) … (3) Then, by performing an inverse Fourier transform on F(u,v), i.e., G(u,v) / H(u,v), to return it to the real plane, the original image f(x,y) is obtained as a restored image.

[0065] If the inverse Fourier transform of H-1 is R, the original image f(x, y) can be obtained by performing convolution processing on the image on the real surface as shown in the following equation (4).

[0066] g(x,y)*R(x,y)=f(x,y) … (4) Here, R(x, y) is called the image restoration filter. When the image is two-dimensional, the image restoration filter R is generally a two-dimensional filter with taps (cells) corresponding to each pixel of the image. Furthermore, the greater the number of taps (cells) of the image restoration filter R, the higher the restoration accuracy. Therefore, a feasible number of taps is set depending on the required image quality, image processing capabilities, aberration characteristics, etc. Because the image restoration filter R must at least reflect the aberration characteristics, it differs from conventional edge enhancement filters with approximately three taps each horizontally and vertically. Because the image restoration filter R is set based on the optical transfer function (OTF), it can correct both the degradation of the amplitude and phase components with high accuracy.

[0067] Furthermore, because actual images contain noise components, using the image restoration filter R created by taking the reciprocal of the optical transfer function OTF as described above will significantly amplify the noise components while restoring the degraded image. This is because the MTF (amplitude component) of the optical system is raised to return it to 1 across all frequencies in a state where the amplitude of noise is added to the amplitude component of the image. The MTF, which is the amplitude degradation caused by the optical system, returns to 1, but at the same time the power spectrum of the noise is also raised, resulting in amplified noise according to the degree to which the MTF is raised (restoration gain).

[0068] Therefore, if noise is included, a good image for viewing cannot be obtained. This is expressed by the following equations (5-1) and (5-2).

[0069] G(u,v)=H(u,v)·F(u,v)+N(u,v) … (5-1) G(u,v) / H(u,v)=F(u,v)+N(u,v) / H(u,v) … ( 5-2) where N is the noise component.

[0070] For images containing noise components, there is a method of controlling the degree of restoration according to the intensity ratio SNR of the image signal to the noise signal, such as a Wiener filter expressed by the following equation (6).

[0071]

number

[0072] Here, M(u,v) is the frequency characteristic of the Wiener filter, and |H(u,v)| is the absolute value of the optical transfer function OTF (MTF). In this method, the smaller the MTF for each frequency, the smaller the restoration gain (degree of restoration), and the larger the MTF, the larger the restoration gain. Generally, the MTF of an imaging optical system is high on the low frequency side and low on the high frequency side, so this method essentially reduces the restoration gain on the high frequency side of the image.

[0073] Next, the image restoration filter will be described with reference to Fig. 13 and Fig. 14. Fig. 13 and Fig. 14 are explanatory diagrams of the image restoration filter. The number of taps of the image restoration filter is determined depending on the aberration characteristics of the imaging optical system and the required restoration accuracy. The image restoration filter of Fig. 13 is, as an example, a two-dimensional filter with 11 × 11 taps. Although the values ​​(coefficients) in each tap are omitted in Fig. 13, a cross section of this image restoration filter is shown in Fig. 14. The distribution of the values ​​(coefficient values) of each tap of the image restoration filter has the function of ideally returning a signal value (PSF) that has been spatially spread due to aberration to the original single point.

[0074] Each tap of the image restoration filter is subjected to convolution processing (convolution integral, product-sum) in the image restoration process, corresponding to each pixel of the image. In the convolution processing, in order to improve the signal value of a specific pixel, the pixel is aligned with the center of the image restoration filter. Then, for each corresponding pixel in the image and the image restoration filter, the signal value of the image and the coefficient value of the filter are multiplied, and the sum of these products is replaced as the signal value of the center pixel.

[0075] Next, the characteristics of image restoration in real space and frequency space will be described with reference to FIGS. 15 and 16. FIG. 15 is an explanatory diagram of the point spread function PSF, where FIG. 15(a) shows the point spread function PSF before image restoration and FIG. 15(b) shows the point spread function PSF after image restoration. FIG. 16 is an explanatory diagram of the amplitude component MTF (FIG. 16(M)) and phase component PTF (FIG. 16(P)) of the optical transfer function OTF. The dashed line (a) in FIG. 16(M) shows the MTF before image restoration, and the dashed line (b) shows the MTF after image restoration. Furthermore, the dashed line (a) in FIG. 16(P) shows the PTF before image restoration, and the dashed line (b) shows the PTF after image restoration. As shown in FIG. 15(a), the point spread function PSF before image restoration has an asymmetric spread, and due to this asymmetry, the phase component PTF has a nonlinear value with respect to frequency. The image restoration process amplifies the amplitude component MTF and corrects the phase component PTF to zero, so that the point spread function PSF after image restoration has a symmetrical and sharp shape.

[0076] In this way, the image restoration filter can be obtained by inverse Fourier transforming a function designed based on the inverse function of the optical transfer function OTF of the imaging optical system. The image restoration filter used in this embodiment can be changed as appropriate, and for example, the Wiener filter described above can be used. When using a Wiener filter, it is possible to create a real-space image restoration filter that is actually convolved with the image by performing an inverse Fourier transform on Equation (6).

[0077] Furthermore, since the optical transfer function (OTF) due to aberration changes depending on the image height (image position) of the imaging optical system even in one imaging state, it is necessary to change the image restoration filter depending on the image height. On the other hand, the optical transfer function (OTF) due to diffraction, whose influence becomes more dominant as the F-number increases, can be treated as a uniform OTF for all image heights if the influence of vignetting in the optical system is small.

[0078] When the target of image restoration processing is diffraction (diffraction blur) and not aberration, the image restoration filter depends only on the aperture value and the wavelength of light, and not on the image height (image position). Therefore, a uniform (constant) image restoration filter can be used for a single image. That is, an image restoration filter for correcting diffraction blur is generated based on the optical transfer function of the diffraction blur that occurs depending on the aperture value. Regarding wavelength, optical transfer functions at multiple wavelengths can be calculated, and an optical transfer function for each color component can be generated by weighting each wavelength based on the assumed spectrum of the light source and the light sensitivity information of the image sensor. Alternatively, calculations can be performed using a representative wavelength for each predetermined color component. Then, an image restoration filter can be generated based on the optical transfer function for each color component.

[0079] Therefore, when only diffraction is to be corrected, a plurality of image restoration filters that depend on the aperture value can be stored in advance, and the image can be processed using a uniform (constant) image restoration filter depending on the aperture value shooting condition. It is also preferable to take into account the aperture degradation component due to the shape of the pixel aperture and the characteristics of the optical low-pass filter.

[0080] Next, an image processing system 200 in this embodiment will be described with reference to Fig. 17. Fig. 17 is a block diagram of the image processing system 200. The external view of the image processing system 200 is the same as Fig. 3 described in the first embodiment. The image processing system 200 has an image processing device 203 connected via a wired or wireless network. An imaging device 202, a display device 204, a recording medium 205, and an output device 206 are connected to the image processing device 203 via wired or wireless connections.

[0081] A captured image of a subject space captured using the imaging device 202 is input to the image processing device 203. The captured image is blurred due to aberration and diffraction caused by the optical system 202a in the imaging device 202 and the optical low-pass filter of the image sensor 202b, resulting in attenuation of subject information. The image processing device 203 performs blur sharpening on the captured image using image restoration processing to generate a blur-sharpened image. The image processing device 203 also acquires a saturation influence map. Details of the saturation influence map will be described later. The image processing device 203 also has a function of adjusting the intensity of blur sharpening by taking a weighted average of the captured image and the blur-sharpened image. The user can adjust the intensity of blur sharpening while checking the image displayed on the display unit 204. The blur-sharpened image after intensity adjustment is stored in the storage unit 203a or the recording medium 205 and, as necessary, output to an output device 206 such as a printer. The captured image may be grayscale or may have multiple color components. In addition, it may be an undeveloped RAW image or a developed image.

[0082] Next, blur sharpening of a captured image executed by the image processing device 203 will be described with reference to Fig. 18. Fig. 18 is a flowchart showing generation of a blur-sharpened image. The image processing device 203 has a storage unit 203a, an acquisition unit 203b, and a sharpening unit 203c, any of which executes the following steps.

[0083] First, in step S301, the acquisition unit 203b acquires a captured image. Then, in step S302, the image processing device 203 acquires an image restoration filter to be used in the image restoration process described later. In this embodiment, an example will be described in which aberration information (optical information) is acquired based on the shooting conditions, and an image restoration filter is acquired based on the aberration information.

[0084] First, the image processing device acquires the shooting conditions (shooting condition information) under which the imaging device generated the captured image by capturing an image. As described above, the shooting conditions include the focal length, aperture value (F-number), and shooting distance of the imaging optical system, as well as identification information (camera ID) of the imaging device. Furthermore, in an imaging device in which the imaging optical system is replaceable, the shooting conditions may include identification information (lens ID) of the imaging optical system (interchangeable lens). The shooting condition information may be acquired as information attached to the captured image, as described above, or may be acquired via wired or wireless communication or a storage medium.

[0085] Next, the image processing device 203 acquires aberration information appropriate for the shooting conditions. In this embodiment, the aberration information is an optical transfer function OTF. The image processing device selects and acquires an appropriate optical transfer function OTF according to the shooting conditions from a plurality of pre-stored optical transfer functions OTF. Furthermore, when shooting conditions such as aperture value, shooting distance, and zoom lens focal length are specific shooting conditions, the optical transfer function OTF corresponding to the shooting conditions can be generated by interpolation processing from the optical transfer functions OTF for other shooting conditions pre-stored. In this case, it is possible to reduce the amount of data of the optical transfer functions OTF to be stored. For example, bilinear interpolation (linear interpolation) or bicubic interpolation is used as the interpolation processing, but it is not limited thereto.

[0086] In this embodiment, the image processing device 203 acquires an optical transfer function OTF as aberration information, but is not limited to this. Instead of the optical transfer function OTF, aberration information such as a point spread function PSF may be acquired. Furthermore, in this embodiment, the image processing device may acquire coefficient data that approximates the aberration information by fitting it to a predetermined function, and reconstruct the optical transfer function OTF or point spread function PSF from the coefficient data. For example, the optical transfer function OTF may be fitted using Legendre polynomials. Alternatively, fitting may be performed using other functions such as Chebushev polynomials. Furthermore, in this embodiment, the optical transfer function OTF is discretely arranged at multiple positions within the captured image.

[0087] Next, the image processing device 203 converts the optical transfer function OTF into an image restoration filter, i.e., generates an image restoration filter using the optical transfer functions OTF arranged at multiple positions. The image restoration filter is generated by creating restoration filter characteristics in frequency space based on the optical transfer function OTF and converting them into a filter (image restoration filter) in real space by inverse Fourier transform.

[0088] Furthermore, when the correction target of the image restoration process is blur that does not include aberration and does not depend on the image height (image position), such as diffraction (diffraction blur), a uniform (fixed) optical transfer function OTF or image restoration filter may be used within one image. Although the generation and acquisition of an image restoration filter have been described above, the present invention is not limited to this. An image restoration filter may be generated and stored in advance, and the image restoration filter may be acquired based on the shooting conditions.

[0089] Next, in step S303, the sharpening unit 203c performs image restoration processing on the captured image to generate a blur-sharpened image (first image) in which the blur of the captured image is sharpened. The image restoration processing is performed based on the image restoration filter acquired in step S302.

[0090] During the convolution of the image restoration filter, pixels other than the position where the image restoration filter is located can be generated by interpolation using multiple filters located nearby. In this case, the image restoration filter includes a first image restoration filter at a first position in the captured image and a second image restoration filter at a second position in the captured image. The first image restoration filter is generated using an expanded optical transfer function. The second image restoration filter is generated by interpolation using the first image restoration filter. By performing such interpolation processing, the image restoration filter can be changed for each pixel, for example.

[0091] Next, in step S304, the sharpening unit 203c estimates a saturation influence map. In this embodiment, the saturation influence map is generated based on the brightness saturation regions in the captured image and aberration information of the imaging optical system. That is, the saturation influence map is estimated by convolving the brightness saturation map, which is a map representing the brightness saturation regions in the captured image, with the PSF, which indicates the blur of the imaging optical system. However, since information about the structure of the subject space is lost in brightness saturated regions (brightness saturated regions), it is preferable to use a brightness saturation map obtained by estimating the original signal values ​​of the brightness saturation regions.

[0092] Next, the image processing device 203 combines the captured image with the blur-sharpened image. The combination of the captured image and the blur-sharpened image is the same as in the flowchart of Fig. 7, so a detailed description will be omitted. Note that in this embodiment, image restoration processing is used as the blur-sharpening method, but this is not limiting, and various sharpening methods such as sharpness and unsharp masking may also be used.

[0093] With the above configuration, it is possible to provide an image processing system that can generate an image with an appropriate correction effect according to the brightness and scene around the saturated region when sharpening blur. [Example]

[0094] Next, an image processing system 300 according to a third embodiment of the present invention will be described. Fig. 19 is a block diagram of the image processing system 300. Fig. 20 is an external view of the image processing system 300. The image processing system 300 has a training device 301, an image capturing device 302, and an image processing device 303. The training device 301 and the image processing device 303, and the image processing device 303 and the image capturing device 302 are each connected via a wired or wireless network. The image capturing device 302 has an optical system 321, an image capturing element 322, a storage unit 323, a communication unit 324, and a display unit 325. The captured image is transmitted to the image processing device 303 via the communication unit 324.

[0095] The image processing device 303 receives the captured image via the communication unit 332 and performs blur sharpening using information on the configuration and weights of the machine learning model stored in the storage unit 331. The information on the configuration and weights of the machine learning model is trained by the training device 301, is acquired in advance from the training device 301, and is stored in the storage unit 331. The image processing device 303 also has a function of adjusting the intensity of blur sharpening. A blur-sharpened image (model output) in which the blur of the captured image has been sharpened, and a weighted average image in which the intensity has been adjusted are transmitted to the imaging device 302, stored in the storage unit 323, and displayed on the display unit 325.

[0096] The generation of learning data and weight learning (learning phase) performed by the training device 301, the sharpening of blur in the captured image using a trained machine learning model executed by the image processing device 303 (estimation phase), and the synthesis of the captured image and model output are the same as in Example 1, so they will be omitted.

[0097] With the above configuration, it is possible to provide an image processing system that can generate an image with an appropriate correction effect depending on the brightness and scene around the saturated region when sharpening blur using a machine learning model. [Example]

[0098] Next, an image processing system 400 according to a fourth embodiment of the present invention will be described. FIG. 21 is a block diagram of the image processing system 400. FIG. 22 is an external view of the image processing system 400. The image processing system 400 includes a learning device 401, a lens device 402, an imaging device 403, a control device (first device) 404, an image estimation device (second device) 405, and networks 406 and 407. The learning device 401 and the image estimation device 405 are, for example, servers. The control device 404 is a device operated by a user, such as a personal computer or a mobile terminal. The learning device 401 and the image estimation device 405, and the control device 404 and the image estimation device 405, are capable of communicating with each other.

[0099] The learning device 401 has a storage unit 401a, an acquisition unit 401b, a calculation unit 401c, and an update unit 401d, and learns the weights of a machine learning model that sharpens blur from a captured image captured using a lens device 402 and an imaging device 403. The learning method is the same as in Example 1, and therefore will not be described here. The imaging device 403 has an imaging element 403a, which photoelectrically converts an optical image formed by the lens device 402 to acquire the captured image. The lens device 402 and the imaging device 403 are detachable, and can be combined with a plurality of types of each other.

[0100] The control device 404 has a communication unit 404a, a display unit 404b, a storage unit 404c, and an acquisition unit 404d, and controls, in accordance with a user's operation, the processing to be executed on the captured image acquired from the wired or wirelessly connected imaging device 403. Alternatively, the captured image captured by the imaging device 403 may be stored in advance in the storage unit 404c, and the captured image may be read out.

[0101] The image estimation device 405 includes a communication unit 405a, an acquisition unit 405b, a storage unit 405c, and a sharpening unit 405d. The image estimation device 405 executes a process of sharpening the blur of a captured image in response to a request from a control device 404 connected via a network 406. The image estimation device 405 acquires information on learned weights from a learning device 401 connected via the network 406 when estimating blur sharpening or in advance, and uses the information to estimate blur sharpening of a captured image. The estimated image after blur sharpening estimation is subjected to sharpening intensity adjustment, then transmitted again to the control device 404, stored in the storage unit 404c, and displayed on the display unit 404b. Note that the generation of learning data and weight learning (learning phase) performed by the learning device 401 are the same as those in the first embodiment, and therefore will not be described here.

[0102] Next, blur sharpening of a captured image executed by the control device 404 and the image estimation device 405 will be described with reference to Fig. 23. Fig. 23 is a flowchart of model output and sharpening strength adjustment.

[0103] First, in step S401, the acquisition unit 404d acquires a captured image and a sharpening strength specified by a user. Then, in step S402, the communication unit 404a transmits the captured image and a request for execution of blur sharpening estimation processing to the image estimation device 405.

[0104] Next, in step S403, the communication unit 405a receives and acquires the captured image and processing request that have been sent. Next, in step S404, the acquisition unit 405b acquires information on learned weights corresponding to the captured image from the storage unit 405c. The weight information is read out in advance from the storage unit 401a and stored in the storage unit 405c. Next, in step S405, the sharpening unit 405d uses the machine learning model to generate, from the captured image, a blur-sharpened image (model output, first image) in which the blur of the captured image has been sharpened. The machine learning model has the configuration shown in FIG. 1, as in training. As in training, a brightness saturation map representing brightness-saturated regions of the captured image is generated and input, and a saturation influence map and model output are generated.

[0105] Next, in step S406, the sharpening unit 405d generates a weight map. The method of generating the weight map and the method of combining the captured image and the blur-sharpened image (model output), which will be described later, are the same as in the first embodiment. The intensity can be adjusted by adjusting the weight map according to the sharpening intensity specified by the user. For example, the intensity of the saturated region can be adjusted by changing the relational expression between the average signal value and the adjustment value shown in FIG. 8. Alternatively, when a second weight map related to the intensity of the non-saturated region and a third weight map related to the intensity of the saturated region are used, the intensities of the non-saturated region and the saturated region may be adjusted by adjusting the second and third weight maps. Alternatively, the entire weight map may be adjusted.

[0106] Next, in step S407, the sharpening unit 405d combines the captured image with the blur-sharpened image (model output) based on the weight map. Next, in step S408, the communication unit 405a transmits the combined image to the control device 404. Next, in step S409, the communication unit 404a acquires the transmitted combined image. With the above configuration, it is possible to provide an image processing system that can generate an image with an appropriate correction effect depending on the brightness and scene around the saturated region when sharpening blur using a machine learning model. (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0107] According to each embodiment, it is possible to provide an image processing method, an image processing device, an image processing system, and a program that are capable of performing appropriate blur correction according to the brightness and scene of an image.

[0108] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0109] 103 Image processing device 103b Acquisition unit (acquisition means) 103c sharpening unit (first generating means, second generating means)

Claims

1. generating a first image by correcting blur components of a captured image obtained by imaging; generating a second image based on the captured image, the first image, information about a saturated region in the captured image, information about brightness of the captured image, or information about a scene in the captured image; An image processing method characterized in that the information about the saturated region is information representing a blurred region surrounding a brightness saturated region of the captured image, which is generated by inputting the captured image into a machine learning model.

2. In the step of generating the second image, the second image is generated based on the captured image, the first image, and weighting information; 2. The image processing method according to claim 1, wherein the weight information is generated based on information about the saturated region and information about the brightness or information about the scene.

3. The image processing method described in Claim 2, characterized in that the weight information is generated based on a third image obtained by taking the difference between the captured image and information about the saturated region.

4. The image processing method described in Claim 3, characterized in that the weight information is generated based on the signal values ​​of pixels in the third image that correspond to pixels in the non-saturated region in the captured image, among the multiple pixels in the third image.

5. An image processing method as described in claim 3 or 4, characterized in that the weight information is generated based on statistics regarding signal values ​​for each divided area in the third image.

6. An image processing method described in any one of claims 3 to 5, characterized in that the weight of the captured image represented by the weight information becomes larger as the brightness of the third image increases.

7. The image processing method described in Claim 6, characterized in that the brightness of the third image is determined based on an average signal value of the third image.

8. An image processing method described in any one of claims 2 to 7, characterized in that the weight of the captured image represented by the weight information becomes larger as the brightness of the captured image increases.

9. An image processing method described in any one of claims 2 to 8, characterized in that the second image is generated by taking a weighted average of the captured image and the first image using the weight information.

10. 10. The image processing method according to claim 1, wherein in the step of generating the first image, the blur component is corrected by inputting the captured image into a machine learning model.

11. 11. The image processing method according to claim 1, wherein the information about brightness is a statistical quantity related to signal values ​​of the captured image.

12. 12. The image processing method according to claim 1, wherein the information about brightness is information based on at least one of an average value, a median value, a variance, and a histogram of signal values ​​of the captured image.

13. 13. The image processing method according to claim 1, wherein the information about the scene is information about the type of the scene or information about an image capture mode for the captured image.

14. 14. The image processing method according to claim 1, wherein the blur component is based on optical information of an optical system used in the image capture.

15. A program that causes a computer to execute the image processing method described in any one of claims 1 to 14.

16. a first generation means for generating a first image by correcting a blur component of a captured image obtained by capturing an image; a second generation means for generating a second image based on the captured image, the first image, information about a saturated region in the captured image, information about brightness of the captured image, or information about a scene in the captured image, The image processing device is characterized in that the information regarding the saturated region is information representing a blurred region surrounding a brightness saturated region of the captured image, which is generated by inputting the captured image into a machine learning model.

17. An image processing system comprising the image processing device according to claim 16 and a control device capable of communicating with the image processing device, the control device has a transmission means for transmitting a request to cause the image processing device to execute processing on the captured image, The image processing system is characterized in that the image processing device has a receiving means for receiving the request, and executes processing on the captured image in response to the request.

Images