Image processing method, image processing device, image processing system, and program

Abstract

To provide an image processing method capable of controlling the sharpness and adverse effects of image blur in accordance with the performance of an optical system. [Solution] The image processing method includes the steps of inputting a captured image into a machine learning model to generate a first image, acquiring information about the optical performance of the optical system used to capture the captured image, and generating a second image based on the captured image, the first image, and a first weight map, wherein the first weight map is generated based on the information about the optical performance and information about the saturated region of the captured image.

Description

[Technical Field]

[0001] The present invention relates to an image processing method for sharpening blur caused by an optical system in an image captured using an optical system. [Background technology]

[0002] Patent Document 1 discloses a method for sharpening blur in a captured image using a convolutional neural network (CNN), which is one of the machine learning models. A training dataset is generated by blurring an image having a signal value equal to or greater than the brightness saturation value of the captured image, and the CNN is trained using the training dataset, thereby suppressing adverse effects even around brightness-saturated regions and performing blur sharpening. The document also discloses a method for adjusting the strength of sharpening by performing a weighted average of the captured image and an estimated image (blur-sharpened image) based on the brightness-saturated regions. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-166628 Summary of the Invention [Problem to be solved by the invention]

[0004] The method disclosed in Patent Document 1 cannot suppress undershooting, ringing, and other problems depending on the input image, and these problems may occur in the estimated image. Specifically, these problems are likely to occur when the subject is significantly blurred due to aberrations in the optical system. While it is possible to control the sharpness and problems of blur by performing a weighted average of the input image and the estimated image based on the brightness-saturated region, this method is unable to control the sharpness and problems of blur according to the performance of the optical system, and therefore is unable to perform appropriate control.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing method, an image processing device, an image processing system, and a program that are capable of controlling the sharpness and adverse effects of image blur in accordance with the performance of the optical system. [Means for solving the problem]

[0006] An image processing method according to one aspect of the present invention includes the steps of: generating a first image by correcting a blur component of a captured image obtained by imaging using an optical system; acquiring information about optical performance of the optical system; and generating a second image based on the captured image, the first image, the information about the optical performance, and information about a saturated region of the captured image, wherein the information about the saturated region is generated by inputting the captured image into a machine learning model. , which is information representing a blurred area around a brightness saturated area of ​​the captured image. .

[0007] Other objects and features of the present invention are illustrated in the following examples. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an image processing method, an image processing device, an image processing system, and a program that are capable of controlling the sharpness and adverse effects of image blur in accordance with the performance of the optical system. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a machine learning model according to a first embodiment. [Figure 2] 1 is a block diagram of an image processing system according to a first embodiment. [Figure 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4] 10A to 10C are explanatory diagrams of adverse effects caused by sharpening in the first to third embodiments. [Figure 5] 1 is a flowchart of training a machine learning model in Examples 1 to 3. [Figure 6]10 is a flowchart of generating a model output in the first and second embodiments. [Figure 7] 10 is a flowchart of adjusting the sharpening strength in the first embodiment. [Figure 8] FIG. 10 is an explanatory diagram of division points of optical performance indexes in Examples 1 to 3. [Figure 9] FIG. 2 is a schematic diagram of an imaging state space in which optical performance indicators are arranged in the first to third embodiments. [Figure 10] FIG. 10 is a diagram showing a relational expression between an optical performance index and a weight in the first to third embodiments. [Figure 11] FIG. 4 is an explanatory diagram of a second weight map and a third weight map in the first embodiment. [Figure 12] 4A and 4B are explanatory diagrams of a captured image and a saturation influence map in the first embodiment. [Figure 13] FIG. 4 is an explanatory diagram of a first weight map in the first embodiment. [Figure 14] FIG. 10 is a block diagram of an image processing system according to a second embodiment. [Figure 15] FIG. 10 is an external view of an image processing system according to a second embodiment. [Figure 16] 10 is a flowchart of adjusting the sharpening strength in the second embodiment. [Figure 17] FIG. 10 is a block diagram of an image processing system according to a third embodiment. [Figure 18] FIG. 11 is an external view of an image processing system according to a third embodiment. [Figure 19] 13 is a flowchart of model output and sharpening strength adjustment in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and redundant explanations will be omitted.

[0011] Before describing the specific embodiments, the gist of the present invention will be explained. The present invention generates an estimated image from a captured image captured using an optical system (image capturing optical system) by sharpening blur caused by the optical system using a machine learning model. Then, a weighting map (first weighting information) is generated based on the performance (optical performance) of the optical system and the brightness saturation region, and a weighted average is calculated between the captured image and the estimated image. Here, blur caused by the optical system includes blur caused by aberration, diffraction, and defocus, the effect of an optical low-pass filter, and pixel aperture degradation of the image sensor.

[0012] Machine learning models include, for example, neural networks, genetic programming, Bayesian networks, etc. Neural networks include convolutional neural networks (CNNs), generative adversarial networks (GANs), recurrent neural networks (RNNs), etc.

[0013] Blur sharpening refers to the process of restoring frequency components of a subject that have been reduced or lost due to blur. When blur sharpening is performed, undershoot (darkness) and ringing may not be suppressed depending on the captured image, and these problems may occur in the estimated image. Specifically, problems occur when the subject is significantly blurred due to aberrations in the optical system, or when there are brightness saturated areas in the image. Brightness saturated areas can occur in the image depending on the dynamic range of the image sensor and the exposure at the time of capture. In brightness saturated areas, it is difficult to obtain information about the structure of the subject's space, and problems are likely to occur.

[0014] Therefore, in each embodiment, the captured image and the estimated image are weighted-averaged using a weight map generated based on the performance of the optical system and the brightness saturation area, which makes it possible to control the sharpness and adverse effects of blur.

[0015] In the following, the stage of learning the weights of the machine learning model will be referred to as the learning phase, and the stage of sharpening blur using the machine learning model using the learned weights will be referred to as the estimation phase. [Example]

[0016] First, an image processing system 100 according to a first embodiment of the present invention will be described. In this embodiment, a machine learning model is used to sharpen blur in a captured image that includes saturation in brightness. The blur to be sharpened is targeted at aberrations and diffraction occurring in the optical system, and blur caused by an optical low-pass filter. However, the effects of the present invention can be similarly achieved when sharpening blur caused by pixel aperture, defocus, or shaking. The present invention can also be similarly implemented to achieve the effects of tasks other than blur sharpening.

[0017] FIG. 2 is a block diagram of the image processing system 100. FIG. 3 is an external view of the image processing system 100. The image processing system 100 has a training device 101 and an image processing device 103, which are connected via a wired or wireless network. An imaging device 102, a display device 104, a recording medium 105, and an output device 106 are connected to the image processing device 103 via wired or wireless connections. A captured image of a subject space captured using the imaging device 102 is input to the image processing device 103. Blurring occurs in the captured image due to aberration and diffraction caused by the optical system (imaging optical system) 102a in the imaging device 102 and the optical low-pass filter of the image sensor 102b, and information about the subject is attenuated.

[0018] The image processing device 103 performs blur sharpening on the captured image using a machine learning model to generate a saturation influence map and a blur-sharpened image (model output, first image). Details of the saturation influence map will be described later. The machine learning model is trained by the training device 101, and the image processing device 103 acquires information about the machine learning model from the training device 101 in advance and stores it in the storage unit 103a. The image processing device 103 also has a function of adjusting the intensity of blur sharpening by taking a weighted average of the captured image and the blur-sharpened image. Details of training and estimation of the machine learning model and adjusting the intensity of blur sharpening will be described later. A user can adjust the intensity of blur sharpening while checking the image displayed on the display device 104. The blur-sharpened image after intensity adjustment is stored in the storage unit 103a or the recording medium 105 and, as necessary, output to an output device 106 such as a printer. The captured image may be grayscale or have multiple color components. Furthermore, the captured image may be an undeveloped RAW image or a developed image.

[0019] Next, with reference to Figures 4(A) to (C), a description will be given of the degradation of estimation accuracy that occurs when blur sharpening is performed using a machine learning model. Figures 4(A) to (C) are explanatory diagrams of the adverse effects of sharpening, showing spatial changes in image signal values. Here, since the image is an 8-bit image, the saturation value is 255. The solid lines in Figures 4(A) to (C) are captured images (blurred images), and the dotted lines are blur-sharpened images in which the blur of the captured images has been sharpened using a machine learning model.

[0020] Figure 4(A) shows the results of sharpening a non-saturated object with significant blur due to optical system aberrations. Figure 4(B) shows a non-saturated object with minimal blur due to optical system aberrations. Figure 4(C) shows the results of sharpening a saturated object with minimal blur due to optical system aberrations. When the blur due to optical system aberrations is significant, undershoot occurs on the dark side of the edge. Furthermore, even when the blur due to optical system aberrations is minimal, sharpening a saturated object results in undershoots that did not occur in non-saturated objects and a decrease in pixel values ​​that were originally saturated. In saturated areas, information about the structure of the object space is lost, and false edges can appear at the boundaries between areas, making it impossible to extract accurate feature values ​​for the object. This reduces the estimation accuracy of the machine learning model. These results demonstrate that the adverse effects of sharpening depend on the performance of the optical system and the saturated area.

[0021] The aforementioned correction uses a machine learning model that incorporates a method of using a captured image and a brightness saturation map corresponding to the captured image as input data for the machine learning model, and a method of generating a saturation influence map. In other words, while it is possible to reduce adverse effects by using these methods, it is difficult to completely eliminate them. The method of using a brightness saturation map and the method of generating a saturation influence map will be described in detail below.

[0022] We will now explain the brightness saturation map. A brightness saturation map is a map that represents brightness saturated areas in a captured image. In areas where brightness saturation occurs (brightness saturated areas), information about the structure of the subject space is lost, and false edges may appear at the boundaries of each area, making it impossible to extract correct feature quantities for the subject. Therefore, by inputting a brightness saturation map, the neural network can identify problematic areas such as those mentioned above, thereby preventing a decrease in estimation accuracy.

[0023] Next, the saturation influence map will be described. Even when a brightness saturation map is used, the machine learning model may not make an accurate judgment. For example, if a region of interest is near a brightness saturated region, the machine learning model can determine that the region of interest is affected by brightness saturation because there is a brightness saturated region near the region of interest. However, if the region of interest is located far from the brightness saturated region, it is not easy to determine whether the region is affected by brightness saturation, and ambiguity increases. As a result, the machine learning model may make an incorrect judgment at a position far from the brightness saturated region. For this reason, when the task is blur sharpening, a sharpening process corresponding to a saturated blur image is performed on a non-saturated blur image. In this case, artifacts occur in the image with the sharpened blur, reducing the accuracy of the task. Therefore, it is preferable to use a machine learning model to generate a saturation influence map from a captured image in which blur occurs.

[0024] A saturation influence map is a map (a spatially arranged signal sequence) that represents the magnitude and range of signal values ​​that have spread due to blurring of subjects in brightness-saturated areas of a captured image. In other words, a saturation influence map is information based on saturated areas in a captured image. By having a machine learning model generate a saturation influence map, the machine learning model can accurately estimate the presence and magnitude of the influence of brightness saturation in a captured image. By generating a saturation influence map, the machine learning model can execute processing that should be executed on areas affected by brightness saturation and processing that should be executed on other areas, respectively, in the appropriate areas. Therefore, by having a machine learning model generate a saturation influence map, task accuracy is improved compared to when the generation of a saturation influence map is not involved (i.e., only recognition labels and blur-sharpened images are generated directly from the captured image).

[0025] While the above two methods are effective, as explained with reference to FIGS. 4A to 4C, it is difficult to completely eliminate the adverse effects. Therefore, the adverse effects are suppressed by taking a weighted average of the captured image and the blur-sharpened image. The dashed-dotted line in FIG. 4A represents the signal value obtained by taking the weighted average of the captured image and the blur-sharpened image. By taking the weighted average, the blur-sharpening effect is maintained while the undershoot in the dark areas is reduced. In this embodiment, the weight map used when taking the weighted average of the captured image and the blur-sharpened image is generated based on the performance of the optical system and the brightness saturation region. By taking the weight map based on the performance of the optical system and the brightness saturation region, it is possible to suppress the adverse effects only in FIGS. 4A and 4C, where the adverse effects occur, while maintaining the sharpness in FIG. 4B, where the adverse effects do not occur. This makes it possible to control the sharpness of the blur and the adverse effects.

[0026] Next, training of the machine learning model executed by the training device 101 will be described with reference to Fig. 5. Fig. 5 is a flowchart of training of the machine learning model. The training device 101 has a storage unit 101a, an acquisition unit 101b, a calculation unit 101c, and an update unit 101d, and any of these components executes the following steps.

[0027] First, in step S101, the acquisition unit 101b acquires one or more original images from the storage unit 101a. The original image is an image having a signal value higher than the second signal value. The second signal value is a signal value corresponding to the saturation brightness value of the captured image. However, since the signal value may be normalized when input to the machine learning model, the second signal value does not necessarily have to match the saturation brightness value of the captured image. Since the machine learning model is trained based on the original image, it is desirable that the original image be an image having various frequency components (edges of different directions and intensities, gradations, flat areas, etc.). The original image may be a real-life image or CG (Computer Graphics).

[0028] Next, in step S102, the calculation unit 101c blurs the original image to generate a blurred image. The blurred image is an image input to the machine learning model during training and corresponds to the captured image during estimation. The blur to be added is the blur to be sharpened. In this embodiment, blur generated by aberration and diffraction of the optical system 102a and the optical low-pass filter of the image sensor 102b is added. The shape of the blur caused by aberration and diffraction of the optical system 102a varies depending on the image plane coordinates (image height and azimuth). It also varies depending on the magnification, aperture, and focus state of the optical system 102a. To simultaneously train a machine learning model that sharpens all of these blurs, it is recommended to generate multiple blurred images using multiple blurs generated by the optical system 102a. Furthermore, in the blurred image, signal values ​​exceeding a second signal value are clipped. This is done to reproduce brightness saturation that occurs during the capture process of the captured image. If necessary, noise generated by the image sensor 102b may be added to the blurred image.

[0029] Next, in step S103, the calculation unit 101c sets a first region based on an image based on the original image and a signal value threshold. In this embodiment, a blurred image is used as the image based on the original image, but the original image itself may also be used. The first region is set by comparing the signal value of the blurred image with the signal value threshold. More specifically, the first region is defined as a region where the signal value of the blurred image is equal to or greater than the signal value threshold. In the first embodiment, the signal value threshold is the second signal value. Therefore, the first region represents a brightness-saturated region of the blurred image. However, the signal value threshold and the second signal value do not necessarily have to match. The signal value threshold may be set to a value slightly smaller than the second signal value (for example, 0.9 times).

[0030] Next, in step S104, the calculation unit 101c generates a first region image having the signal value of the original image in the first region. The first region image has a signal value different from that of the original image in regions other than the first region. More preferably, the first region image has a first signal value in regions other than the first region. In this embodiment, the first signal value is 0, but the invention is not limited to this. In Example 1, the first region image has the signal value of the original image only in regions where the blurred image is brightness saturated, and the signal value in other regions is 0.

[0031] Next, in step S105, the calculation unit 101c blurs the first region image to generate a saturation influence correct map. The blur to be applied is the same as the blur applied to the blurred image. As a result, a saturation influence correct map is generated, which is a map (a spatially arranged signal sequence) that represents the magnitude and range of signal values ​​that have expanded due to deterioration during imaging from the subject in the brightness-saturated region of the blurred image. In the first embodiment, the saturation influence correct map is clipped with the second signal value, as with the blurred image, but clipping is not necessarily required.

[0032] Next, in step S106, the acquisition unit 101b acquires the correct model output. In this embodiment, since the task is blur sharpening, the correct model output is an image with less blur than the blurred image. In embodiment 1, the correct model output is generated by clipping the original image with the second signal value. If the original image lacks high-frequency components, an image obtained by reducing the original image may be used as the correct model output. In this case, reduction is also performed similarly when generating the blurred image in step S102. Note that step S106 may be executed any time after step S101 and before step S107.

[0033] Next, in step S107, the calculation unit 101c generates a saturation influence map and a model output based on the blurred image using a machine learning model. FIG. 1 is a configuration diagram of the machine learning model. Note that in this embodiment, the machine learning model shown in FIG. 1 is used, but is not limited to this. A blurred image 201 and a brightness saturation map 202 are input to the machine learning model. The brightness saturation map 202 is a map that indicates brightness saturated areas of the blurred image 201 (where the signal value is equal to or greater than a second signal value). For example, the brightness saturation map 202 can be generated by binarizing the blurred image 201 using the second signal value. However, the brightness saturation map 202 is not essential. The blurred image 201 and the brightness saturation map 202 are linked in the channel direction and input to the machine learning model. However, the invention is not limited to this. For example, the blurred image 201 and the brightness saturation map 202 may be converted into feature maps, and these feature maps may be linked in the channel direction. Furthermore, information other than the brightness saturation map 202 may be added to the input.

[0034] The machine learning model has multiple layers, and in each layer, a linear sum of the layer's input and weights is calculated. The initial values ​​of the weights may be determined using random numbers or the like. In the first embodiment, the machine learning model is a CNN that uses the convolution of the input and a filter as the linear sum (the value of each element of the filter corresponds to the weight, and may also include a sum with a bias). However, the invention is not limited to this. Furthermore, in each layer, nonlinear transformation is performed using an activation function such as a ReLU (Rectified Linear Unit) or a sigmoid function as necessary. Furthermore, the machine learning model may have a residual block or a skip connection (also called a shortcut connection) as necessary. As a result of passing through multiple layers (16 convolutional layers in this embodiment), a saturation influence map 203 is generated. In this embodiment, the saturation influence map 203 is generated by taking the element-by-element sum of the output of layer 211 and the intensity saturation map 202, but this is not limited to this. The saturation influence map may also be generated directly as the output of layer 211. Alternatively, the saturation influence map 203 may be the result of performing any processing on the output of the layer 211 .

[0035] Next, the saturation influence map 203 and the blurred image 201 are concatenated in the channel direction and input to a subsequent layer, and as a result of passing through multiple layers (16 convolutional layers in the first embodiment), a model output 204 is generated. The model output 204 is also generated by taking the sum of the output of layer 212 and the blurred image 201 for each element, but the configuration is not limited to this. Note that in the first embodiment, convolution is performed with 64 types of 3×3 filters in each layer (however, in the layers 211 and 212, the number of filter types is the same as the number of channels of the blurred image 201), but the configuration is not limited to this.

[0036] Next, in step S108, the update unit 101d updates the weights of the machine learning model based on the error function. In the first embodiment, the error function is a weighted sum of the error between the saturation influence map 203 and the saturation influence correct map and the error between the model output 204 and the correct model output. MSE (Mean Squared Error) is used to calculate the error. Both weights are set to 1. However, the error function and weights are not limited to this. Backpropagation or the like may be used to update the weights. The error may also be taken for the residual component. In the case of the residual component, the error between the difference component between the saturation influence map 203 and the brightness saturation map 202 and the difference component between the saturation influence correct map and the brightness saturation map 202 is used. Similarly, the error between the difference component between the model output 204 and the blurred image 201 and the difference component between the correct model output and the blurred image 201 is used.

[0037] Next, in step S109, the update unit 101d determines whether training of the machine learning model is complete. Completion can be determined by, for example, whether the number of iterations of weight update has reached a predetermined number, or whether the amount of change in weight during update is smaller than a predetermined value. If it is determined in step S109 that training is not complete, the process returns to step S101, and the acquisition unit 101b acquires one or more new original images. On the other hand, if it is determined that training is complete, the update unit 101d ends training and stores the configuration of the machine learning model and information about the weights in the storage unit 101a.

[0038] The above training method enables the machine learning model to estimate a saturation influence map that represents the magnitude and range of signal values ​​that have spread due to blurring of subjects in brightness-saturated regions of a blurred image (a captured image at the time of estimation). Explicitly estimating the saturation influence map enables the machine learning model to sharpen the blur for both saturated and non-saturated blurred images in the appropriate regions, thereby suppressing the occurrence of artifacts.

[0039] Next, blur sharpening of a captured image using a trained machine learning model executed by the image processing device 103 will be described with reference to Fig. 6. Fig. 6 is a flowchart of generating a model output. The image processing device 103 has a storage unit 103a, an acquisition unit (acquisition means) 103b, and a sharpening unit (first generation means, second generation means) 103c, any of which executes the following steps.

[0040] First, in step S201, the acquisition unit 103b acquires a captured image obtained through the optical system 102a and a machine learning model. Information on the configuration and weights of the machine learning model is acquired from the storage unit 103a. Next, in step S202, the sharpening unit (first generation means) 103c uses the machine learning model to generate a blur-sharpened image (model output, first image) from the captured image, in which the blur of the captured image has been sharpened. That is, the sharpening unit 103c corrects the blur component of the captured image to generate the first image. Note that the machine learning model has the configuration shown in FIG. 1, as in training. As in training, a brightness saturation map representing brightness-saturated regions of the captured image is generated and input, and a saturation influence map and model output are generated.

[0041] Next, the synthesis of a captured image and a model output (adjustment of the sharpening strength) executed by the image processing device 103 will be described with reference to Fig. 7. Fig. 7 is a flowchart of the sharpening strength adjustment.

[0042] First, in step S211, the acquisition unit 103b acquires the shooting state from the captured image. The shooting state is (z, f, d) indicating the zoom position, aperture diameter, and subject distance of the optical system 102a, and the pixel pitch of the image sensor 102b.

[0043] Next, in step S212, the acquisition unit 103b acquires information (optical performance index) related to the optical performance of the optical system 102a based on the shooting conditions acquired in step S211. The optical performance index is stored in the storage unit 103a. The optical performance index is information related to the optical performance of the optical system 102a used to capture the captured image, independent of the object space, and does not include information that is not independent of the object space, such as a saturation influence map. In this embodiment, the magnitude (peak value) and range (extent) of the point spread function (PSF) are used as the optical performance index. The peak value is the maximum signal value of the PSF, and the range refers to the number of pixels having a value equal to or greater than a certain threshold. When sharpening blur using a machine learning model, even if the peak value is the same, blur with a smaller number of pixels having a value equal to or greater than a certain threshold causes less adverse effects. Therefore, in this embodiment, the optical performance index is used. Furthermore, because the peak value and range of the PSF depend on the pixel pitch of the image capture device 102, optical performance indexes corresponding to multiple pixel pitches are stored, and intermediate values ​​are generated by interpolation. The optical performance index may be any index that reflects the optical performance, and may be a different numerical value.

[0044] FIG. 8 is an explanatory diagram of optical performance index division points, showing the image circle 111 of the optical system 102a, the effective pixel area 112 of the image sensor 102b, and the optical performance index 114 stored in the storage unit 103a. In this embodiment, the optical performance index 114 (peak value of the PSF) for 10 points from the on-axis to the most off-axis is acquired. Note that in this embodiment, the optical performance index is set to 10 points, but the number of divisions may be increased or decreased. Furthermore, if the effective pixel area 113 is smaller than the effective pixel area 112 of the image sensor 102b, the necessary optical performance index may be acquired according to its size. Alternatively, the optical performance index up to the most off-axis may be acquired in advance, a weight map may be generated, and then the weight map may be cut out to fit the effective pixel area.

[0045] Here, in order to reduce the number of optical performance indices (number of data), only optical performance indices for discretely selected imaging states are stored in the storage unit 103a. Therefore, if an optical performance index corresponding to the imaging state acquired in step S211 or an optical performance index corresponding to an imaging state close to the imaging state is not stored in the storage unit 103a, an optical performance index as close as possible to the imaging state is selected. Then, the optical performance index that is actually used is created by correcting the optical performance index so as to be optimized for the imaging state acquired in step S211.

[0046] FIG. 9 is a schematic diagram of an imaging state space in which optical performance indices are arranged, and schematically shows optical performance indices for discretely selected imaging states stored in the storage unit 103a. As described above, the optical performance indices stored in the storage unit 103a are discretely arranged in an imaging state space whose axes are the three imaging states: zoom position (state A), aperture diameter (state B), and subject distance (state C). The coordinates of each point (black circle) in the imaging state space indicate the optical performance indices stored in the storage unit 103a. Note that, although FIG. 9 shows the optical performance indices arranged at lattice points on lines orthogonal to each imaging state, the optical performance indices may also be arranged outside the lattice points.

[0047] The types of imaging states are not limited to zoom position, iris aperture diameter, and subject distance. Furthermore, the number of types of imaging states does not have to be three. A four-dimensional or higher imaging state space may be configured using four or more imaging states, and optical performance indicators may be discretely arranged within the space. Alternatively, the types of imaging states may be one or two of zoom position, iris aperture diameter, and subject distance.

[0048] Next, a specific method for selecting and creating (correcting) an optical performance index will be described. In Fig. 9, the imaging state indicated by a large white circle is assumed to be the actual imaging state acquired in step S211. If an optical performance index stored (stored) in the storage unit 103a exists in or near the actual imaging state, that optical performance index is selected. If no optical performance index exists in or near the actual imaging state, the optical performance index is selected or created (corrected) by the following method.

[0049] First, the acquisition unit 103b calculates the distances in the imaging state space between the actual imaging state and imaging states corresponding to the plurality of optical performance indices. Then, it selects the optical performance index corresponding to the imaging state with the shortest distance among the calculated distances. By selecting such an optical performance index, the difference (state difference) between the actual imaging state and the imaging state corresponding to the optical performance index becomes the smallest. In FIG. 9, it is assumed that the stored filter corresponding to the imaging state indicated by the small white circle is selected.

[0050] Next, the acquisition unit 103b calculates state difference amounts ΔA, ΔB, and ΔC between the imaging state corresponding to the selected optical performance index and the actual imaging state. The acquisition unit 103b then calculates a state correction coefficient based on the state difference amounts ΔA, ΔB, and ΔC. The acquisition unit 103b then corrects the selected optical performance index using the state correction coefficient. This makes it possible to create an optical performance index that corresponds to the actual imaging state. Alternatively, as an alternative method, multiple optical performance indexes located near the actual imaging state are selected, and the multiple optical performance indexes are interpolated according to the state difference amounts, making it possible to create an optical performance index that is suitable for the imaging state.

[0051] Next, in step S213, the sharpening unit 103c generates a second weight map and a third weight map from the optical performance index. That is, the sharpening unit 103c generates a second weight map (second weight information) and a third weight map (third weight information) having different weights based on information related to optical performance. Here, the generation of the weight maps will be described in detail. The weight map is used to determine the proportion of each image when weighting the captured image and the blur-sharpened image, and has continuous signal values ​​ranging from 0 to 1. For example, if the value of the weight map determines the proportion of the blur-sharpened image, if the value is 0, the weighted average image remains the captured image, and if the value of the weight map is 0.5, the weighted average image is obtained by adding 50% of the pixel values ​​of the captured image and the blur-sharpened image. In this embodiment, the value of the weight map represents the weight of the blur-sharpened image.

[0052] When determining the weights of the second weight map and the third weight map, the weights of the blur-sharpened image are determined for each image height based on the optical performance index acquired in step S212 and the relational expression shown in Fig. 10. Fig. 10 is a diagram showing the relational expression between the optical performance index and the weights. In Fig. 10, the horizontal axis represents the optical performance index, and the vertical axis represents the weights of the blur-sharpened image.

[0053] The solid line 121 is for the non-saturated region, and the dotted line 122 is for the saturated region. Even with the same optical performance index, the weight of the blur-sharpened image is reduced in the saturated region, where adverse effects are likely to occur. In this embodiment, a linear equation is used, but the relational expression is not limited to linear expressions. The relational expression can also be freely changed. Note that image heights where an optical performance index is not held are generated by interpolation from image height points where an optical performance index is held. A second weight map for the non-saturated region is generated from the solid line 121, and a third weight map for the saturated region is generated from the dotted line 122. As examples, FIG. 11(A) shows the second weight map, and FIG. 11(B) shows the third weight map. The brighter the weight map, the higher the weight of the output image, and the darker the weight map, the lower the weight of the output image. In a typical optical system, optical performance deteriorates as one moves off-axis, often resulting in a gradation like that shown in FIGS. 11(A) and 11(B).

[0054] Next, in step S214, the sharpening unit 103c generates a first weight map based on the second weight map, the third weight map, and the saturation influence map. That is, the first weight map is generated based on information about optical performance (the second weight map and the third weight map) and information about saturated regions in the captured image (information based on the saturated regions, i.e., the saturation influence map). In this embodiment, the information about saturated regions is the saturation influence map, and not all RGB may be saturated. Furthermore, the saturation influence map is normalized by the second signal value and used to combine the second weight map and the third weight map. That is, if the normalized value is between 0 and 1, both the second weight map and the third weight map contribute.

[0055] As an example, FIG. 12(A) shows a captured image, and FIG. 12(B) shows a saturation influence map. FIG. 13 shows a first weight map. The first weight map is generated by combining the second weight map and the third weight map based on a normalized saturation influence map (information about the saturated region). This allows the weight of the blur-sharpened image to be reduced in the saturation influence region, where adverse effects are likely to occur. Note that normalizing the saturation influence map output by the machine learning model with the second signal value is not essential; it is also possible to adjust the balance between the blur-sharpening effect and adverse effects by changing or blurring the normalized signal value. Alternatively, a brightness saturation map blurred for each image height may be used, using the blur of the optical system acting on the captured image.

[0056] Next, in step S215, the sharpening unit (second generating means) 103c calculates a weighted average of the captured image and the blur-sharpened image (model output, first image) based on the first weight map to generate an intensity-adjusted image (second image) 205. By combining the captured image and the blur-sharpened image based on the first weight map, it is possible to increase the weight of the captured image only around areas where adverse effects are likely to occur, while suppressing a decrease in the blur-sharpening effect.

[0057] With the above configuration, it is possible to provide an image processing system that can control the sharpness and adverse effects of blur in a regression task using machine learning on a blurred image. [Example]

[0058] Next, an image processing system 300 according to a second embodiment of the present invention will be described. In this embodiment, the third map is not generated, and the first weight map is generated by lowering the weight of the saturation-affected region in the second weight map. Furthermore, the intensity of blur sharpening is adjusted not only based on the performance of the optical system and the saturation-affected region, but also on the subject distance and ISO sensitivity.

[0059] Fig. 14 is a block diagram of the image processing system 300. Fig. 15 is an external view of the image processing system 300. The image processing system 300 has a training device 301, an imaging device 302, and an image processing device 303. The training device 301 and the image processing device 303, and the image processing device 303 and the imaging device 302 are each connected via a wired or wireless network. The imaging device 302 has an optical system 321, an imaging element 322, a storage unit 323, a communication unit 324, and a display unit 325. The captured image is transmitted to the image processing device 303 via the communication unit 324.

[0060] The image processing device 303 receives the captured image via the communication unit 332 and performs blur sharpening using information on the configuration and weights of the machine learning model stored in the storage unit 331. The information on the configuration and weights of the machine learning model is trained by the training device 301, is acquired in advance from the training device 301, and is stored in the storage unit 331. The image processing device 303 also has a function of adjusting the intensity of blur sharpening. A blur-sharpened image (model output) in which the blur of the captured image has been sharpened, and a weighted average image in which the intensity has been adjusted are transmitted to the imaging device 302, stored in the storage unit 323, and displayed on the display unit 325.

[0061] The generation of training data and weight learning (training phase) performed by the training device 301, the sharpening of blur in captured images using a trained machine learning model (estimation phase) performed by the image processing device 303, and the synthesis of captured images and model output are the same as in Example 1. Therefore, a description of these will be omitted.

[0062] Next, the synthesis of a captured image and a model output (adjustment of the sharpening strength) executed by the image processing device 303 will be described with reference to Fig. 16. Fig. 16 is a flowchart of the sharpening strength adjustment.

[0063] First, in step S311, the acquisition unit 333 acquires the shooting state from the captured image. Subsequently, in step S312, the acquisition unit 333 acquires an optical performance index based on the shooting state acquired in step S311. The optical performance index is stored in the storage unit 331. In this embodiment, the peak value of the PSF is used as the optical performance index.

[0064] Next, in step S313, the sharpening unit 334 generates a second weight map from the optical performance index. Next, in step S314, the sharpening unit 334 generates a first weight map from the saturation influence map and the second weight map. In this embodiment, the first weight map is generated without generating a third weight map. Specifically, the first weight map is generated by lowering the intensity of only the saturation influence region of the second weight map. Note that the third weight map may be generated first, and then the first weight map may be generated by increasing the intensity of regions other than the saturation influence region.

[0065] Next, in step S315, the sharpening unit 334 adjusts the first weight map according to the subject distance and ISO sensitivity. Here, the reason for making adjustments according to the subject distance and ISO sensitivity will be explained. First, adjustments according to the subject distance are preferably performed when the subject distance is near the minimum focusing distance of the optical system. In a typical optical system, performance is low near the minimum focusing distance, so the amount of correction required for sharpening is large. When a user manually focuses, some users prioritize resolution and others prioritize chromatic aberration. Therefore, if there is a difference between the blur learned by the machine learning model and the blur in the actual captured image, adverse effects are more likely to occur compared to other subject distances. Therefore, when the subject distance is near the minimum focusing distance of the optical system, adverse effects can be suppressed by reducing the strength of sharpening.

[0066] Next, we will explain adjustments according to ISO sensitivity. When sharpening blur using a machine learning model, it may not be possible to distinguish between the subject and noise, resulting in noise amplification. Therefore, when shooting at a high ISO with a high amount of noise, reducing the strength of sharpening can prevent adverse effects. Note that adjustments according to subject distance and ISO sensitivity are not required.

[0067] Subsequently, in step S316, the sharpening unit 334 combines the captured image and the blur-sharpened image (model output, first image) based on the first weight map to generate an intensity-adjusted image (second image).

[0068] With the above configuration, it is possible to provide an image processing system that can control the sharpness and adverse effects of blur in a regression task using machine learning on a blurred image. [Example]

[0069] Next, an image processing system 400 according to a third embodiment of the present invention will be described. FIG. 17 is a block diagram of the image processing system 400. FIG. 18 is an external view of the image processing system 400. The image processing system 400 includes a learning device 401, a lens device 402, an imaging device 403, a control device (first device) 404, an image estimation device (second device) 405, and networks 406 and 407. The learning device 401 and the image estimation device 405 are, for example, servers. The control device 404 is a device operated by a user, such as a personal computer or a mobile terminal. The learning device 401 and the image estimation device 405, and the control device 404 and the image estimation device 405, are capable of communicating with each other.

[0070] The learning device 401 has a storage unit 401a, an acquisition unit 401b, a calculation unit 401c, and an update unit 401d, and learns the weights of a machine learning model that sharpens blur from a captured image captured using a lens device 402 and an imaging device 403. The learning method is the same as in Example 1, and therefore will not be described here. The imaging device 403 has an imaging element 403a, which photoelectrically converts an optical image formed by the lens device 402 to acquire the captured image. The lens device 402 and the imaging device 403 are detachable, and can be combined with a plurality of types of each other.

[0071] The control device 404 has a communication unit 404a, a display unit 404b, a storage unit 404c, and an acquisition unit 404d, and controls, in accordance with a user's operation, the processing to be executed on the captured image acquired from the wired or wirelessly connected imaging device 403. Alternatively, the captured image captured by the imaging device 403 may be stored in advance in the storage unit 404c, and the captured image may be read out.

[0072] The image estimation device 405 includes a communication unit 405a, an acquisition unit 405b, a storage unit 405c, and a sharpening unit 405d. The image estimation device 405 executes a process of sharpening the blur of a captured image in response to a request from a control device 404 connected via a network 406. The image estimation device 405 acquires information on learned weights from a learning device 401 connected via the network 406 when estimating blur sharpening or in advance, and uses the information to estimate blur sharpening of a captured image. The estimated image after blur sharpening estimation is subjected to sharpening intensity adjustment, then transmitted again to the control device 404, stored in the storage unit 404c, and displayed on the display unit 404b. Note that the generation of learning data and weight learning (learning phase) performed by the learning device 401 are the same as those in the first embodiment, and therefore will not be described here.

[0073] Next, blur sharpening of a captured image executed by the control device 404 and the image estimation device 405 will be described with reference to Fig. 19. Fig. 19 is a flowchart of model output and sharpening strength adjustment.

[0074] First, in step S401, the acquisition unit 404d acquires a captured image and a sharpening strength specified by a user. Then, in step S402, the communication unit (transmission means) 404a transmits the captured image and a request for execution of blur sharpening estimation processing to the image estimation device 405.

[0075] Next, in step S403, the communication unit (receiving means) 405a receives and acquires the captured image and processing request transmitted from the control device 404. Next, in step S404, the acquisition unit 405b acquires information on learned weights corresponding to the captured image from the storage unit 405c. The weight information has been read out in advance from the storage unit 401a and stored in the storage unit 405c.

[0076] Next, in step S405, the sharpening unit 405d uses the machine learning model to generate a blur-sharpened image (model output) in which the blur of the captured image has been sharpened from the captured image. The machine learning model has the same configuration as in training, as shown in Fig. 1. As in training, a brightness saturation map representing brightness-saturated regions of the captured image is generated and input, and a saturation influence map and model output are generated.

[0077] Next, in step S406, the sharpening unit 405d generates a first weight map based on the request transmitted from the control device 404. The method of generating the first weight map is the same as in the first embodiment. The intensity can be adjusted by changing the relational expression shown in FIG. 10 according to the intensity of sharpening specified by the user. Next, in step S407, the sharpening unit 405d combines the captured image and the blur-sharpened image (model output) based on the first map. Next, in step S408, the communication unit 405a transmits the combined image to the control device 404.

[0078] Subsequently, in step S409, the communication unit 404a acquires the estimated image transmitted from the image estimation device 405.

[0079] With the above configuration, it is possible to provide an image processing system that can control the sharpness and adverse effects of blur in a regression task using machine learning on a blurred image.

[0080] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0081] According to each embodiment, it is possible to provide an image processing method, an image processing device, an image processing system, and a program that are capable of controlling the sharpness and adverse effects of image blur in accordance with the performance of the optical system.

[0082] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0083] 103 Image processing device 103b Acquisition unit (acquisition means) 103c sharpening unit (first generating means, second generating means)

Claims

1. generating a first image by correcting a blur component of a captured image obtained by imaging using an optical system; obtaining information about the optical performance of the optical system; generating a second image based on the captured image, the first image, information about the optical performance, and information about a saturated region of the captured image, An image processing method characterized in that the information regarding the saturated region is information representing a blurred region surrounding a brightness saturated region of the captured image, which is generated by inputting the captured image into a machine learning model.

2. In the step of generating the second image, the second image is generated based on the captured image, the first image, and first weighting information; 2. The image processing method according to claim 1, wherein the first weight information is generated based on information relating to the optical performance and information relating to a saturated region of the captured image.

3. The image processing method described in Claim 2, characterized in that the first weight information is generated by adjusting second weight information obtained based on information regarding the optical performance based on information regarding the saturation region.

4. An image processing method as described in claim 2 or 3, characterized in that the second image is generated by taking a weighted average of the captured image and the first image using the first weight information.

5. 5. The image processing method according to claim 1, wherein the information about the optical performance is calculated based on information about at least one of a zoom position, an aperture diameter, or a subject distance of the optical system, and information about a point spread function for each image height of the optical system.

6. generating a first image by correcting a blur component of a captured image obtained by imaging using an optical system; obtaining information about the optical performance of the optical system; generating a second image based on the captured image, the first image, and first weighting information; the first weighting information is generated by combining second weighting information and third weighting information obtained based on the information on the optical performance, based on information on a saturated region of the captured image; An image processing method characterized in that the information regarding the saturated region is information representing a blurred region surrounding a brightness saturated region of the captured image, which is generated by inputting the captured image into a machine learning model.

7. 7. The image processing method according to claim 6, wherein the weights of the first image represented by the second weight information and the third weight information are different from each other.

8. 8. The image processing method according to claim 6, wherein the second weight information determines a weight for a non-saturated region in the captured image, and the third weight information determines a weight for a saturated region in the captured image.

9. 9. The image processing method according to claim 6, wherein the weights of the first image represented by the second weight information and the third weight information differ for each image height.

10. A program causing a computer to execute the image processing method according to any one of claims 1 to 9.

11. a first generation means for generating a first image by correcting a blur component of a captured image obtained by imaging using an optical system; an acquisition means for acquiring information about the optical performance of the optical system; a second generation means for generating a second image based on the captured image, the first image, information on the optical performance, and information on a saturated region of the captured image, The image processing device is characterized in that the information regarding the saturated region is information representing a blurred region surrounding a brightness saturated region of the captured image, which is generated by inputting the captured image into a machine learning model.

12. An image processing system comprising the image processing device according to claim 11 and a control device capable of communicating with the image processing device, the control device has a transmission means for transmitting a request to the image processing device to execute processing on the captured image, The image processing system is characterized in that the image processing device has a receiving means for receiving the request, and executes processing on the captured image in response to the request.

Images