Image processing method, image processing apparatus, image processing system, and program

Abstract

To provide an image processing method capable of maintaining a blur correction effect while suppressing adverse effects caused by an aperture value of an optical system and a luminance value of a captured image.SOLUTION: The image processing method includes the steps of: generating information regarding the correction of a captured image based on the captured image; and generating an intensity-adjusted image based on the captured image, the information regarding correction, and a weight map, wherein the weight map is generated based on information regarding an aperture value of an optical system used to capture the captured image and information regarding a luminance value of the captured image.SELECTED DRAWING: Figure 16

Description

Technical Field

[0005] ,

[0001] The present invention relates to an image processing method, an image processing apparatus, an image processing system, and a program.

Background Art

[0002] In tasks of recognition or regression for images, a method using a machine learning model can achieve high accuracy compared to a theory-based method using assumptions and approximations. In the theory-based method, the accuracy decreases due to elements ignored by the assumptions and approximations. However, in the method using a machine learning model, by learning the machine learning model using learning data including those elements, an estimation in line with the learning data without assumptions and approximations can be realized, so the accuracy of the task is improved.

[0003] Patent Document 1 discloses a method of sharpening the blur of a captured image using a convolutional neural network (CNN), which is one of the machine learning models. Patent Document 1 also discloses a method of weighted averaging a captured image and an estimated image (sharpened blurred image) based on a luminance saturation region to adjust the intensity of sharpening.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The method disclosed in Patent Document 1 does not address the drawbacks caused by the diffraction of light. When imaging a high-brightness subject by increasing the aperture value (F-number) of the optical system, Airy disks and light rays occur due to the diffraction of light. When Airy disks and light rays are corrected using a machine learning model, drawbacks such as unnatural enhancement may occur due to insufficient accuracy of the training images or insufficient parameters in the machine learning model. Furthermore, the prominence of these drawbacks varies depending on the peripheral brightness value of the area being corrected in the captured image. For example, unnatural enhancement is noticeable in dark images such as night scenes, but not in bright images taken outdoors during the day.

[0006] Furthermore, since the occurrence of Airy disks and light rays depends on the aperture value of the optical system, the correction effect on images captured at aperture values ​​where no adverse effects occur is also reduced. In addition, the prominence of adverse effects differs depending on the brightness value of the captured image, and is averaged according to predetermined weights. In other words, while adverse effects are reduced around the saturation region in dark images, the correction effect around the saturation region is reduced more than necessary in bright images.

[0007] Therefore, the present invention aims to provide an image processing method that can maintain the blur correction effect while suppressing the adverse effects caused by the aperture value of the optical system and the brightness value of the captured image. [Means for solving the problem]

[0008] One aspect of the present invention is an image processing method comprising the steps of generating information regarding the correction of an captured image based on the captured image, and generating an intensity-adjusted image based on the captured image, the correction information, and a weight map, wherein the weight map is generated based on information regarding the aperture value of the optical system used to capture the captured image and information regarding the brightness value of the captured image.

[0009] Other objects and features of the present invention are described in the following examples. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide an image processing method that can maintain the blur correction effect while suppressing the adverse effects caused by the aperture value of the optical system and the brightness value of the captured image. [Brief explanation of the drawing]

[0011] [Figure 1] This is an explanatory diagram of the machine learning model in Example 1. [Figure 2] This is a block diagram of the image processing system in Example 1. [Figure 3] This is an external view of the image processing system in Example 1. [Figure 4] This diagram illustrates the adverse effects of sharpening in Examples 1-3. [Figure 5] This is an explanatory diagram of the Airy disk in Examples 1 to 3. [Figure 6] This is an explanatory diagram of the light rays in Examples 1 to 3. [Figure 7] This is a flowchart of the machine learning model training in Examples 1-3. [Figure 8] This is a flowchart showing the generation of model outputs in Examples 1 and 2. [Figure 9] This is a flowchart for adjusting the sharpening intensity in Example 1. [Figure 10] This is an explanatory diagram of the captured image and saturation effect map in Example 1. [Figure 11] This is an explanatory diagram of the captured image and saturation effect map in Example 1. [Figure 12] This is an explanatory diagram of the weight map in Example 1. [Figure 13] This is an explanatory diagram of the weight map in Example 1. [Figure 14] This is a block diagram of the image processing system in Example 2. [Figure 15] This is an external view of the image processing system in Example 2. [Figure 16] This is a flowchart for adjusting the sharpening intensity in Example 2. [Figure 17]It is an explanatory diagram of the weight map in Example 2. [Figure 18] It is a block diagram of the image processing system in Example 3. [Figure 19] It is an external view of the image processing system in Example 3. [Figure 20] It is a flowchart of model output and intensity adjustment of sharpening in Example 3.

Modes for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and duplicate descriptions are omitted.

[0013] Before giving a specific description of the embodiments, the gist of the present invention will be described. The present invention generates an estimated image in which blurring caused by an optical system is sharpened using a machine learning model from a captured image captured using the optical system. Then, a weight map is generated based on information regarding the aperture value of the optical system used for capturing the captured image and information regarding the luminance value of the captured image, and the captured image and the estimated image are weighted-averaged. The weight map is used to determine the ratio of each image when weighted-averaging the captured image and the estimated image, and has continuous signal values. For example, when the numerical value of the weight map determines the ratio of the captured image, if the numerical value is 0.5, an intensity adjustment image obtained by weighted-averaging the captured image and the blurred and sharpened image at a ratio of 50% is obtained. Also, if the value of the weight map is 1, the intensity adjustment image is the captured image.

[0014] Blurring caused by the optical system includes aberration, diffraction, defocus blurring, the action of an optical low-pass filter, pixel aperture degradation of an image sensor, and the like. The machine learning model includes, for example, a neural network, genetic programming, a Bayesian network, and the like. The neural network includes CNN (Convolutional Neural Network), GAN (Generative Adversarial Network), RNN (Recurrent Neural Network), and the like.

[0015] Blur sharpening refers to the process of restoring frequency components of a subject that have been reduced or lost due to blur. When sharpening blur, if a high-brightness subject is imaged by increasing the aperture value of the optical system, Airy disks and light rays will occur due to the diffraction phenomenon of light. When Airy disks and light rays are corrected with a machine learning model, unnatural enhancements and other problems may occur due to insufficient accuracy of the training images or insufficient parameters of the machine learning model. Details on Airy disks, light rays, insufficient accuracy of training images, and insufficient parameters of the machine learning model will be discussed later.

[0016] Furthermore, the visibility of this drawback varies depending on the brightness of the image. For example, unnatural enhancement is noticeable in dark images such as nightscapes, but not in bright images taken outdoors during the day. Therefore, the present invention generates a weight map based on information regarding the aperture value of the optical system used to capture the image and information regarding the brightness value of the captured image, and performs a weighted average of the captured image and the estimated image. This makes it possible to maintain the blur correction effect while suppressing the drawbacks caused by the aperture value of the optical system and the brightness value of the captured image. In the following, the stage of learning the weights of the machine learning model will be referred to as the learning phase, and the stage of sharpening the blur using the machine learning model with the learned weights will be referred to as the estimation phase. [Examples]

[0017] First, the image processing system in Embodiment 1 of the present invention will be described. In this embodiment, the task performed by the machine learning model is to sharpen the blur of an captured image, including luminance saturation (increase the resolution of the captured image). The blur to be sharpened is limited to aberrations and diffractions generated in the optical system, as well as blur caused by optical low-pass filters. However, the effects of the invention can also be obtained when sharpening blur caused by pixel aperture, defocus, or blurring. Furthermore, this embodiment can also be applied to tasks other than blur sharpening, and the effects can be obtained in the same way. Specifically, these include upsampling to increase the number of pixels in the captured image, and tasks to transform (transform) the defocus blur of the captured image. Transformation of defocus blur includes, for example, transformation from double-line blur to Gaussian blur or spherical blur. Double-line blur has a point distribution function (PSF) with separated peaks. As a result, a subject that is originally a single line appears to be double-blurred when defocused. Spherical blur has a PSF with flat intensity. Gaussian bokeh has a Gaussian distribution PSF. Other types of defocus bokeh that can be transformed include, for example, defocus bokeh caused by vignetting and ring-shaped defocus bokeh caused by pupil occlusion such as catadioptric lenses.

[0018] Figure 2 is a block diagram of the image processing system 100 in this embodiment. Figure 3 is an external view of the image processing system 100. The image processing system 100 has a learning device 101 and an image processing device 103 connected by a wired or wireless network. An imaging device 102, a display device 104, a recording medium 105, and an output device 106 are connected to the image processing device 103 by wired or wireless. The captured image, which captures the subject space using the imaging device 102, is input to the image processing device 103. The captured image is blurred due to aberrations and diffraction by the optical system 102a in the imaging device 102 and the optical low-pass filter of the image sensor 102b, and the information of the subject is attenuated.

[0019] The image processing device 103 uses a machine learning model to sharpen the blur of an captured image and generates a saturation effect map and a blur-sharpened image (model output). Details of the saturation effect map will be described later. The machine learning model is trained by the learning device 101, and the image processing device 103 has previously acquired information about the machine learning model from the learning device 101 and stored it in the storage unit 103a. The image processing device 103 also has a function to adjust the intensity of blur sharpening by taking a weighted addition of the captured image and the blur-sharpened image. Details of the training and estimation of the machine learning model and the adjustment of the blur sharpening intensity will be described later. The user can adjust the blur sharpening intensity while checking the image displayed on the display device 104. The blur-sharpened image with adjusted intensity is stored in the storage unit 103a or the recording medium 105 and output to an output device 106 such as a printer as needed. The captured image may be grayscale or have multiple color components. It may also be an undeveloped RAW image or a developed image.

[0020] Next, referring to Figures 4(A) to (C) and Figure 5, we will explain the drawbacks caused by the diffraction phenomenon of light that occur when performing blur sharpening using a machine learning model. Figures 4(A) and (B) are cross-sectional views of the PSF when the aperture value of the optical system is increased (vertical axis is signal value, horizontal axis is spatial coordinate), and the scale of the vertical axis is different. Figure 5 is a plan view of the PSF. Due to the diffraction phenomenon of light, a diffraction pattern is created with a bright region in the center and dark concentric rings around it. This is called an Airy disk. Although this diffraction pattern has a weak signal value, it can be visually confirmed in the captured image when a point light source or strong light is imaged. In Figure 4(C), the dashed line 141 shows the ideal signal value when the above PSF is sharpened and the diffraction pattern is reduced, and the dotted line 142 shows the signal value when the diffraction pattern is emphasized by sharpening. For blur sharpening, the dashed line 141 is preferable. However, due to insufficient accuracy of the training images or insufficient parameters in the machine learning model, diffraction pattern enhancement may occur, as shown by the dotted line 142. This is because the edges of the diffraction pattern are mistakenly identified as the subject and sharpened. Insufficient accuracy of training images and insufficient parameters in the machine learning model will be discussed later.

[0021] Next, with reference to Figure 6, we will explain light rays caused by the diffraction phenomenon of light. Light rays refer to the long, thin streaks of light that appear when a point light source or strong light is imaged. Figure 6 shows an example of light rays. The number of streaks depends on the number of aperture blades 143 of the optical system 102a. If the number of aperture blades 143 is odd, twice the number of streaks will appear. On the other hand, if the number of aperture blades 143 is even, the number of streaks will be equal to the number of blades. In Figure 6, the number of aperture blades 143 is 5, so 10 light streaks 144 are generated. The sharpness of the light rays depends on the aperture value of the optical system. As the aperture value increases, the sharpness increases. Also, even with the same aperture value, the appearance of light rays differs depending on the type of optical system 102a. Similar to Airy disks, unnatural enhancement of light rays may occur due to insufficient accuracy of the training images or insufficient parameters in the machine learning model.

[0022] Next, we will explain in detail the insufficient accuracy of the training images. When using a machine learning model to sharpen blur, the accuracy of the sharpening depends on the accuracy of the training images. In other words, training images that accurately reproduce the blur of the optical system 102a being sharpened are required. However, if we try to comprehensively learn the blur at the zoom position, aperture value, and subject distance that the optical system 102a can take during imaging, the number of training images becomes enormous. As a result, problems such as a decrease in the accuracy of individual blur sharpening and failure of the learning to converge occur. Therefore, it is necessary to discretely learn the blur at the zoom position, aperture value, and subject distance that the optical system 102a can take, and have the machine learning model predict the intermediate regions. However, in that case, if we sharpen the blur in the intermediate regions that are not included in the training images, the edges of the diffraction pattern may be mistakenly recognized as the subject and sharpened.

[0023] Next, we will explain in detail the parameter shortage in machine learning models. A machine learning model has multiple layers, and in each layer, a linear sum is taken of the layer's input and weights. If we use a CNN as a machine learning model, where the linear sum is the input and the convolution of the filter (the values ​​of each element of the filter correspond to the weights, and may also include the sum with the bias), the number of filter layers corresponds to the number of parameters. In other words, a parameter shortage means a shortage of filter layers. Since there is a trade-off relationship between the number of filter layers, training time, and processing speed, a parameter shortage can occur. Using a machine learning model trained with methods that use captured images and corresponding brightness saturation maps as input data, and methods that generate saturation effect maps, can reduce the negative effects, but it is difficult to completely eliminate them. We will explain in detail the methods that use brightness saturation maps and the methods that generate saturation effect maps.

[0024] Next, let's discuss the luminance saturation map. A luminance saturation map is a map that represents the luminance saturation regions in an captured image. In regions where luminance saturation occurs (luminance saturation regions), information about the structure of the subject space is lost, false edges may appear at the boundaries of each region, and it becomes impossible to extract the correct features of the subject. Therefore, by inputting a luminance saturation map, the neural network can identify the problematic regions mentioned above, thereby suppressing the decrease in estimation accuracy.

[0025] Next, we will explain saturation effect maps. Even when using luminance saturation maps, machine learning models may not make correct judgments. For example, if the area of ​​interest is near a luminance-saturated area, the machine learning model can determine that the area of ​​interest is affected by luminance saturation because there is a luminance-saturated area nearby. However, if the area of ​​interest is far from the luminance-saturated area, it is not easy to determine whether or not this area is affected by luminance saturation, and the ambiguity increases. As a result, the machine learning model may make a misjudgment at locations far from the luminance-saturated area. In this case, if the task is blur sharpening, the sharpening process corresponding to the saturated blur image will be performed on the unsaturated blur image. In this case, artifacts occur in the image after the blur has been sharpened, and the accuracy of the task decreases. Therefore, it is preferable to generate a saturation effect map from the captured image in which blur has occurred using a machine learning model.

[0026] A saturation effect map is a map (a spatially arranged sequence of signals) that represents the magnitude and range of signal values ​​that have spread due to blurring in areas of brightness saturation in an image. By having a machine learning model generate a saturation effect map, the model can accurately estimate the presence and magnitude of brightness saturation in the image. Once a saturation effect map is generated, the machine learning model can execute the appropriate processing on the areas affected by brightness saturation and the appropriate processing on the other areas. Therefore, having a machine learning model generate a saturation effect map improves the accuracy of the task compared to cases where a saturation effect map is not generated (where only recognition labels and blur-sharpened images are generated directly from the image).

[0027] While the two methods described above are effective, it is difficult to completely eliminate their drawbacks. Therefore, this embodiment generates a weight map based on information regarding the aperture value of the optical system used to capture the image and information regarding the brightness value of the captured image, and then performs a weighted average of the captured image and the estimated image. This makes it possible to suppress the drawbacks caused by the aperture value of the optical system and the brightness value of the captured image while maintaining the blur correction effect.

[0028] Next, with reference to Figure 7, the training of the machine learning model performed by the learning device 101 will be described. Figure 7 is a flowchart of the training of the machine learning model. The learning device 101 has a storage unit 101a, an acquisition unit 101b, an arithmetic unit 101c, and an update unit 101d, and each step in Figure 7 is mainly performed by the respective parts of the learning device 101.

[0029] First, in step S101, the acquisition unit 101b acquires one or more original images from the storage unit 101a. The original image is an image with a signal value higher than the second signal value. The second signal value is a signal value corresponding to the brightness saturation value of the captured image. However, since the signal value may be normalized when inputting into the machine learning model, the second signal value and the brightness saturation value of the captured image do not necessarily have to match. Since the machine learning model is trained based on the original image, it is desirable that the original image is an image with various frequency components (edges with different directions and intensities, gradients, flat areas, etc.). The original image may be a real photograph or a computer graphics image.

[0030] Next, in step S102, the calculation unit 101c adds blur to the original image to generate a blurred image. The blurred image is the image input to the machine learning model during training and corresponds to the captured image during estimation. The blur added is the blur that will be sharpened. In this embodiment, blur generated by the aberrations and diffraction of the optical system 102a and the optical low-pass filter of the image sensor 102b is added. The shape of the blur due to the aberrations and diffraction of the optical system 102a changes depending on the image plane coordinates (image height and azimuth). It also changes depending on the magnification, aperture, and focus state of the optical system 102a. If you want to train a machine learning model that sharpens all of these blurs at once, it is advisable to generate multiple blurred images using multiple blurs generated by the optical system 102a. In addition, in the blurred image, signal values ​​exceeding the second signal value are clipped. This is done to reproduce the brightness saturation that occurs during the image acquisition process of the captured image. If necessary, noise generated by the image sensor 102b may be added to the blurred image.

[0031] Next, in step S103, the calculation unit 101c sets a first region based on an image based on the original image and a threshold signal value. In this embodiment, a blurred image is used as the image based on the original image, but the original image itself may also be used. The first region is set by comparing the signal value of the blurred image with the threshold signal value. More specifically, the region where the signal value of the blurred image is equal to or greater than the threshold signal value is defined as the first region. In this embodiment, the threshold signal value is the second signal value. Therefore, the first region represents the luminance-saturated region (saturated region) of the blurred image. However, the threshold signal value and the second signal value do not have to be the same. The threshold signal value may be set to a value slightly smaller than the second signal value (for example, 0.9 times).

[0032] Next, in step S104, the calculation unit 101c generates a first region image having the signal values ​​of the original image in the first region. The first region image has different signal values ​​from the original image in regions other than the first region. More preferably, the first region image has the first signal value in regions other than the first region. In this embodiment, the first signal value is 0, but the invention is not limited thereto. In this embodiment, the first region image has the signal values ​​of the original image only in the region where the blurred image is luminance saturated, and the signal values ​​in the other regions are 0.

[0033] Next, in step S105, the calculation unit 101c applies blur to the first region image and generates a saturation effect ground truth map. The applied blur is the same as the blur applied to the blurred image. This generates a saturation effect ground truth map, which is a map (a spatially arranged sequence of signals) that represents the magnitude and range of the signal values ​​that have spread due to blur (degradation) during imaging, from a subject in the brightness-saturated region of the blurred image. In this embodiment, the saturation effect ground truth map is clipped with the second signal value, similar to the blurred image, but clipping is not necessarily required.

[0034] Next, in step S106, the acquisition unit 101b acquires the ground truth model output. Since the task in this embodiment is blur sharpening, the ground truth model output is an image with less blur than the blurred image. In this embodiment, the ground truth model output is generated by clipping the original image with a second signal value. If the original image lacks high-frequency components, the ground truth model output may be an image obtained by reducing the original image. In this case, the same reduction is performed when generating the blurred image in step S102. Also, step S106 may be executed at any time, as long as it is after step S101 and before step S107.

[0035] Next, in step S107, the calculation unit 101c generates a saturation effect map and a model output based on the blurred image using a machine learning model. In this embodiment, the machine learning model shown in Figure 1 is used, but the invention is not limited to this. The blurred image 201 and the luminance saturation map 202 are input to the machine learning model. The luminance saturation map 202 is a map that shows the luminance-saturated region (where the signal value is greater than or equal to the second signal value) of the blurred image 201. For example, it can be generated by binarizing the blurred image 201 with the second signal value. However, the luminance saturation map 202 is not necessarily required. The blurred image 201 and the luminance saturation map 202 are concatenated in the channel direction and input to the machine learning model. However, the invention is not limited to this. For example, the blurred image 201 and the luminance saturation map 202 may each be converted into feature maps, and these feature maps may be concatenated in the channel direction. In addition, information other than the luminance saturation map 202 may be added to the input.

[0036] The machine learning model has multiple layers, and in each layer, a linear sum is taken of the layer's input and weights. The initial values ​​of the weights can be determined by random numbers or the like. In this embodiment, the machine learning model is a CNN that uses the convolution of the input and the filter (the values ​​of each element of the filter correspond to the weights, and may also include the sum with the bias) as the linear sum, but the invention is not limited to this. In addition, each layer may perform a nonlinear transformation using an activation function such as ReLU (Rectified Linear Unit) or a sigmoid function as needed. Furthermore, the machine learning model may have residual blocks or Skip Connections (also called Shortcut Connections) as needed. As a result of passing through multiple layers (16 convolutional layers in this embodiment), a saturation influence map 203 is generated.

[0037] In this embodiment, the saturation effect map 203 is obtained by taking the element-wise sum of the output of layer 211 and the luminance saturation map 202, but the configuration is not limited to this. The saturation effect map may be generated directly as the output of layer 211. Alternatively, the saturation effect map 203 may be obtained as the result of performing any processing on the output of layer 211. Next, the saturation effect map 203 and the blurred image 201 are concatenated in the channel direction and input to the subsequent layer, and the model output 204 is generated as a result of passing through multiple layers (16 convolutional layers in this embodiment). The model output 204 is also generated by taking the element-wise sum of the output of layer 212 and the blurred image 201, but the configuration is not limited to this. In this embodiment, each layer performs convolution with 64 types of 3x3 filters (however, layers 211 and 212 have the same number of filter types as the number of channels in the blurred image 201), but the configuration is not limited to this.

[0038] Next, in step S108, the update unit 101d updates the weights of the machine learning model based on the error function. In this embodiment, the error function is a weighted sum of the error between the saturation effect map 203 and the saturation effect ground truth map, and the error between the model output 204 and the ground truth model output. Mean Squared Error (MSE) is used to calculate the error. Both weights are set to 1. However, the error function and weights are not limited to these. Backpropagation or similar methods may be used to update the weights. The error may also be taken for the residual component. In the case of the residual component, the error of the difference component between the saturation effect map 203 and the luminance saturation map 202, and the difference component between the saturation effect ground truth map and the luminance saturation map 202 is used. Similarly, the error of the difference component between the model output 204 and the blurred image 201, and the difference component between the ground truth model output and the blurred image 201 are used.

[0039] Next, in step S109, the update unit 101d determines whether the training of the machine learning model is complete. Completion of training can be determined by whether the number of iterations of weight updates has reached a predetermined number, or whether the amount of change in weights during the update is less than a predetermined value. If it is determined in step S109 that training is not complete, the process returns to step S101, and the acquisition unit 101b acquires one or more new original images. On the other hand, if it is determined that training is complete, the update unit 101d terminates training and stores the configuration of the machine learning model and the weight information in the storage unit 101a.

[0040] Through the learning method described above, the machine learning model can estimate a saturation effect map that represents the magnitude and range of the signal values ​​that have spread due to blurring in the brightness-saturated region of the subject in the blurred image (or captured image during estimation). By explicitly estimating the saturation effect map, the machine learning model can perform blur sharpening on the appropriate regions for both saturated and unsaturated blurred images, thereby suppressing the occurrence of artifacts.

[0041] Next, referring to Figure 8, we will describe the blur sharpening of captured images using a trained machine learning model, which is performed in the image processing device 103. Figure 8 is a flowchart of the generation of the model output. The image processing device 103 has a storage unit 103a, an acquisition unit 103b, and a sharpening unit 103c, and each step in Figure 8 is mainly performed by the respective parts of the image processing device 103.

[0042] First, in step S201, the acquisition unit 103b acquires the captured image and the machine learning model. Information on the configuration and weights of the machine learning model is acquired from the storage unit 103a.

[0043] Next, in step S202, the sharpening unit (first generation means) 103c generates correction information using a machine learning model. In this embodiment, the correction information is a blur-sharpened image (model output) in which the blur of the captured image has been sharpened from the captured image. Note that it may be a correction component of blur sharpening instead of a blur-sharpened image (image corrected from the captured image). The machine learning model has the same configuration as during training, as shown in Figure 1. As during training, a luminance saturation map representing the luminance-saturated region of the captured image is generated and input, and a saturation effect map and model output are generated. As an example, Figures 10(A) and 11(A) show the blur-sharpened image (captured image), and Figures 10(B) and 11(B) show the saturation effect map. Figures 10(A) and (B) are scenes with a bright average luminance value of the captured image. Figures 11(A) and (B) are scenes with a dark average luminance value of the captured image. The method for calculating the average luminance value will be described later.

[0044] Next, with reference to Figure 9, the synthesis of the captured image and the model output, performed by the image processing device 103, will be described. Figure 9 is a flowchart of the sharpening intensity adjustment. Each step in Figure 9 is mainly performed by the various parts of the image processing device 103.

[0045] First, in step S211, the acquisition unit 103b acquires the imaging state from the captured image. The imaging state refers to, for example, the aperture value (F-number) of the optical system 102a and the pixel pitch of the image sensor 102b. The appearance of light rays and Airy disks in the captured image depends on the aperture value of the optical system 102a and the pixel pitch of the image sensor 102b. Specifically, the larger the aperture value of the optical system 102a and the smaller the pixel pitch, the more prominent the light rays and Airy disks become. Therefore, a weight map is generated according to the aperture value and pixel pitch acquired in step S211.

[0046] Next, in step S212, the acquisition unit 103b acquires information regarding the brightness values ​​of the captured image. Here, the information regarding the brightness values ​​of the captured image refers to statistics relating to the brightness values ​​of the captured image, which include at least one of the mean, median, variance, or histogram of the brightness values ​​of the captured image. Furthermore, the statistics relating to the brightness values ​​of the captured image, which are information regarding the brightness values ​​of the captured image, may be for the entire captured image, or statistics for each region into which the captured image has been divided may be used.

[0047] In this embodiment, the average brightness value of the captured image is obtained as information regarding the brightness value of the captured image. However, when obtaining the average brightness value of the captured image, if there are many saturated regions in the captured image, even a dark image such as a night scene may be judged as a bright image because a large average brightness value is obtained. Therefore, it is preferable to obtain the average brightness value of a second image obtained by removing the saturation effect map from the captured image. In this case, the information regarding the brightness value of the captured image is generated based on statistics regarding the brightness value of the second image obtained by removing the saturation effect map from the captured image. By obtaining the average brightness value of the non-saturated regions in the captured image in the second image, it is possible to appropriately determine whether the captured image is a bright scene or a dark scene. Note that the use of a saturation effect map is not mandatory; any information regarding saturated regions is sufficient. For example, the second image may be generated by removing the brightness saturation map from the captured image regarding the brightness saturation of the captured image.

[0048] Next, in step S213, the sharpening unit 103c generates a weight map based on information regarding the aperture value of the optical system 102a and information regarding the brightness value of the captured image. In this embodiment, if the aperture value of the optical system 102a is F22 (a predetermined aperture value) or higher, and the pixel pitch of the image sensor 102b is less than 6 μm (a predetermined pixel pitch), the sharpening unit 103c performs intensity adjustment. In other cases, the sharpening unit 103c generates a weight map in which all weights of the captured image are 0. Alternatively, the processing in steps S212 to S215 may be omitted without generating a weight map. The aperture value (a predetermined aperture value) and the pixel pitch threshold (a predetermined pixel pitch) for performing intensity adjustment can be determined to any number. For example, intensity adjustment may be performed when the aperture value of the optical system 102a is F16 or higher, and the pixel pitch of the image sensor 102b is less than 4 μm. In this embodiment, the weight map may be generated based on the aperture value without considering the pixel pitch.

[0049] Furthermore, in this embodiment, intensity adjustment is not limited to whether or not it is performed based on a predetermined aperture value (e.g., F22). For example, a weight map may be used in which the weight of the captured image changes continuously (stepwise) based on the aperture value. That is, if the aperture value can be set to a first aperture value or a second aperture value greater than the first aperture value, the weight map will be data where the weight of the captured image is greater for the second aperture value than for the first aperture value.

[0050] Next, we will explain the weight map based on brightness values. In this embodiment, the weight map is data in which the smaller the brightness value, the greater the weight of the captured image. For example, when the average brightness value of the captured image is used as information about the brightness value of the captured image, the smaller the average brightness value, the greater the weight of the captured image. Specifically, the relationship between the average brightness value and the weight of the weight map is stored as a linear function, and the weight corresponding to the average brightness value of the captured image is obtained to generate the weight map.

[0051] For example, the weights of the captured images corresponding to the average brightness value, obtained from the relational expression shown in Figure 12, are used. Figure 12 is an explanatory diagram of the weight map, where the horizontal axis represents the average brightness value and the vertical axis represents the weight of the captured image. However, the relationship between the average brightness value and the adjustment value of the weight map is not limited to this. Figure 12 shows the average brightness value in an undeveloped RAW image, but the average brightness value after development may also be used. When calculating the average brightness value in a RAW image, it is preferable to subtract the signal value in the optical black region of the image sensor 102b before calculating the average brightness value. This makes it possible to calculate the average brightness value that does not depend on the ISO sensitivity or the image sensor 102b. Also, if the average brightness value is obtained for each region into which the captured image is divided, the weights may be obtained from the average brightness value of each region to generate a weight map. Details of this will be described later in Example 2.

[0052] Furthermore, when obtaining statistical quantities related to the signal values ​​of the captured image, such as the mean, median, variance, or histogram of the brightness values ​​of the captured image, a weight map is generated such that the smaller the statistic, the greater the weight of the captured image. For example, when using a histogram of the brightness values ​​of the captured image, a weight map is generated such that the smaller the centroid or peak of the histogram, the greater the weight of the captured image.

[0053] Next, in step S214 of Figure 9, the sharpening unit 103c adjusts the weight map based on information regarding the saturation region of the captured image. In this embodiment, the information regarding the saturation region is a saturation effect map, and it is not necessary for all RGB components to be saturated. The saturation effect map is normalized from 0 to 1 using the set signal values, and this is used to adjust the weight map. Specifically, the normalized weight map is applied (multiplied) to the weight map generated in step S213. This applies the weights of the weight map generated in step S213 to the saturation region effect, making it possible to adjust the intensity only in the saturation effect region where light rays and Airy disks are more prominent. Note that applying the saturation effect map is not mandatory, and the entire captured image may be the target of intensity adjustment. Using the saturation effect map makes it possible to adjust the correction intensity up to the saturation effect region. Note that a luminance saturation map may be used instead of a saturation effect map, or a luminance saturation map blurred for each image height may be used.

[0054] As an example, Figures 13(A) and (B) show the final weight maps. In this embodiment, the weight map is such that the larger the pixel value, the greater the weight of the captured image, and the smaller the pixel value, the less the weight of the captured image. Figure 13(A) shows a bright scene with an average brightness value, and Figure 13(B) shows a dark scene with an average brightness value. In the saturation effect region of Figure 13(B), where the negative effects are more noticeable, the weight of the captured image is large.

[0055] Preferably, the weight map is generated based on information regarding the optical performance of the optical system used to capture the image. Specifically, the weight of the image is increased when the optical performance is low, and decreased when the optical performance is high. Information regarding optical performance can be calculated based on at least one of the zoom position, aperture diameter, or subject distance at the time of image capture, and the magnitude and range of the PSF signal value for each image height of the optical system. Note that using PSF is not mandatory; any information regarding optical performance is acceptable. For example, an optical transfer function may be used.

[0056] Next, in step S215 of Figure 9, the sharpening unit (second generation means) 103c uses the weight map generated in step S214 to perform a weighted average (combination) of the captured image and the blur-sharpened image to generate an intensity-adjusted image 205.

[0057] With the above configuration, this embodiment provides an image processing system that can maintain the blur correction effect while suppressing the adverse effects caused by the aperture value of the optical system and the brightness value of the captured image. [Examples]

[0058] Next, the image processing system in Embodiment 2 of the present invention will be described. In this embodiment, the average brightness value is obtained for each region into which the captured image is divided, and a weight corresponding to the average brightness value is obtained for each region to generate a weight map. Figure 14 is a block diagram of the image processing system 300 in this embodiment. Figure 15 is an external view of the image processing system 300. The image processing system 300 includes a learning device 301, an imaging device 302, and an image processing device 303. The learning device 301 and the image processing device 303, and the image processing device 303 and the imaging device 302 are connected by a wired or wireless network, respectively.

[0059] The imaging device 302 includes an optical system 321, an image sensor 322, a storage unit 323, a communication unit 324, and a display unit 325. The captured image is transmitted to the image processing device 303 via the communication unit 324. The image processing device 303 receives the captured image via the communication unit 332 and performs blur sharpening using the configuration and weight information of the machine learning model stored in the storage unit 331. The configuration and weight information of the machine learning model is learned by the learning device 301, is acquired in advance from the learning device 301, and is stored in the storage unit 331. Furthermore, the image processing device 303 has a function to adjust the intensity of blur sharpening. The blur-sharpened image (model output) with the blur of the captured image sharpened and the intensity-adjusted image with the intensity adjusted are transmitted to the imaging device 302, stored in the storage unit 323, and displayed in the display unit 325.

[0060] The generation of training data and training of weights performed by the learning device 301 (training phase) and the blur sharpening of captured images using the trained machine learning model performed by the image processing device 303 (estimation phase) are the same as in Example 1 and will therefore be omitted.

[0061] Next, with reference to Figure 16, the synthesis of the captured image and model output performed by the image processing device 303 will be described. Figure 16 is a flowchart of the sharpening intensity adjustment. Each step in Figure 16 is mainly performed by the various parts of the image processing device 303.

[0062] First, in step S311, the acquisition unit 333 acquires the imaging state from the captured image. The imaging state includes, but is not limited to, the aperture value of the optical system 321 and the pixel pitch state of the image sensor 322 in the imaging device 302. In this embodiment, intensity adjustment is performed when the aperture value of the optical system 321 is F16 or higher and the pixel pitch of the image sensor 322 is 4 μm or higher. In other cases, in the generation of the weight map described later, a weight map is generated in which all weights of the captured image are 0, and intensity adjustment is not performed. Alternatively, the processing in steps S312 to S314 may be omitted without generating a weight map.

[0063] Next, in step S312, the acquisition unit 333 acquires information regarding the brightness values ​​of the captured image. In this embodiment, the average brightness value is acquired for each region into which the captured image is divided, and a weight corresponding to the average brightness value is acquired for each region to generate a weight map. Figures 17(A) and (B) are explanatory diagrams of the weight map, with Figure 17(A) showing the captured image and Figure 17(B) showing the average brightness values ​​for each region. In Figure 17(B), 2001 is a region with a low average brightness value, 2002 is a region with a high average brightness value, and 2003 is a region with an intermediate average brightness value.

[0064] Next, in step S313 of Figure 16, the sharpening unit 334 generates a weight map based on the aperture value of the optical system 321 and information regarding the brightness value of the captured image. The method for generating the weight map is the same as in Example 1, so its explanation is omitted. Next, in step S314, the sharpening unit 334 uses the weight map generated in step S312 to perform a weighted average (combination) of the captured image and the blur-sharpened image (model output) to generate an intensity-adjusted image 205.

[0065] With the above configuration, this embodiment provides an image processing system that can maintain the blur correction effect while suppressing the adverse effects caused by the aperture value of the optical system and the brightness value of the captured image. [Examples]

[0066] Next, an image processing system in Embodiment 3 of the present invention will be described. Figure 18 is a block diagram of the image processing system 400 in this embodiment. Figure 19 is an external view of the image processing system 400. The image processing system 400 includes a learning device 401, a lens device 402, an imaging device 403, a control device (first device) 404, an image estimation device (second device) 405, and networks 406 and 407.

[0067] The learning device 401 and the image estimation device 405 are, for example, servers. The control device 404 is a user-operated device such as a personal computer or mobile terminal. The learning device 401 has a storage unit 401a, an acquisition unit 401b, a calculation unit 401c, and an update unit 401d, and learns the weights of a machine learning model that sharpens blur from captured images captured using the lens device 402 and the imaging device 403. The learning method, i.e., the generation of learning data and learning of weights (learning phase) performed by the learning device 401, is the same as in Example 1 and will therefore be omitted.

[0068] The imaging device 403 has an image sensor 403a, which converts the optical image formed by the lens device 402 into an optical image via photoelectric conversion to acquire an image. The lens device 402 and the imaging device 403 are detachable and can be combined with multiple types of lenses. The control device 404 has a communication unit 404a, a display unit 404b, a storage unit 404c, and an acquisition unit 404d, and controls the processing to be performed on the image acquired from the imaging device 403, which is connected by wire or wireless, according to the user's operation. Alternatively, the image captured by the imaging device 403 may be stored in the storage unit 404c in advance, and the image may be read out.

[0069] The image estimation device 405 includes a communication unit 405a, an acquisition unit 405b, a storage unit 405c, and a sharpening unit 405d, and is configured to communicate with the control device 404. The image estimation device 405 performs a blur sharpening process on the captured image in response to a request from the control device 404, which is connected via the network 406. The image estimation device 405 acquires learned weight information from the learning device 401, which is connected via the network 406, either at the time of blur sharpening estimation or in advance, and uses it to estimate the blur sharpening of the captured image. The estimated image after blur sharpening estimation is transmitted back to the control device 404 after the sharpening intensity has been adjusted, stored in the storage unit 404c, and displayed on the display unit 404b.

[0070] Next, with reference to Figure 20, we will describe the blur sharpening of the captured image performed by the control device 404 and the image estimation device 405. Figure 20 is a flowchart of the model output and sharpening intensity adjustment. Each step in Figure 20 is mainly performed by the respective parts of the control device 404 or the image estimation device 405.

[0071] First, in step S401, the acquisition unit 404d of the control device 404 acquires the captured image and the sharpening intensity specified by the user. Next, in step S402, the communication unit (transmission means) 404a transmits the captured image and a request to the image estimation device 405 for the execution of the blur sharpening estimation process.

[0072] Next, in step S403, the communication unit (receiving means) 405a of the image estimation device 405 receives and acquires the captured image and processing request transmitted from the control device 404. Next, in step S404, the acquisition unit 405b acquires the learned weight information corresponding to the captured image from the storage unit 405c. The weight information is read in advance from the storage unit 401a and stored in the storage unit 405c. Next, in step S405, the sharpening unit 405d uses a machine learning model to generate a blur-sharpened image (model output) from the captured image, in which the blur of the captured image has been sharpened. The machine learning model has the same configuration as during training, as shown in Figure 1. As during training, a luminance saturation map representing the luminance-saturated region of the captured image is generated and input, and a saturation effect map and model output are generated.

[0073] Next, in step S406, the sharpening unit 405d generates a weight map. The method for generating the weight map is the same as in Embodiment 1. The default weight map is adjusted according to the sharpening intensity specified by the user. Note that a pre-adjusted weight map may be kept within a range where the intensity can be adjusted. Next, in step S407, the sharpening unit 405d synthesizes the captured image and the blur-sharpened image (model output) based on the weight map. Next, in step S408, the communication unit 405a transmits the synthesized image to the control device 404. Next, in step S409, the communication unit 404a of the control device 404 acquires the estimated image transmitted from the image estimation device 405.

[0074] With the above configuration, this embodiment provides an image processing system that can maintain the blur correction effect while suppressing the adverse effects caused by the aperture value of the optical system and the brightness value of the captured image. (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. Furthermore, the image processing device only needs to be a device having the image processing function of the present invention, and can be implemented in the form of an imaging device or a PC.

[0075] According to each embodiment, it is possible to provide an image processing method, an image processing device, an image processing program, and a storage medium that can maintain the blur correction effect while suppressing the adverse effects caused by the aperture value of the optical system and the brightness value of the captured image.

[0076] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence. [Explanation of symbols]

[0077] 103 Image Processing Device 103c Sharpened section (first generating means, second generating means)

Claims

1. A step of generating information regarding correction of a captured image based on the captured image obtained by imaging using an optical system; generating a weight map based on information about an aperture value of the optical system and information about a luminance value of the captured image; An image processing method comprising the step of generating a first image based on the captured image, information regarding the correction, and the weight map.

2. 2. The image processing method according to claim 1, wherein the weight map is generated based on information regarding a saturated region in the captured image.

3. 3. The image processing method according to claim 2, wherein the information regarding the saturated region is information indicating a region in which a subject in the saturated region of the captured image has expanded due to blurring occurring during imaging, and a signal value in the region.

4. The image processing method described in Claim 3, characterized in that information regarding the luminance values ​​of the captured image is generated based on statistics regarding the luminance values ​​of a second image based on the difference between the luminance values ​​of the captured image and the luminance values ​​in the region.

5. The image processing method of claim 4, wherein the statistics include at least one of a mean, a median, a variance, or a histogram.

6. The image processing method according to claim 2 , wherein the information regarding the saturated region is generated by a machine learning model.

7. An image processing method described in any one of claims 1 to 6, characterized in that in the step of generating the weight map, the weight map is generated based on information regarding the optical performance of the optical system.

8. 8. The image processing method according to claim 7, wherein the information regarding the optical performance is calculated based on at least one of a zoom position, an aperture value, and a subject distance at the time of capturing the image, and a point spread function for each image height of the optical system.

9. The image processing method according to any one of claims 1 to 8, characterized in that the information regarding the correction is a corrected image obtained by performing the correction on the captured image, or a correction component which is a difference between the corrected image and the captured image.

10. 9. The image processing method according to claim 1, wherein the correction includes at least one of increasing the resolution of the captured image and converting a shape of defocus blur of the captured image.

11. 11. The image processing method according to claim 1, wherein the weighting map weights the captured image more heavily as the luminance value decreases.

12. The statistic regarding the luminance values ​​of the second image is an average value of the luminance values ​​in the second image, 6. The image processing method according to claim 5, wherein the weighting map weights the captured image more heavily as the average value decreases.

13. The aperture value can be set to a first aperture value or a second aperture value greater than the first aperture value; 13. The image processing method according to claim 1, wherein a weight in the weight map for the second aperture value is greater than a weight in the weight map for the first aperture value.

14. A first generating means for generating information regarding correction of an image based on an image obtained by imaging using an optical system; a second generating means for generating a weight map based on information regarding an aperture value of the optical system and information regarding a luminance value of the captured image; and a third generating means for generating a first image based on the captured image, the information related to the correction, and the weighting map.

15. An image processing system comprising the image processing device according to claim 14 and a control device capable of communicating with the image processing device, the control device has a transmission means for transmitting a request for causing the image processing device to execute processing on the captured image, The image processing system is characterized in that the image processing device has a receiving means for receiving the request, and executes processing on the captured image in response to the request.

16. A program causing a computer to execute the image processing method according to any one of claims 1 to 13.

Images