Image processing method, image processing device, image processing system, and program

Abstract

To provide an image processing method capable of maintaining a blur correction effect while suppressing adverse effects caused by an aperture value of an optical system and a luminance value of a captured image.SOLUTION: The image processing method includes the steps of: generating information regarding the correction of a captured image based on the captured image; and generating an intensity-adjusted image based on the captured image, the information regarding correction, and a weight map, wherein the weight map is generated based on information regarding an aperture value of an optical system used to capture the captured image and information regarding a luminance value of the captured image.SELECTED DRAWING: Figure 16

Description

[Technical Field]

[0001] The present invention relates to an image processing method, an image processing device, an image processing system, and a program. [Background technology]

[0002] In image recognition or regression tasks, methods using machine learning models can achieve higher accuracy than theory-based methods that use assumptions and approximations. Theory-based methods suffer from reduced accuracy due to factors ignored by assumptions and approximations. However, methods using machine learning models train the machine learning model using training data that includes these factors, enabling estimations that are consistent with the training data without assumptions or approximations, thereby improving the accuracy of the task.

[0003] Patent Document 1 discloses a method for sharpening blur in a captured image using a convolutional neural network (CNN), which is one of machine learning models, and also discloses a method for adjusting the strength of sharpening by weighting the captured image and an estimated image (blur-sharpened image) based on the brightness saturation region. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2020-166628 Summary of the Invention [Problem to be solved by the invention]

[0005] The method disclosed in Patent Document 1 does not address the problems caused by the diffraction of light. When capturing an image of a highly luminous subject with a large aperture value (F-number) of an optical system, airy disks and light beams occur due to the diffraction of light. When correcting airy disks and light beams using a machine learning model, problems such as unnatural emphasis may occur due to insufficient accuracy of the training image or insufficient parameters of the machine learning model. Furthermore, the degree to which this problem is noticeable varies depending on the peripheral luminance value of the correction target in the captured image. For example, unnatural emphasis is noticeable in dark images such as night scenes, but is not noticeable in bright images captured outdoors during the day.

[0006] Furthermore, because the occurrence of airy disks and streaks of light depends on the aperture value of the optical system, the correction effect of captured images captured at aperture values ​​where no adverse effects occur is also reduced. Furthermore, the effects are averaged according to a predetermined weight without taking into account the differing conspicuousness of adverse effects depending on the brightness value of the captured image. In other words, while adverse effects are reduced around saturated areas in dark images, the correction effect around saturated areas is reduced more than necessary in bright images.

[0007] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing method that can maintain the effect of correcting blur while suppressing adverse effects caused by the aperture value of the optical system and the luminance value of the captured image. [Means for solving the problem]

[0008] According to one aspect of the present invention, an image processing method ,light Based on the images obtained by imaging using the , the computer generating information regarding correction of the captured image; and , the computer generating a weight map; and calculating a weight map based on the captured image, the information about the correction, and the weight map. , the computer and generating a first image.

[0009] Other objects and features of the present invention are illustrated in the following examples. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide an image processing method that can maintain the effect of correcting blur while suppressing adverse effects caused by the aperture value of the optical system and the luminance value of the captured image. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 2 is an explanatory diagram of a machine learning model in the first embodiment. [Figure 2] 1 is a block diagram of an image processing system according to a first embodiment. [Figure 3] 1 is an external view of an image processing system according to a first embodiment. [Figure 4] FIG. 10 is an explanatory diagram of adverse effects caused by sharpening in Examples 1 to 3. [Figure 5] FIG. 1 is an explanatory diagram of an airy disk in Examples 1 to 3. [Figure 6] FIG. 1 is an explanatory diagram of a beam of light in Examples 1 to 3. [Figure 7] 1 is a flowchart of learning of a machine learning model in Examples 1 to 3. [Figure 8] 10 is a flowchart of generating a model output in the first and second embodiments. [Figure 9] 10 is a flowchart of adjusting the sharpening strength in the first embodiment. [Figure 10] 4A and 4B are explanatory diagrams of a captured image and a saturation influence map in the first embodiment. [Figure 11] 4A and 4B are explanatory diagrams of a captured image and a saturation influence map in the first embodiment. [Figure 12] FIG. 3 is an explanatory diagram of a weight map in the first embodiment. [Figure 13] FIG. 3 is an explanatory diagram of a weight map in the first embodiment. [Figure 14] FIG. 10 is a block diagram of an image processing system according to a second embodiment. [Figure 15] FIG. 10 is an external view of an image processing system according to a second embodiment. [Figure 16]10 is a flowchart of adjusting the sharpening strength in the second embodiment. [Figure 17] FIG. 10 is an explanatory diagram of a weight map in the second embodiment. [Figure 18] FIG. 10 is a block diagram of an image processing system according to a third embodiment. [Figure 19] FIG. 11 is an external view of an image processing system according to a third embodiment. [Figure 20] 13 is a flowchart of model output and sharpening strength adjustment in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and redundant explanations will be omitted.

[0013] Before describing the specific embodiments, the gist of the present invention will be described. The present invention generates an estimated image from a captured image captured using an optical system by sharpening the blur caused by the optical system using a machine learning model. Then, a weight map is generated based on information about the aperture value of the optical system used to capture the captured image and information about the luminance value of the captured image, and a weighted average is calculated between the captured image and the estimated image. The weight map is used to determine the proportion of each image when calculating the weighted average of the captured image and the estimated image, and has continuous signal values. For example, if the value of the weight map determines the proportion of the captured image, a value of 0.5 results in an intensity-adjusted image obtained by weighting the proportion of the captured image and the blur-sharpened image at 50%. Furthermore, if the value of the weight map is 1, the intensity-adjusted image is the captured image.

[0014] Blur caused by the optical system includes blur due to aberration, diffraction, defocus, the effect of an optical low-pass filter, and degradation of the pixel aperture of the image sensor. Machine learning models include, for example, neural networks, genetic programming, and Bayesian networks. Neural networks include convolutional neural networks (CNNs), generative adversarial networks (GANs), and recurrent neural networks (RNNs).

[0015] Blur sharpening refers to the process of restoring frequency components of a subject that have been reduced or lost due to blur. When sharpening blur, if the aperture value of the optical system is increased to capture a high-brightness subject, airy disks and light beams occur due to the diffraction of light. When airy disks and light beams are corrected using a machine learning model, problems such as unnatural emphasis can occur due to insufficient precision in the training images or insufficient parameters in the machine learning model. Details on airy disks and light beams, insufficient precision in the training images, and insufficient parameters in the machine learning model will be discussed later.

[0016] Furthermore, the degree to which this adverse effect is noticeable varies depending on the brightness of the image. For example, unnatural emphasis is noticeable in dark images such as night scenes, but is not noticeable in bright images captured outdoors during the day. Therefore, the present invention generates a weight map based on information about the aperture value of the optical system used to capture the captured image and information about the luminance value of the captured image, and performs a weighted average of the captured image and the estimated image. This makes it possible to maintain the blur correction effect while suppressing adverse effects caused by the aperture value of the optical system and the luminance value of the captured image. Hereinafter, the stage of learning the weights of the machine learning model will be referred to as the learning phase, and the stage of sharpening the blur using the machine learning model using the learned weights will be referred to as the estimation phase. [Example]

[0017] First, an image processing system according to a first embodiment of the present invention will be described. In this embodiment, the task performed by the machine learning model is to sharpen blur in a captured image that includes brightness saturation (to increase the resolution of the captured image). The blur to be sharpened is the blur caused by aberrations, diffraction, and an optical low-pass filter that occur in the optical system. However, the effects of the present invention can be similarly obtained when sharpening blur caused by pixel aperture, defocus, or shaking. This embodiment can also be applied to tasks other than blur sharpening to obtain similar effects. Specifically, this includes tasks such as upsampling to increase the number of pixels in a captured image and converting (shape conversion) defocus blur in a captured image. Conversion of defocus blur includes, for example, converting double-line blur to Gaussian blur or spherical blur. Double-line blur has a point spread function (PSF) with separate peaks. This causes a subject that is actually a single line to appear doubly blurred when defocused. Sphere blur has a PSF with flat intensity. Gaussian blur has a PSF with a Gaussian distribution. Other defocus blurs that can be transformed include, for example, defocus blur caused by vignetting and ring-shaped defocus blur caused by pupil obstruction by a catadioptric lens or the like.

[0018] FIG. 2 is a block diagram of an image processing system 100 in this embodiment. FIG. 3 is an external view of the image processing system 100. The image processing system 100 has a learning device 101 and an image processing device 103, which are connected via a wired or wireless network. An imaging device 102, a display device 104, a recording medium 105, and an output device 106 are connected to the image processing device 103 via wired or wireless connections. A captured image of a subject space captured using the imaging device 102 is input to the image processing device 103. Blurring occurs in the captured image due to aberration and diffraction caused by the optical system 102a in the imaging device 102 and the optical low-pass filter of the image sensor 102b, and information about the subject is attenuated.

[0019] The image processing device 103 performs blur sharpening on the captured image using a machine learning model, and generates a saturation influence map and a blur-sharpened image (model output). Details of the saturation influence map will be described later. The machine learning model is learned by the learning device 101, and the image processing device 103 acquires information about the machine learning model from the learning device 101 in advance and stores it in the storage unit 103a. The image processing device 103 also has a function of adjusting the intensity of blur sharpening by performing a weighted addition of the captured image and the blur-sharpened image. Details of the learning and estimation of the machine learning model and the adjustment of the intensity of blur sharpening will be described later. A user can adjust the intensity of blur sharpening while checking the image displayed on the display device 104. The blur-sharpened image after the intensity adjustment is stored in the storage unit 103a or the recording medium 105 and, as necessary, output to an output device 106 such as a printer. The captured image may be grayscale or have multiple color components. Furthermore, the captured image may be an undeveloped RAW image or a developed image.

[0020] Next, with reference to Figures 4(A) to 4(C) and Figure 5, we will explain the adverse effects of optical diffraction that occur when blur sharpening is performed using a machine learning model. Figures 4(A) and 4(B) are cross-sectional views of the PSF (vertical axis represents signal value, horizontal axis represents spatial coordinate) when the aperture value of the optical system is increased, but the scales of the vertical axes are different. Figure 5 is a plan view of the PSF. The optical diffraction phenomenon produces a diffraction pattern with a bright central area surrounded by dark concentric rings. This is called an Airy disk. Although this diffraction pattern has a weak signal value, it can be visually confirmed in the captured image when a point light source or strong light is captured. In Figure 4(C), the dashed-dotted line 141 represents the ideal signal value obtained by sharpening the PSF and reducing the diffraction pattern, and the dotted line 142 represents the signal value obtained when the diffraction pattern is enhanced by sharpening. The dashed-dotted line 141 is preferable for blur sharpening. However, due to insufficient precision of the training images or insufficient parameters of the machine learning model, the diffraction pattern may be emphasized, as shown by the dotted line 142. This is because the edges of the diffraction pattern are mistakenly recognized as the subject and sharpened. Insufficient precision of the training images and insufficient parameters of the machine learning model will be described later.

[0021] Next, with reference to FIG. 6, we will explain light beams caused by the diffraction phenomenon of light. Light beams refer to elongated, thin streaks of light that appear when capturing an image of a point light source or intense light. FIG. 6 shows an example of light beams. The number of streaks depends on the number of aperture blades 143 in the optical system 102a. When the number of aperture blades 143 is odd, twice as many streaks appear. On the other hand, when the number of aperture blades 143 is even, the number of streaks appears. Since the number of aperture blades 143 shown in FIG. 6 is five, ten light streaks 144 appear. The sharpness of the light beams depends on the aperture value of the optical system. As the aperture value increases, the sharpness increases. Furthermore, even with the same aperture value, the appearance of the light beams varies depending on the type of optical system 102a. As with the Airy disk, unnatural emphasis of the light beams can occur due to insufficient precision in the training images or insufficient parameters in the machine learning model.

[0022] Next, we will explain the lack of accuracy of the training images in detail. When using a machine learning model to sharpen blur, the accuracy of the sharpening depends on the accuracy of the training images. In other words, training images that accurately reproduce the blur of the optical system 102a, which is the target of sharpening, are required. However, if we attempt to comprehensively learn the blur at the zoom positions, aperture values, and subject distances that the optical system 102a can assume during image capture, the number of training images becomes enormous. As a result, problems such as a decrease in the accuracy of individual blur sharpening and non-convergence of learning arise. Therefore, it is necessary to discretely learn the blur at the zoom positions, aperture values, and subject distances that the optical system 102a can assume and have the machine learning model predict the intermediate region. However, in this case, sharpening the blur in the intermediate region not included in the training images may result in the edges of the diffraction pattern being mistakenly recognized as the subject and sharpened.

[0023] Next, we will explain parameter deficiency in machine learning models in detail. Machine learning models have multiple layers, and each layer takes a linear sum of the layer's input and weights. When a CNN is used as the machine learning model, which uses the convolution of the input and filter as a linear sum (the value of each filter element corresponds to the weight, and may also include a sum with a bias), the number of filter layers corresponds to the number of parameters. In other words, parameter deficiency refers to an insufficient number of filter layers. Because there is a trade-off between the number of filter layers, training time, and processing speed, parameter deficiency can occur. Using a machine learning model trained using a method that uses captured images and corresponding brightness saturation maps as input data for the machine learning model, and a method that generates a saturation influence map, it is possible to reduce the adverse effects, but it is difficult to completely eliminate them. We will explain in detail the method using brightness saturation maps and the method generating a saturation influence map.

[0024] Next, we will explain the brightness saturation map. A brightness saturation map is a map that represents brightness saturated areas in a captured image. In areas where brightness saturation occurs (brightness saturated areas), information about the structure of the subject space is lost, and false edges may appear at the boundaries between areas, making it impossible to extract correct feature quantities for the subject. Therefore, by inputting a brightness saturation map, the neural network can identify problematic areas such as those mentioned above, thereby preventing a decrease in estimation accuracy.

[0025] Next, the saturation influence map will be described. Even when a brightness saturation map is used, the machine learning model may not make an accurate judgment. For example, if a region of interest is near a brightness saturated region, the machine learning model can determine that the region of interest is affected by brightness saturation because there is a brightness saturated region near the region of interest. However, if the region of interest is located far from the brightness saturated region, it is not easy to determine whether the region is affected by brightness saturation, and ambiguity increases. As a result, the machine learning model may make an incorrect judgment at a position far from the brightness saturated region. For this reason, when the task is blur sharpening, a sharpening process corresponding to a saturated blur image is performed on a non-saturated blur image. In this case, artifacts occur in the image with the sharpened blur, reducing the accuracy of the task. Therefore, it is preferable to use a machine learning model to generate a saturation influence map from a captured image in which blur occurs.

[0026] A saturation influence map is a map (a spatially arranged signal sequence) that represents the magnitude and range of signal values ​​that have spread due to blurring of subjects in brightness-saturated areas of a captured image. By having a machine learning model generate a saturation influence map, the machine learning model can accurately estimate the presence and magnitude of the influence of brightness saturation in the captured image. By generating a saturation influence map, the machine learning model can execute the processing that should be executed on areas affected by brightness saturation and the processing that should be executed on other areas, respectively, in the appropriate areas. Therefore, by having a machine learning model generate a saturation influence map, the accuracy of the task is improved compared to when the generation of a saturation influence map is not involved (i.e., only recognition labels and blur-sharpened images are generated directly from the captured image).

[0027] While the above two methods are effective, it is difficult to completely eliminate the adverse effects. Therefore, in this embodiment, a weight map is generated based on information about the aperture value of the optical system used to capture the captured image and information about the luminance value of the captured image, and a weighted average is calculated between the captured image and the estimated image. This makes it possible to maintain the blur correction effect while suppressing the adverse effects caused by the aperture value of the optical system and the luminance value of the captured image.

[0028] Next, learning of a machine learning model executed by the learning device 101 will be described with reference to Fig. 7. Fig. 7 is a flowchart of learning of a machine learning model. The learning device 101 has a storage unit 101a, an acquisition unit 101b, a calculation unit 101c, and an update unit 101d, and each step in Fig. 7 is mainly executed by each unit of the learning device 101.

[0029] First, in step S101, the acquisition unit 101b acquires one or more original images from the storage unit 101a. The original image is an image having a signal value higher than a second signal value. The second signal value is a signal value corresponding to the saturation brightness value of the captured image. However, since the signal value may be normalized when input to the machine learning model, the second signal value does not necessarily have to match the saturation brightness value of the captured image. Since the machine learning model is trained based on the original image, it is desirable that the original image be an image having various frequency components (edges of different directions and intensities, gradations, flat areas, etc.). The original image may be a real-life image or CG (Computer Graphics).

[0030] Next, in step S102, the calculation unit 101c applies blur to the original image to generate a blurred image. The blurred image is an image input to the machine learning model during training and corresponds to the captured image during estimation. The blur to be applied is the blur to be sharpened. In this embodiment, blur generated by the aberration and diffraction of the optical system 102a and the optical low-pass filter of the image sensor 102b is applied. The shape of the blur caused by the aberration and diffraction of the optical system 102a varies depending on the image plane coordinates (image height and azimuth). It also varies depending on the magnification, aperture, and focus state of the optical system 102a. To simultaneously train a machine learning model that sharpens all of these blurs, it is recommended to generate multiple blurred images using multiple blurs generated by the optical system 102a. Furthermore, in the blurred image, signal values ​​exceeding a second signal value are clipped. This is done to reproduce brightness saturation that occurs during the capture process of the captured image. If necessary, noise generated by the image sensor 102b may be applied to the blurred image.

[0031] Next, in step S103, the calculation unit 101c sets a first region based on an image based on the original image and a signal value threshold. In this embodiment, a blurred image is used as the image based on the original image, but the original image itself may also be used. The first region is set by comparing the signal value of the blurred image with the signal value threshold. More specifically, the first region is defined as a region where the signal value of the blurred image is equal to or greater than the signal value threshold. In this embodiment, the signal value threshold is the second signal value. Therefore, the first region represents a brightness saturated region of the blurred image (saturated region). However, the signal value threshold and the second signal value do not have to match. The signal value threshold may be set to a value slightly smaller than the second signal value (for example, 0.9 times).

[0032] Next, in step S104, the calculation unit 101c generates a first region image having the signal value of the original image in the first region. The first region image has a signal value different from that of the original image in regions other than the first region. More preferably, the first region image has a first signal value in regions other than the first region. In this embodiment, the first signal value is 0, but the invention is not limited to this. In this embodiment, the first region image has the signal value of the original image only in regions where the blurred image is saturated in brightness, and the signal value in other regions is 0.

[0033] Next, in step S105, the calculation unit 101c blurs the first region image to generate a saturation influence correct map. The blur applied is the same as the blur applied to the blurred image. As a result, a saturation influence correct map is generated, which is a map (a spatially arranged signal sequence) that represents the magnitude and range of signal values ​​that have expanded due to blur (deterioration) during imaging from the subject in the brightness-saturated region of the blurred image. In this embodiment, as with the blurred image, the saturation influence correct map is clipped with the second signal value, but clipping is not necessarily required.

[0034] Next, in step S106, the acquisition unit 101b acquires the correct model output. Since the task in this embodiment is blur sharpening, the correct model output is an image with less blur than the blurred image. In this embodiment, the correct model output is generated by clipping the original image with the second signal value. If the original image lacks high-frequency components, an image obtained by reducing the original image may be used as the correct model output. In this case, reduction is also performed similarly when generating the blurred image in step S102. Furthermore, step S106 may be executed any time after step S101 and before step S107.

[0035] Next, in step S107, the calculation unit 101c uses a machine learning model to generate a saturation influence map and a model output based on the blurred image. In this embodiment, the machine learning model shown in FIG. 1 is used, but the invention is not limited to this. A blurred image 201 and a brightness saturation map 202 are input to the machine learning model. The brightness saturation map 202 is a map that indicates brightness saturated areas of the blurred image 201 (where the signal value is equal to or greater than a second signal value). For example, the brightness saturation map 202 can be generated by binarizing the blurred image 201 using the second signal value. However, the brightness saturation map 202 is not necessarily required. The blurred image 201 and the brightness saturation map 202 are linked in the channel direction and input to the machine learning model. However, the invention is not limited to this. For example, the blurred image 201 and the brightness saturation map 202 may be converted into feature maps, and these feature maps may be linked in the channel direction. Furthermore, information other than the brightness saturation map 202 may be added to the input.

[0036] The machine learning model has multiple layers, and in each layer, a linear sum of the layer's input and weights is calculated. The initial values ​​of the weights may be determined using random numbers or the like. In this embodiment, the machine learning model is a CNN that uses the convolution of the input and filter as the linear sum (the value of each element of the filter corresponds to the weight, and may also include a sum with a bias), but the invention is not limited to this. In addition, in each layer, nonlinear transformation is performed using an activation function such as a ReLU (Rectified Linear Unit) or a sigmoid function as necessary. Furthermore, the machine learning model may have a residual block or a skip connection (also called a shortcut connection) as necessary. As a result of passing through multiple layers (16 convolutional layers in this embodiment), a saturation influence map 203 is generated.

[0037] In this embodiment, saturation influence map 203 is generated by taking the element-by-element sum of the output of layer 211 and intensity saturation map 202, but the configuration is not limited to this. The saturation influence map may be generated directly as the output of layer 211. Alternatively, the saturation influence map 203 may be generated by performing any processing on the output of layer 211. Next, saturation influence map 203 and blurred image 201 are concatenated in the channel direction and input to a subsequent layer, and model output 204 is generated after passing through multiple layers (16 convolutional layers in this embodiment). Model output 204 is also generated by taking the element-by-element sum of the output of layer 212 and blurred image 201, but the configuration is not limited to this. Note that in this embodiment, convolution is performed with 64 types of 3×3 filters in each layer (however, in layers 211 and 212, the number of filter types is the same as the number of channels of blurred image 201), but the configuration is not limited to this.

[0038] Next, in step S108, the update unit 101d updates the weights of the machine learning model based on the error function. In this embodiment, the error function is a weighted sum of the error between the saturation influence map 203 and the saturation influence correct map and the error between the model output 204 and the correct model output. MSE (Mean Squared Error) is used to calculate the error. Both weights are set to 1. However, the error function and weights are not limited to this. Backpropagation or the like may be used to update the weights. The error may also be calculated for the residual component. For the residual component, the error between the difference component between the saturation influence map 203 and the brightness saturation map 202 and the difference component between the saturation influence correct map and the brightness saturation map 202 is used. Similarly, the error between the difference component between the model output 204 and the blurred image 201 and the difference component between the correct model output and the blurred image 201 is used.

[0039] Next, in step S109, the update unit 101d determines whether learning of the machine learning model is complete. Completion of learning can be determined by, for example, whether the number of iterations of weight update has reached a predetermined number, or whether the amount of change in weight during update is smaller than a predetermined value. If it is determined in step S109 that learning is not complete, the process returns to step S101, and the acquisition unit 101b acquires one or more new original images. On the other hand, if it is determined that learning is complete, the update unit 101d ends learning and stores the configuration of the machine learning model and information about the weights in the storage unit 101a.

[0040] Using the above learning method, the machine learning model can estimate a saturation influence map that represents the magnitude and range of signal values ​​that have spread due to blurring of subjects in brightness-saturated regions of a blurred image (a captured image at the time of estimation).By explicitly estimating the saturation influence map, the machine learning model can sharpen the blur for both saturated and non-saturated blurred images in the appropriate regions, thereby suppressing the occurrence of artifacts.

[0041] Next, blur sharpening of a captured image using a trained machine learning model executed by the image processing device 103 will be described with reference to Fig. 8. Fig. 8 is a flowchart of generating a model output. The image processing device 103 has a storage unit 103a, an acquisition unit 103b, and a sharpening unit 103c, and each step in Fig. 8 is mainly executed by each unit of the image processing device 103.

[0042] First, in step S201, the acquisition unit 103b acquires a captured image and a machine learning model. Information about the configuration and weights of the machine learning model is acquired from the storage unit 103a.

[0043] Next, in step S202, the sharpening unit (first generating means) 103c generates information related to the correction using a machine learning model. In this embodiment, the information related to the correction is a blur-sharpened image (model output) obtained by sharpening the blur of the captured image from the captured image. Note that instead of the blur-sharpened image (image obtained by correcting the captured image), a blur-sharpening correction component may be used. The machine learning model has the configuration shown in FIG. 1, as in the learning process. As in the learning process, a brightness saturation map representing brightness-saturated regions of the captured image is generated and input, and a saturation influence map and model output are generated. As examples, FIGS. 10(A) and 11(A) show a blur-sharpened image (captured image), and FIGS. 10(B) and 11(B) show saturation influence maps. FIGS. 10(A) and 10(B) show a scene in which the average brightness value of the captured image is bright. FIGS. 11(A) and 11(B) show a scene in which the average brightness value of the captured image is dark. Note that a method for calculating the average brightness value will be described later.

[0044] Next, the synthesis of a captured image and a model output, which is executed by the image processing device 103, will be described with reference to Fig. 9. Fig. 9 is a flowchart of adjusting the sharpening strength. Each step in Fig. 9 is mainly executed by each unit of the image processing device 103.

[0045] First, in step S211, the acquisition unit 103b acquires the imaging state from the captured image. The imaging state is, for example, the aperture value (F-number) of the optical system 102a and the pixel pitch of the image sensor 102b. The appearance of the rays of light and the airy disk in the captured image depends on the aperture value of the optical system 102a and the pixel pitch of the image sensor 102b. Specifically, the larger the aperture value of the optical system 102a and the smaller the pixel pitch, the more noticeable the rays of light and the airy disk become. Therefore, a weight map is generated according to the aperture value and pixel pitch acquired in step S211.

[0046] Next, in step S212, the acquiring unit 103b acquires information about the luminance values ​​of the captured image. Here, the information about the luminance values ​​of the captured image is a statistical quantity about the luminance values ​​of the captured image, and is at least one of the average, median, variance, and histogram of the luminance values ​​of the captured image. Furthermore, the statistical quantity about the luminance values ​​of the captured image, which is information about the luminance values ​​of the captured image, may be information about the luminance values ​​of the captured image as a whole, or may be a statistical quantity for each region obtained by dividing the captured image.

[0047] In this embodiment, the average luminance value of the captured image is acquired as information about the luminance value of the captured image. However, when acquiring the average luminance value of the captured image, if the captured image contains many saturated areas, a large average luminance value may be acquired even for a dark image such as a night scene, resulting in the image being determined to be bright. Therefore, it is preferable to acquire the average luminance value of a second image obtained by excluding the saturation influence map from the captured image. In this case, information about the luminance value of the captured image is generated based on statistics regarding the luminance values ​​of the second image obtained by excluding the saturation influence map from the captured image. By acquiring the average luminance value of non-saturated areas in the second image, it is possible to appropriately determine whether the captured image represents a bright scene or a dark scene. Note that the use of the saturation influence map is not essential; information about saturated areas will suffice. For example, the second image may be generated by excluding the luminance saturation map from the captured image regarding the luminance saturation of the captured image.

[0048] Next, in step S213, the sharpening unit 103c generates a weight map based on information related to the aperture value of the optical system 102a and information related to the luminance value of the captured image. In this embodiment, if the aperture value of the optical system 102a is equal to or greater than F22 (predetermined aperture value) and the pixel pitch of the image sensor 102b is less than 6 μm (predetermined pixel pitch), the sharpening unit 103c performs intensity adjustment. In other cases, the sharpening unit 103c generates a weight map in which all weights of the captured image are 0. Alternatively, the processes of steps S212 to S215 may be omitted without generating a weight map. The aperture value (predetermined aperture value) and pixel pitch threshold (predetermined pixel pitch) for performing intensity adjustment may be set to any numerical values. For example, intensity adjustment may be performed if the aperture value of the optical system 102a is equal to or greater than F16 and the pixel pitch of the image sensor 102b is less than 4 μm. In this embodiment, the weight map may be generated based on the aperture value without considering the pixel pitch.

[0049] In this embodiment, the intensity adjustment is not limited to whether it is performed based on a predetermined aperture value (e.g., F22). For example, a weight map in which the weight of the captured image changes continuously (in stages) based on the aperture value may be used. That is, if the aperture value can be set to a first aperture value or a second aperture value greater than the first aperture value, the weight map contains data in which the weight of the captured image is greater for the second aperture value than for the first aperture value.

[0050] Next, a weight map based on brightness values ​​will be described. In this embodiment, the weight map is data in which the smaller the brightness value, the greater the weight of the captured image. For example, when the average brightness value of the captured image is used as information regarding the brightness value of the captured image, the smaller the average brightness value, the greater the weight of the captured image. Specifically, the relationship between the average brightness value and the weight of the weight map is stored as a linear function, and a weight corresponding to the average brightness value of the captured image is obtained to generate the weight map.

[0051] For example, a weight of the captured image corresponding to the average luminance value obtained from the relational expression shown in FIG. 12 is used. FIG. 12 is an explanatory diagram of a weight map, with the horizontal axis representing the average luminance value and the vertical axis representing the weight of the captured image. However, the relationship between the average luminance value and the adjustment value of the weight map is not limited to this. While FIG. 12 shows the average luminance value of an undeveloped RAW image, the average luminance value after development may also be used. When calculating the average luminance value of a RAW image, it is preferable to subtract the signal value in the optical black region of the image sensor 102b before calculating the average luminance value. This enables calculation of an average luminance value that is independent of the ISO sensitivity and the image sensor 102b. Furthermore, if the average luminance value is obtained for each region into which the captured image is divided, a weight may be obtained from the average luminance value of each region to generate a weight map. This will be described in detail later in Example 2.

[0052] In addition, even when the average value, median value, variance, or histogram of the luminance values ​​of the captured image is acquired as a statistical quantity related to the signal values ​​of the captured image, the weight map is generated so that the smaller the statistical quantity, the greater the weight of the captured image. For example, when a histogram of the luminance values ​​of the captured image is used, the weight map is generated so that the smaller the center of gravity or peak of the histogram, the greater the weight of the captured image.

[0053] Next, in step S214 of FIG. 9 , the sharpening unit 103c adjusts the weight map based on information about saturated regions in the captured image. In this embodiment, the information about saturated regions is a saturation influence map, and not all RGB may be saturated. The saturation influence map is normalized from 0 to 1 using a set signal value and used to adjust the weight map. Specifically, the normalized weight map is applied (multiplied) to the weight map generated in step S213. This allows the weights of the weight map generated in step S213 to affect the saturated region influence, enabling intensity adjustment only in saturated affected regions where streaks of light and airy disks are more noticeable. Note that application of the saturation influence map is not essential, and the entire captured image may be subject to intensity adjustment. Using the saturation influence map makes it possible to adjust the correction intensity up to the saturated affected region. Note that instead of the saturation influence map, a brightness saturation map may be used, or a brightness saturation map blurred for each image height may be used.

[0054] As an example, the final weight map is shown in Figures 13(A) and (B). In this embodiment, the weight map indicates that the larger the pixel value, the larger the weight of the captured image, and the smaller the pixel value, the smaller the weight of the captured image. Figure 13(A) shows a scene with a bright average luminance value, and Figure 13(B) shows a scene with a dark average luminance value. The weight of the captured image is large in the saturation affected area in Figure 13(B), which is an area where adverse effects are easily noticeable.

[0055] It is preferable that the weight map be generated based on information about the optical performance of the optical system used to capture the captured image. Specifically, if the optical performance is low, the weight of the captured image is increased, and if the optical performance is high, the weight of the captured image is decreased. The information about the optical performance can be calculated based on at least one of the zoom position, aperture diameter, and subject distance when the captured image is captured, and the magnitude and range of the PSF signal value for each image height of the optical system. Note that it is not essential to use the PSF; any information about the optical performance will suffice. For example, an optical transfer function may be used.

[0056] Next, in step S215 of FIG. 9, the sharpening unit (second generating means) 103c uses the weight map generated in step S214 to perform weighted averaging (combining) of the captured image and the blur-sharpened image to generate an intensity-adjusted image 205.

[0057] With the above configuration, this embodiment can provide an image processing system that can maintain the blur correction effect while suppressing adverse effects caused by the aperture value of the optical system and the luminance value of the captured image. [Example]

[0058] Next, an image processing system according to a second embodiment of the present invention will be described. In this embodiment, an average luminance value is obtained for each region into which a captured image is divided, and a weight corresponding to the average luminance value is obtained for each region to generate a weight map. FIG. 14 is a block diagram of an image processing system 300 according to this embodiment. FIG. 15 is an external view of the image processing system 300. The image processing system 300 has a learning device 301, an image capturing device 302, and an image processing device 303. The learning device 301 and the image processing device 303, and the image processing device 303 and the image capturing device 302 are each connected via a wired or wireless network.

[0059] The imaging device 302 has an optical system 321, an imaging element 322, a storage unit 323, a communication unit 324, and a display unit 325. The captured image is transmitted to the image processing device 303 via the communication unit 324. The image processing device 303 receives the captured image via the communication unit 332 and performs blur sharpening using information on the configuration and weights of the machine learning model stored in the storage unit 331. The information on the configuration and weights of the machine learning model is learned by the learning device 301, acquired in advance from the learning device 301, and stored in the storage unit 331. Furthermore, the image processing device 303 has a function of adjusting the intensity of blur sharpening. A blur-sharpened image (model output) in which the blur of the captured image has been sharpened and an intensity-adjusted image in which the intensity has been adjusted are transmitted to the imaging device 302, stored in the storage unit 323, and displayed on the display unit 325.

[0060] The generation of learning data and weight learning (learning phase) performed by the learning device 301 and the sharpening of blur in the captured image using a trained machine learning model (estimation phase) performed by the image processing device 303 are the same as in Example 1, so they will be omitted.

[0061] Next, the synthesis of a captured image and a model output, which is executed by the image processing device 303, will be described with reference to Fig. 16. Fig. 16 is a flowchart of adjusting the sharpening strength. Each step in Fig. 16 is mainly executed by each unit of the image processing device 303.

[0062] First, in step S311, the acquisition unit 333 acquires the imaging state from the captured image. The imaging state includes, but is not limited to, the aperture value of the optical system 321 in the imaging device 302 and the pixel pitch of the image sensor 322. In this embodiment, intensity adjustment is performed when the aperture value of the optical system 321 is F16 or greater and the pixel pitch of the image sensor 322 is 4 μm or greater. In other cases, when generating a weight map, which will be described later, a weight map in which all weights of the captured image are 0 is generated, and intensity adjustment is not performed. Alternatively, the processing of steps S312 to S314 may be omitted without generating a weight map.

[0063] Next, in step S312, the acquisition unit 333 acquires information regarding the luminance values ​​of the captured image. In this embodiment, the average luminance value is acquired for each region into which the captured image is divided, and a weight corresponding to the average luminance value is acquired for each region to generate a weight map. FIGS. 17(A) and (B) are explanatory diagrams of the weight map, where FIG. 17(A) shows the captured image and FIG. 17(B) shows the average luminance value for each region. In FIG. 17(B), 2001 is a region with a low average luminance value, 2002 is a region with a high average luminance value, and 2003 is a region with an intermediate average luminance value.

[0064] 16, the sharpening unit 334 generates a weight map based on information relating to the aperture value of the optical system 321 and the luminance value of the captured image. Note that the method of generating the weight map is the same as in the first embodiment, and therefore its description will be omitted. Next, in step S314, the sharpening unit 334 uses the weight map generated in step S312 to perform a weighted average (combination) of the captured image and the blur-sharpened image (model output) to generate the intensity-adjusted image 205.

[0065] With the above configuration, this embodiment can provide an image processing system that can maintain the blur correction effect while suppressing adverse effects caused by the aperture value of the optical system and the luminance value of the captured image. [Example]

[0066] Next, an image processing system according to a third embodiment of the present invention will be described. Fig. 18 is a block diagram of an image processing system 400 according to this embodiment. Fig. 19 is an external view of the image processing system 400. The image processing system 400 includes a learning device 401, a lens device 402, an imaging device 403, a control device (first device) 404, an image estimation device (second device) 405, and networks 406 and 407.

[0067] The learning device 401 and the image estimation device 405 are each, for example, a server. The control device 404 is a device operated by a user, such as a personal computer or a mobile terminal. The learning device 401 has a storage unit 401a, an acquisition unit 401b, a calculation unit 401c, and an update unit 401d, and learns weights of a machine learning model that sharpens blur from an image captured using a lens device 402 and an imaging device 403. Note that the learning method, i.e., the generation of learning data and weight learning (learning phase) performed by the learning device 401, is omitted because it is the same as in Example 1.

[0068] The imaging device 403 has an imaging element 403a, which photoelectrically converts an optical image formed by the lens device 402 to acquire a captured image. The lens device 402 and the imaging device 403 are detachable and can be combined with each other in multiple types. The control device 404 has a communication unit 404a, a display unit 404b, a storage unit 404c, and an acquisition unit 404d, and controls the processing to be performed on the captured image acquired from the imaging device 403 connected by wire or wirelessly in accordance with a user's operation. Alternatively, the captured image captured by the imaging device 403 may be stored in advance in the storage unit 404c, and the captured image may be read out.

[0069] The image estimation device 405 has a communication unit 405a, an acquisition unit 405b, a storage unit 405c, and a sharpening unit 405d, and is configured to be able to communicate with the control device 404. The image estimation device 405 executes a process of sharpening the blur of a captured image in response to a request from the control device 404 connected via a network 406. The image estimation device 405 acquires information on learned weights from the learning device 401 connected via the network 406 when estimating blur sharpening or in advance, and uses the information to estimate blur sharpening of the captured image. The estimated image after blur sharpening estimation is subjected to sharpening intensity adjustment, and then transmitted again to the control device 404, stored in the storage unit 404c, and displayed on the display unit 404b.

[0070] Next, blur sharpening of a captured image executed by the control device 404 and the image estimation device 405 will be described with reference to Fig. 20. Fig. 20 is a flowchart of model output and sharpening intensity adjustment. Each step in Fig. 20 is mainly executed by each unit of the control device 404 or the image estimation device 405.

[0071] First, in step S401, the acquisition unit 404d of the control device 404 acquires a captured image and a sharpening intensity designated by a user. Then, in step S402, the communication unit (transmission means) 404a transmits the captured image and a request for execution of blur sharpening estimation processing to the image estimation device 405.

[0072] Next, in step S403, the communication unit (receiving means) 405a of the image estimation device 405 receives and acquires the captured image and processing request transmitted from the control device 404. Next, in step S404, the acquisition unit 405b acquires information on learned weights corresponding to the captured image from the storage unit 405c. The weight information is read out in advance from the storage unit 401a and stored in the storage unit 405c. Next, in step S405, the sharpening unit 405d uses a machine learning model to generate, from the captured image, a blur-sharpened image (model output) in which the blur of the captured image has been sharpened. The machine learning model has the configuration shown in FIG. 1, as in the learning period. As in the learning period, a brightness saturation map representing brightness-saturated regions of the captured image is generated and input, and a saturation influence map and model output are generated.

[0073] Next, in step S406, the sharpening unit 405d generates a weight map. The method of generating the weight map is the same as in the first embodiment. The default weight map is adjusted according to the sharpening strength specified by the user. Note that a weight map that has been adjusted in advance may be stored within an adjustable strength range. Next, in step S407, the sharpening unit 405d combines the captured image with the blur-sharpened image (model output) based on the weight map. Next, in step S408, the communication unit 405a transmits the combined image to the control device 404. Next, in step S409, the communication unit 404a of the control device 404 acquires the estimated image transmitted from the image estimation device 405.

[0074] With the above configuration, this embodiment can provide an image processing system that can maintain the blur correction effect while suppressing adverse effects caused by the aperture value of the optical system and the luminance value of the captured image. (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions. The image processing device can be any device that has the image processing function of the present invention, and can be realized in the form of an imaging device or a PC.

[0075] According to each embodiment, it is possible to provide an image processing method, an image processing device, an image processing program, and a storage medium that can maintain the blur correction effect while suppressing the adverse effects caused by the aperture value of the optical system and the brightness value of the captured image.

[0076] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0077] 103 Image processing device 103c sharpening unit (first generating means, second generating means)

Claims

1. A process in which a computer generates information regarding correction of a captured image based on the captured image obtained by imaging using an optical system; generating a weight map by a computer based on information about the aperture value of the optical system and information about the luminance value of the captured image; and generating a first image by a computer based on the captured image, the information related to the correction, and the weight map.

2. 2. The image processing method according to claim 1, wherein in the step of generating the weight map, the weight map is generated based on information about saturated regions in the captured image.

3. 3. The image processing method according to claim 2, wherein the information about the saturated region is information representing a region in which the subject in the saturated region of the captured image has expanded due to blurring that occurs during the capturing of the image.

4. The image processing method according to claim 3, wherein information regarding the brightness values ​​of the captured image is generated based on statistics regarding the brightness values ​​of a second image based on the difference between the brightness values ​​of the captured image and the brightness values ​​in the region.

5. 5. The image processing method according to claim 4, wherein the statistics include at least one of a mean, a median, a variance, and a histogram.

6. 6. The image processing method according to claim 2, wherein the information about the saturated region is generated using a machine learning model.

7. 7. The image processing method according to claim 1, wherein in the step of generating the weight map, the weight map is generated based on information about optical performance of the optical system.

8. 8. The image processing method according to claim 7, wherein the information about the optical performance is calculated based on at least one of a zoom position, an aperture value, and a subject distance at the time of capturing the image, and a point spread function for each image height of the optical system.

9. The image processing method according to any one of claims 1 to 8, characterized in that the information regarding the correction is a corrected image obtained by performing the correction on the captured image, or a correction component which is the difference between the corrected image and the captured image.

10. 9. The image processing method according to claim 1, wherein the correction includes at least one of increasing the resolution of the captured image and converting the shape of defocus blur of the captured image.

11. 11. The image processing method according to claim 1, wherein the weight map weights the captured image so that the smaller the luminance value, the greater the weight.

12. the statistic regarding the luminance values ​​of the second image is an average value of the luminance values ​​in the second image; 6. The image processing method according to claim 5, wherein the weight map weights the captured image more heavily as the average value decreases.

13. the aperture value can be set to a first aperture value or a second aperture value greater than the first aperture value; 13. The image processing method according to claim 1, wherein a weight in the weight map for the second aperture value is greater than a weight in the weight map for the first aperture value.

14. a first generating means for generating information related to correction of a captured image based on the captured image obtained by imaging using an optical system; a second generating means for generating a weight map based on information about the aperture value of the optical system and information about the luminance value of the captured image; and third generating means for generating a first image based on the captured image, the information related to the correction, and the weight map.

15. An image processing system comprising the image processing device according to claim 14 and a control device capable of communicating with the image processing device, the control device has a transmission means for transmitting a request to cause the image processing device to execute processing on the captured image, The image processing system is characterized in that the image processing device has a receiving means for receiving the request, and executes processing on the captured image in response to the request.

16. A program causing a computer to execute the image processing method according to any one of claims 1 to 13.

Images