Image processing method and image processing device

Abstract

To solve a problem that adjusting sharpening intensity when sharpening blur in a captured image may reduce sharpening effect on a focused subject image and increase image processing load.SOLUTION: An image processing method includes: a first step (S202) of generating a first image by sharpening blur in a captured image; and a second step (S211-S216) of controlling intensity adjustment to generate a second image in which the sharpening intensity is adjusted for the first image based on information about the distance to a subject contained in the captured image. In the second step, the intensity adjustment is controlled based on information about sharpening in a defocused region of the first image.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a technique for generating an image in which blur caused by an optical system is sharpened from an image captured using the optical system. [Background technology]

[0002] As a method for sharpening the blur of a subject image caused by aberration in an optical system, Patent Document 1 discloses an image processing method that uses a map relating to the distance to the subject (distance map) to sharpen only the blur of a focused subject image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-44825 Summary of the Invention [Problem to be solved by the invention]

[0004] However, if sharpening is performed without being able to distinguish between focused and unfocused subject images with high accuracy from the distance map, the sharpening effect on focused subject images will be reduced and the image processing load will increase. [Means for solving the problem]

[0005] An image processing method according to one aspect of the present invention includes a first step of generating a first image by sharpening blur in a captured image, and a second step of controlling intensity adjustment to generate a second image in which the intensity of sharpening is adjusted for the first image based on information about the distance to a subject included in the captured image. The control of intensity adjustment in the second step is performed based on information about sharpening in a defocused region of the first image. A program for causing a computer to execute processing according to the image processing method also constitutes another aspect of the present invention.

[0006] Another aspect of the present invention provides an image processing device including: a first means for sharpening a captured image to generate a first image; and a second means for controlling intensity adjustment to generate a second image in which the intensity of the sharpening is adjusted for the first image based on information about the distance to a subject included in the captured image. The second means controls the intensity adjustment based on information about sharpening in a defocused region of the first image. [Effects of the Invention]

[0007] According to the present invention, blur sharpening can be performed well on an in-focus subject image. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing the configuration of a machine learning model in a first embodiment. [Figure 2] FIG. 1 is a block diagram showing the configuration of an image processing system according to a first embodiment. [Figure 3] FIG. 1 is an external view of an image processing system according to a first embodiment. [Figure 4] 1 is a flowchart showing a training process for a machine learning model in Examples 1 to 3. [Figure 5] 10 is a flowchart showing a model output generation process in the first and second embodiments. [Figure 6] 10 is a flowchart showing the sharpening intensity adjustment process in the first embodiment. [Figure 7] 4A to 4C are diagrams showing captured images and defocus maps in Examples 1 to 3. [Figure 8] FIG. 4 is a diagram showing correction components for sharpening in the first embodiment. [Figure 9] 4A to 4C are diagrams showing intensity-adjusted images in Examples 1 to 3. [Figure 10] FIG. 10 is a block diagram showing the configuration of an image processing system according to a second embodiment. [Figure 11] FIG. 10 is an external view of an image processing system according to a second embodiment. [Figure 12]10 is a flowchart showing the sharpening intensity adjustment process in the second embodiment. [Figure 13] FIG. 10 is a diagram showing edge components of a captured image in the second embodiment. [Figure 14] FIG. 10 is a diagram showing division points of optical performance indexes in the second embodiment. [Figure 15] FIG. 10 is a schematic diagram of an imaging state space in which optical performance indicators are arranged in the second embodiment. [Figure 16] FIG. 10 is a diagram showing a decision matrix for process branching in the second embodiment. [Figure 17] FIG. 10 is a block diagram showing the configuration of an image processing system according to a third embodiment. [Figure 18] FIG. 11 is an external view of an image processing system according to a third embodiment. [Figure 19] 11 is a flowchart showing a model output generation process and a sharpening intensity adjustment process in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, before describing the image processing methods of specific embodiments 1 to 3, an overview of the image processing methods of each embodiment will be described. The image processing methods are methods for performing processing related to sharpening of a captured image.

[0010] Specifically, the image processing method generates a sharpened image (first image) by sharpening blur caused by the optical system contained in a captured image acquired by imaging using the optical system using a machine learning model. The image processing method also generates an intensity-adjusted image (second image) by adjusting the sharpening intensity of the sharpened image based on information regarding the distance to a subject contained in the captured image (hereinafter referred to as a distance map). The sharpening intensity adjustment is a process for generating an intensity-adjusted image by combining (weighted averaging, etc.) the captured image and the sharpened image based on a weight map obtained from the distance map. In this case, the image processing method controls the intensity adjustment based on information regarding the sharpening of blur of an out-of-focus subject image in the sharpened image (hereinafter referred to as sharpening information). The control of the intensity adjustment referred to here includes selecting whether or not to perform intensity adjustment and changing the degree of intensity adjustment.

[0011] The blur caused by the optical system includes blur caused by aberration or diffraction of the optical system, and blur caused by the optical action of an optical low-pass filter or deterioration of the pixel aperture of an image sensor.

[0012] The machine learning model is, for example, a neural network, which includes a convolutional neural network (CNN), a generative adversarial network (GAN), a recurrent neural network (RNN), and the like.

[0013] Blur sharpening refers to image processing that restores frequency components of a subject image that have been reduced or lost due to blur. When a machine learning model is trained to sharpen blur without using a distance map, the blur of not only the in-focus subject image in the captured image but also the out-of-focus subject image is sharpened. This is because it is not possible to distinguish between an in-focus subject image and an out-of-focus subject image containing high-frequency components from the captured image alone. By adjusting the strength of the sharpening by taking a weighted average of the captured image and the sharpened image based on a weight map, it is possible to suppress the sharpening of the blur of the out-of-focus subject image.

[0014] In this embodiment, the reason for selecting whether or not to adjust the sharpening intensity and changing the degree of intensity adjustment is that sharpening of out-of-focus subject images is not always performed. For example, if an out-of-focus subject image does not have an edge (high-frequency component), the out-of-focus subject image is not sharpened. If normal intensity adjustment is performed when sharpening of out-of-focus subject images is not performed, the sharpening effect on in-focus subject images will be reduced if the distance map cannot accurately distinguish between in-focus and out-of-focus subject images. Furthermore, the increased image processing using the distance map increases the processing load.

[0015] Therefore, in this embodiment, the intensity of sharpening is adjusted as image processing using a distance map only when sharpening of an unfocused subject image is performed, thereby preventing a decrease in the sharpening effect of an in-focus subject image and an increase in the image processing load.

[0016] In the following description, the stage of learning the weights of the machine learning model is referred to as the learning phase, and the stage of performing blur sharpening using the machine learning model using the learned weights is referred to as the estimation phase. [Example]

[0017] FIG. 2 shows the configuration of an image processing system according to a first embodiment, and FIG. 3 shows the external appearance of the image processing system. In this embodiment, the task performed by the machine learning model is to sharpen blur in a captured image. The blur to be sharpened is blur caused by aberrations and diffraction in the optical system and by an optical low-pass filter. However, this embodiment may also be applied to cases where blur caused by pixel aperture degradation or image shake is to be sharpened.

[0018] The image processing system 100 includes a training device 101 and an image processing device 103, which are interconnected via a wired or wireless network. An imaging device 102, a display device 104, a recording medium 105, and an output device 106 are connected to the image processing device 103 so as to be capable of wired or wireless communication.

[0019] The imaging device 102 generates a captured image by capturing an image of a subject space through an optical system 102a using an imaging element 102b. The captured image is input to an image processing device 103. The captured image contains blurring of the subject image due to aberration and diffraction of the optical system 102a and an optical low-pass filter provided in the imaging element 102b, which attenuates information about the subject (edges, etc.).

[0020] The image processing device 103 sharpens blurred captured images using a machine learning model to generate a sharpened image (first image). The machine learning model is trained by the training device 101. The image processing device 103 acquires information about the machine learning model from the training device 101 in advance and stores it in the storage unit 103a. FIG. 1 shows the machine learning model. However, a machine learning model other than the machine learning model shown in FIG. 1 may also be used.

[0021] Furthermore, the image processing device 103 adjusts the sharpening intensity by taking a weighted average of the captured image and the sharpened image, and generates an intensity-adjusted image (second image). Details of the training and estimation of the machine learning model and the sharpening intensity adjustment will be described later. The sharpened image (second image) generated by the intensity adjustment is stored as an output image in the storage unit 103a or the recording medium 105, and is output to an output device 106 such as a printer. The captured image may be a grayscale image or a color image having multiple color components, and may be an undeveloped RAW image or a developed image.

[0022] Next, the process of training a machine learning model executed by a computer-implemented training device 101 according to a program will be described with reference to the flowchart in Figure 4. The training device 101 has a storage unit 101a, an acquisition unit 101b, a calculation unit 101c, and an update unit 101d, which execute the following steps (processes).

[0023] In step S101, the acquisition unit 101b acquires one or more original images from the storage unit 101a. Since the machine learning model is trained based on the original images, it is desirable that the original images have various frequency components (edges with different orientations and intensities, gradations, flat areas, etc.). The original images may be real images or CG (Computer Graphics) images.

[0024] In step S102, the calculation unit 101c applies blur to the original image to generate a blurred image. The blurred image is an image input to the machine learning model during training and corresponds to the captured image during estimation. The blur applied here is the blur to be sharpened. In this embodiment, blur generated by aberration and diffraction of the optical system 102a and an optical low-pass filter provided in the image sensor 102b is applied. The shape of the blur caused by the aberration and diffraction of the optical system 102a varies depending on the image plane coordinates (image height and azimuth). It also varies depending on the zoom state, aperture state, and focus state of the optical system 102a. When simultaneously training a machine learning model that sharpens all of these blurs, it is preferable to generate multiple blurred images using multiple blurs generated by the optical system 102a. If necessary, noise generated by the image sensor 102b may be applied to the blurred image.

[0025] In step S103, the acquisition unit 101b acquires a correct model output. In this embodiment, since the task is to sharpen blur, the correct model output is an image with less blur than the blurred image. If the original image lacks high-frequency components, an image obtained by reducing the original image may be used as the correct model output. In this case, reduction is also performed when generating a blurred image in step S102. Furthermore, step S103 may be executed at any time after step S101 and before step S104.

[0026] In step S104, the calculation unit 101c uses the machine learning model to generate a model output based on the blurred image. In Fig. 1, a blurred image 111 is input to the machine learning model.

[0027] The machine learning model has multiple layers, and in each layer, a linear sum of the layer's input and weights (the value of each element of the filter corresponds to the weight, and the filter may include a bias) is calculated. The initial value of the weight may be determined using a random number or the like. In this embodiment, the machine learning model is a CNN that uses the convolution result of the input and the filter as the linear sum. However, a machine learning model other than a CNN may also be used. Furthermore, in each layer, nonlinear transformation using an activation function such as a ReLU (Rectified Linear Unit) or a sigmoid function may be performed as necessary. Furthermore, the machine learning model may have a residual block or a skip connection (also called a shortcut connection) as necessary. The machine learning model generates a model output 112 as a result of passing through multiple layers.

[0028] In step S105, the update unit 101d updates the weights of the machine learning model based on the error function. In this embodiment, the error function is a function that indicates the error between the model output 112 and the correct model output. MSE (Mean Squared Error) is used to calculate the error. However, the error function is not limited to this. Backpropagation or the like may be used to update the weights. The error may also be for the residual component. In the case of the residual component, the error between the difference component between the model output 112 and the blurred image 111 and the difference component between the correct model output and the blurred image 111 is used.

[0029] In step S106, the update unit 101d determines whether training of the machine learning model is complete. Completion of training can be determined by, for example, whether the number of iterations of weight update has reached a predetermined number, or whether the amount of change in weight during update is smaller than a predetermined value. If training is not complete, the process returns to step S101, and the acquisition unit 101b acquires one or more new original images. On the other hand, if training is complete, the update unit 101d ends training and stores the configuration of the machine learning model and information on the weights in the storage unit 101a.

[0030] 5, a process (image processing method) for sharpening blur in a captured image using a trained machine learning model, which is executed according to a program by the image processing device 103, will be described. The image processing device 103 has a storage unit 103a, an acquisition unit 103b, and a sharpening unit 103c, which correspond to the first means and the second means, and these execute the following steps.

[0031] In step S201, the acquisition unit 103b acquires a captured image and a machine learning model. Information about the machine learning model and weights is acquired from the storage unit 103a. The machine learning model has the same configuration as that during training, as shown in FIG. 1.

[0032] In step S202 (first process), the sharpening unit 103c uses a machine learning model to generate a sharpened image (first image) from the captured image, in which blurring of the captured image has been sharpened. The first image, which is the output (model output) of the machine learning model, is not limited to a sharpened image, but may be a correction component applied to the captured image for sharpening. A sharpened image can be generated by applying (adding, etc.) the correction component to the captured image. Furthermore, the use of a machine learning model is not essential for generating the first image. For example, a sharpened first image may be obtained by correcting blurring due to aberration from the captured image using processing based on a Wiener filter.

[0033] Using the flowchart in FIG. 6, a process (image processing method) relating to sharpening intensity adjustment, which is executed by the image processing device 103 according to a program, will be described.

[0034] In step S211 (second process from S216), the acquisition unit 103b acquires a distance map. In this embodiment, a defocus map is acquired as the distance map. The defocus map is a map that indicates the defocus blur that has acted on the subject image in the captured image, in other words, a map that indicates the amount of defocus for the subject as a numerical value. The defocus map can be acquired by capturing parallax images that have parallax from each other, or by using DFD (Depth from Defocus), etc. For example, the map may be one in which the focal plane is set to 0, and the direction away from the imaging device is negative and the direction toward the imaging device is positive, as indicated by numerical values.

[0035] Note that a depth map may be acquired as the distance map. A depth map is a map that indicates the distance to a subject included in a captured image, and the distance to the subject is expressed as a numerical value. The depth map can be acquired by a distance measuring device such as a ToF (Time of Flight) sensor. For example, when pixel values ​​range from 0 to 255, the distance to the subject can be expressed by a numerical value that approaches one of 0 and 255 as the distance to the subject increases, and the distance to the subject can be expressed by a numerical value that approaches the other as the distance to the subject decreases. The pixel value range does not have to be 0 to 255. When using a depth map, information about the in-focus subject may be required to identify the in-focus subject, but this information can be acquired from focus information, etc., during image capture.

[0036] The distance map may be any map related to the distance to the subject other than the depth map or defocus map.

[0037] 7(A) shows an example of a captured image 111, and Fig. 7(B) shows a defocus map 116 corresponding to the captured image 111. A focused subject image 114 and an unfocused subject image (background image) 115 exist within the captured image 111, and the numerical values ​​(defocus amounts) of the defocus map 116 differ depending on the defocus amounts of the focused subject image 114 and the unfocused subject image (background image) 115. Note that step S211 may be executed at any time before step S213.

[0038] In step S212, the sharpening unit 103c acquires sharpening information. In this embodiment, information regarding correction components for sharpening is acquired as the sharpening information. The information regarding the correction components may be information indicating the correction components themselves, or may be information that can be converted into correction components. The correction components for sharpening can be acquired from the sharpened image (first image) generated in step S202, in which the blur of the captured image has been sharpened. If the first image is a sharpened image, the correction components can be generated by taking the difference between the captured image and the sharpened image. Also, if the first image is a correction component for sharpening, the correction components are used as is.

[0039] Fig. 8(A) shows correction components for sharpening corresponding to the captured image 111 of Fig. 7(A). In the captured image 111, in a non-focused region (hereinafter referred to as a defocused region) including an unfocused subject image (hereinafter referred to as a defocused subject image), edges, which are high-frequency components, are attenuated due to defocusing, and therefore the correction components are smaller than in a region of a focused subject image (hereinafter referred to as a focused region).

[0040] In step S213, the sharpening unit 103c determines whether the correction component in the defocus region is equal to or greater than a predetermined threshold value (i.e., satisfies a predetermined condition). Depending on the result of this determination, the sharpening unit 103c selects (determines) which of the first image and the second image will be used as the output image. If the correction component is equal to or greater than the threshold value, the process proceeds to step S214; if it is less than the threshold value, the process proceeds to step S216. The fact that the correction component in the defocus region is equal to or greater than the threshold value means that the blur in the defocus region has been sharpened in the sharpened image, and the fact that the correction component in the defocus region is less than the threshold means that the blur in the defocus region has not been sharpened in the sharpened image.

[0041] A specific determination method in step S213 will be described below. First, the defocus map 116 in Fig. 7B is binarized to generate a mask image in which the in-focus region is 0 and the defocus region is 1, as shown in Fig. 8B.

[0042] Next, the product of the correction component shown in FIG. 8(A) and the mask image shown in FIG. 8(B) is calculated, and the correction component data shown in FIG. 8(C) is generated by extracting the correction component in the defocus region.

[0043] Then, the maximum value of the correction component in the defocus area indicated by this correction component data is obtained, and the grand position is compared with the above-mentioned threshold. For example, if the pixel value of the correction component is in the range of 0 to 255, the threshold is set to 50. If the maximum value of the correction component is 100, the process proceeds to step S214, and if the maximum value of the correction component is 20, the process proceeds to step S216. The comparison with the threshold does not have to be with the maximum value of the correction component, and the threshold does not have to be 50.

[0044] Furthermore, since the defocus map may not accurately extract the in-focus area, it is preferable to perform blurring on the mask image to adjust it so that it completely includes the in-focus area. This makes it possible to use only the correction component of the defocus area for determination. Note that, assuming that blurring is performed, the defocus map acquired in step S211 does not necessarily need to acquire the in-focus area at the pixel level. Therefore, it is not necessary to acquire a highly accurate defocus map using a large-scale machine learning model, and a lightweight, high-speed machine learning model or a non-machine learning method may be used to acquire the defocus map.

[0045] On the other hand, if the defocus map used in step S214 described next extracts the in-focus region with high accuracy, it is possible to prevent a decrease in the effect of sharpening the blur of the in-focus object image.

[0046] In step S214, the sharpening unit 103c generates an intensity-adjusted image (second image) that is an image that has been intensity-adjusted by combining the captured image and the sharpened image (first image) based on the weight map, i.e., by performing a weighted average. The weight map in this embodiment is generated based on the defocus map acquired in step S211. The weight map is, for example, a two-dimensional map with the same number of pixels as the captured image, with in-focus regions being 1 and defocus regions being 0. By generating the weight map based on the defocus map, it is possible to generate a weight map that distinguishes between in-focus regions and defocus regions.

[0047] Note that it is not necessary to use the defocus map acquired in step S211 to generate the weight map; a defocus map acquired by another method may be used. The weight map may also be generated based on the depth map and focus information at the time of image capture. It is preferable that the boundary between the in-focus region and the out-of-focus region in the weight map changes continuously. This makes it possible to prevent discontinuous changes in sharpness from occurring in the intensity-adjusted image after weighted averaging.

[0048] 9 shows, from the top, examples of a captured image 111, a sharpened image 112, and an intensity-adjusted image 113. By combining the captured image 111 and the sharpened image 112 by taking a weighted average based on a weight map generated from the defocus map, it is possible to reduce the degree of sharpening of the blur of a defocused subject image while suppressing a decrease in the sharpening effect of the blur of a focused subject image. Note that the method for adjusting the intensity of sharpening is not limited to weighted averaging; for example, a correction component for sharpening may be added to the captured image based on the weight map.

[0049] In step S215, the acquisition unit 103b acquires the intensity-adjusted image (second image) as an output image. That is, the intensity of sharpening is adjusted and the intensity-adjusted image is used as the output image.

[0050] In step S216, the acquiring unit 103b acquires the sharpened image (first image) as an output image. That is, the sharpened image is set as the output image without adjusting the sharpening strength.

[0051] In the present embodiment described above, only when the blur of a defocused subject image has been sharpened is the intensity of sharpening adjusted as image processing using a distance map (i.e., a weight map). This makes it possible to prevent a decrease in the effect of sharpening the blur of a focused subject image and an unnecessary increase in the processing load.

[0052] 6 shows a case where the control for adjusting the intensity of sharpening is performed by selecting whether or not to perform intensity adjustment based on sharpening information, but control for changing the degree of intensity adjustment based on sharpening information may also be performed. Specifically, when the correction component of the defocus area is less than a threshold (no sharpening of the blur of the defocused object image is performed), the degree of intensity adjustment may be lowered (a degree that only slight intensity adjustment is performed) compared to when the correction component is equal to or greater than the threshold. This also makes it possible to prevent a decrease in the effect of sharpening the blur of the focused object image and an unnecessary increase in processing load. This also applies to other embodiments described later. [Example]

[0053] 10 shows the configuration of an image processing system according to the second embodiment, and FIG. 11 shows the appearance of the image processing system. The image processing system 300 of this embodiment includes a training device 301, an imaging device 302, and an image processing device 303.

[0054] The training device 301 and the image processing device 303, and the image processing device 303 and the imaging device 302 are connected to each other via a wired or wireless network.

[0055] The imaging device 302 includes an optical system 321, an imaging element 322, a storage unit 323, a communication unit 324, and a display unit 325. A captured image generated by the imaging device 302 is transmitted to the image processing device 303 via the communication unit 324.

[0056] The image processing device 303 receives a captured image via the communication unit 332, and sharpens blur in the captured image using information about the configuration and weights of the machine learning model stored in the storage unit 331. The machine learning model and weights have been trained by the training device 301, and are acquired from the training device 301 in advance and stored in the storage unit 331.

[0057] Furthermore, the image processing device 303 adjusts the intensity of sharpening to generate an intensity-adjusted image. The blur-sharpened image in which the blur of the captured image has been sharpened and the intensity-adjusted image in which the intensity of sharpening has been adjusted are transmitted to the image capturing device 302 and stored in the storage unit 323 or displayed on the display unit 325.

[0058] The generation of learning data and weight learning (learning phase) performed by the training device 301, and the blurring and sharpening of captured images using a trained machine learning model (estimation phase) performed by the image processing device 303 are the same as in the first embodiment.

[0059] The process of adjusting the sharpening intensity executed by the image processing device 303 will be described with reference to the flowchart of FIG.

[0060] In step S311, the acquisition unit 333 acquires a defocus map in the same manner as in the first embodiment.

[0061] In step S312, the acquisition unit 333 acquires edge components of the captured image. In this embodiment, as the sharpening information, an optical performance index (which will be described later) as information on the optical performance of the optical system 321 used in capturing the captured image, and edge components as information on the edges of the captured image are acquired. The edge components may be acquired not only from the captured image but also from the sharpened image (first image).

[0062] Fig. 13(A) shows edge components acquired from the captured image 111 shown in Fig. 7(A). Although the edges (high frequency components) are attenuated in the defocused region of the captured image 111, the edge components still exist. In this case, if the optical characteristics of the optical system 321 are poor, the blur of the defocused subject image is sharpened.

[0063] In step S313, the acquisition unit 333 acquires information about the imaging state (or imaging conditions) from the captured image. The imaging state is, for example, the zoom position (focal length) of the optical system 321, the aperture diameter, the subject distance (z, f, d), and the pixel pitch of the image sensor 322.

[0064] In step S314, the acquisition unit 333 acquires an optical performance index as information relating to the optical performance of the optical system 321 from the storage unit 331 in which this information has been stored in advance, based on the imaging state acquired in step S313.

[0065] In this embodiment, the magnitude (peak value) of the point spread function (PSF) is used as the optical performance index. Note that the optical performance index may be any index that reflects optical performance, and therefore a modulation transfer function (MTF) or an optical transfer function (OTF) may also be used. When an MTF is used, the optical performance index is, for example, the average value of the MTF from half the Nyquist frequency to the Nyquist frequency. In this case, the calculation interval for the average value is not particularly limited. The peak value refers to the maximum signal value of the PSF. Furthermore, since the peak value of the PSF depends on the pixel pitch of the image capture device 302, peak values ​​corresponding to multiple pixel pitches are stored, and intermediate values ​​are generated by interpolation. The information related to optical performance (optical performance index) may be information indicating the optical performance itself, or may be information that can be converted into optical performance.

[0066] FIG. 14 shows the image circle 119 of the optical system 321 and the effective pixel area 120 of the image sensor 322. The white circles 122 in the figure indicate optical performance indices at multiple (10 in this case) image heights from on-axis to the most off-axis, which are stored in the storage unit 331. In this embodiment, the acquisition unit 333 acquires these 10 optical performance indices. Note that the number of image heights for which optical performance indices are acquired may be other than 10. Furthermore, if the image sensor 322 has an effective pixel area 121 that is smaller than the effective pixel area 120 inscribed in the image circle 119, it is sufficient to acquire the necessary number of optical performance indices according to the size of the effective pixel area 121.

[0067] In order to reduce the number of optical performance indices (number of data), optical performance indices for discretely selected representative imaging states are stored in the storage unit 331. If the actual imaging state acquired in step S313 corresponds to the representative imaging state, the acquisition unit 333 selects an optical performance index for the representative imaging state. If the actual imaging state does not correspond to the representative imaging state, the acquisition unit 333 selects an optical performance index for a representative imaging state that is as close as possible to the actual imaging state, and corrects the optical performance index to optimize it for the actual imaging state, thereby creating an optical performance index for the actual imaging state.

[0068] 15 schematically shows optical performance indices for discretely selected representative imaging states stored in the storage unit 331. As described above, the optical performance indices stored in the storage unit 331 are discretely arranged in an imaging state space with the three imaging states, namely, zoom position (state A), aperture diameter (state B), and subject distance (state C), as axes. The optical performance indices are stored in the storage unit 331 for the representative imaging states indicated by the respective black dots in the imaging state space.

[0069] The imaging state may include types other than zoom position, aperture diameter, and subject distance, or the types of imaging states may be four or more, and optical performance indices may be discretely arranged in a four-dimensional or more imaging state space. Also, optical performance indices may be arranged for one or two of zoom position, aperture diameter, and subject distance.

[0070] A method for correcting (creating) the optical performance index will be specifically described. In Fig. 15, the imaging state indicated by the large white circle is the actual imaging state acquired in step S313. This actual imaging state deviates from the representative imaging state. In this case, the optical performance index is corrected for the representative imaging state that is positioned closest to the actual imaging state.

[0071] First, the acquisition unit 333 calculates the distance between the actual imaging state in the imaging state space and a plurality of representative imaging states around it. Then, the representative imaging state (shown by a small white circle in the figure) with the shortest distance is selected from the calculated distances. The difference between this closest representative imaging state and the actual imaging state (hereinafter referred to as the state difference amount) is the smallest compared to the other state difference amounts.

[0072] Next, the acquisition unit 333 calculates the state difference amounts ΔA, ΔB, and ΔC between the selected representative imaging state and the actual imaging state. Then, based on these state difference amounts ΔA, ΔB, and ΔC, the acquisition unit 333 calculates state correction coefficients. Furthermore, the acquisition unit 333 corrects the optical performance index for the selected representative imaging state using the state correction coefficients. This makes it possible to create an optical performance index for the actual imaging state.

[0073] Alternatively, as another method, optical performance indices for a plurality of representative imaging states near the actual imaging state may be obtained, and the plurality of optical performance indices may be interpolated according to the state difference amount to create an optical performance index for the actual imaging state.

[0074] In step S315, the sharpening unit 334, which corresponds to the first means and the second means, determines whether the optical performance index and edge components serving as sharpening information in the defocus region are equal to or greater than predetermined thresholds. Specifically, the sharpening unit 334 generates a mask image shown in FIG. 13B from the defocus map, as in the first embodiment. Next, the sharpening unit 334 calculates the product of the edge components shown in FIG. 13A and the mask image shown in FIG. 13B to generate edge component data shown in FIG. 13C, which extracts the edge components in the defocus region. The edge components in the defocus region in this edge component data are then compared with the optical performance index for each image height acquired in step S314, against a threshold. The comparison with the threshold may be performed for each pixel in the defocus region or for each grid obtained by dividing the defocus region. When comparing for each pixel, for example, the optical performance index for the image heights of the top 10 pixels with the largest edge components is acquired and compared with the threshold. When comparing for each grid, for example, the maximum value of the edge component within the grid is obtained, and the optical performance index at the image height of the maximum edge component is obtained and compared with the threshold value.

[0075] The method of comparison with the threshold is not limited to these, and other comparison methods may be adopted. Furthermore, the comparison with the threshold does not need to be performed over the entire defocus area. If it is determined in the determination described below that there is a high possibility that the blur of the non-focused subject image will be sharpened, it is not necessary to perform the determination over the remaining defocus area. This makes it possible to reduce the determination processing time.

[0076] Next, the flow of processing according to the determination result in step S314 will be described with reference to Fig. 16. The larger the edge component in the defocused region, the more likely it is that the blur in the defocused subject image will be sharpened. Also, the higher the optical performance index, the less likely it is that the blur in the defocused subject image will be sharpened. This is because when the optical performance index is high, the blur occurring in the captured image is small and the amount of sharpening is small.

[0077] For this reason, in this embodiment, if the optical performance index is less than the threshold and the edge component is equal to or greater than the threshold, it is highly likely that the blur in the defocused subject image will be sharpened, so the process proceeds to step S316. If the optical performance index is equal to or greater than the threshold and the edge component is less than the threshold, it is unlikely that the blur in the defocused subject image will be sharpened, so the process proceeds to step S318. If the optical performance index is less than the threshold and the edge component is also less than the threshold, it is highly likely that the blur in the defocused subject image will be sharpened from the perspective of the optical performance index, but there are few edges to be sharpened in the defocused subject image, so the process proceeds to step S318. If the optical performance index is equal to or greater than the threshold and the edge component is also greater than the threshold, it is unlikely that the blur in the defocused subject image will be sharpened, so the process proceeds to step S318.

[0078] The above-described process flow according to the determination result is an example, and other process flows may be used. In addition, in this embodiment, both the optical performance index and the edge component are used for the determination, but it is sufficient to use at least one of the optical performance index and the edge component.

[0079] In step S316, the sharpening unit 334 generates an intensity-adjusted image (second image) by performing a weighted average of the captured image and the sharpened image (first image) based on the weight map. By combining the captured image and the sharpened image based on the weight map, it is possible to reduce the degree of sharpening of the blur of the defocused subject image while suppressing a decrease in the sharpening effect of the blur of the focused subject image. The method of generating the weight map is the same as in the first embodiment.

[0080] In step S317, the acquisition unit 333 acquires the second image as an output image. That is, the intensity of sharpening is adjusted and the intensity-adjusted image is used as the output image.

[0081] In step S318, the acquisition unit 333 acquires the sharpened image (first image) as the output image. That is, the sharpened image is set as the output image without adjusting the sharpening strength.

[0082] In the present embodiment described above, similarly to embodiment 1, only when the blur of a defocused subject image has been sharpened is the intensity of sharpening adjusted as image processing using a distance map, thereby making it possible to prevent a decrease in the effect of sharpening the blur of a focused subject image and an unnecessary increase in the processing load. [Example]

[0083] Fig. 17 shows the configuration of an image processing system according to a third embodiment, and Fig. 18 shows the external appearance of the image processing system. An image processing system 400 according to this embodiment includes a learning device 401, a lens device 402, an imaging device 403, a control device (first device) 404, an image estimation device (second device) 405, and networks 406 and 407. The learning device 401 and the image estimation device 405 are configured, for example, by a server.

[0084] The control device 404 is a device that can be operated by a user, such as a personal computer or a mobile terminal. The learning device 401 has a storage unit 401a, an acquisition unit 401b, a calculation unit 401c, and an update unit 401d, and learns weights of a machine learning model for sharpening blur in a captured image acquired by imaging performed by a lens device 402 and an imaging device 403. The learning method is the same as in the first embodiment.

[0085] The imaging device 403 has an imaging element 403a, which obtains a captured image by photoelectrically converting an optical image formed by the lens device 402. The lens device 402 and the imaging device 403 are detachable from each other, and each can be combined with a plurality of models.

[0086] The control device 404 has a communication unit 404a, a display unit 404b, a storage unit 404c, and an acquisition unit 404d, and controls, in accordance with a user's operation, processing to be performed on captured images acquired from the wired or wirelessly connected imaging device 403. Note that the control device 404 may store the captured images acquired from the imaging device 403 in advance in the storage unit 404c, and read the captured images from the storage unit 404c when executing processing.

[0087] The image estimation device 405 has a communication unit 405a, an acquisition unit 405b, a storage unit 405c, and a sharpening unit 405d corresponding to the first means and the second means. The image estimation device 405 executes a process of sharpening blurred images in accordance with a request from a control device 404 connected via a network 406. The image estimation device 405 acquires learned weight information from a learning device 401 connected via the network 406 when estimating an estimated image (described later) or in advance, and uses the information to estimate the estimated image. After the sharpening strength is adjusted, the estimated image is transmitted again to the control device 404 and stored in a storage unit 404c or displayed on a display unit 404b.

[0088] The generation of learning data and weight learning (learning phase) performed by the learning device 401 are the same as those in the first embodiment.

[0089] The flowchart in FIG. 19 shows the process of sharpening blur in a captured image, which is executed by the control device 404 and the image estimation device 405 according to a program.

[0090] In step S401, the acquisition unit 404d acquires a captured image.

[0091] In step S402, the communication unit 404a transmits to the image estimation device 405 a request for the execution of the captured image and sharpening estimation processing.

[0092] In step S403, the communication unit 405a receives (acquires) the captured image and the request for execution of the estimation process.

[0093] In step S404, the acquisition unit 405b acquires information on learned weights corresponding to the captured image from the storage unit 405c. The weight information has been read out from the storage unit 401a and stored in the storage unit 405c in advance.

[0094] In step S405, the sharpening unit 405d generates an estimated image (first image) in which blur is sharpened from the captured image using a machine learning model. The machine learning model has the same configuration as in training, as shown in FIG.

[0095] In step S406, the acquisition unit 405b acquires a defocus map. The method for acquiring the defocus map is the same as in the first embodiment.

[0096] In step S407, the acquisition unit 405b acquires correction components for sharpening (sharpening information). The method of acquiring the correction components for sharpening is the same as in the first embodiment.

[0097] In step S408, the sharpening unit 405d determines whether the correction component for sharpening in the defocus region is equal to or greater than a threshold value as a predetermined value. The determination method is the same as in Example 1. If the correction component is equal to or greater than the threshold value, the process proceeds to step S409, and if it is equal to or less than the threshold value, the process proceeds to step S411.

[0098] In step S409, the sharpening unit 405d synthesizes the captured image and the estimated image (first image: an image equivalent to the sharpened image in the first embodiment) based on the weight map to generate a synthesized image (second image: an image equivalent to the intensity-adjusted image in the first embodiment). The synthesis method is the same as in the first embodiment, in which the captured image and the sharpened image are synthesized to generate the intensity-adjusted image.

[0099] In step S410, the communication unit 405a transmits the composite image (second image) to the control device 404. The sharpening strength is adjusted and the composite image is used as the output image.

[0100] In step S411, the communication unit 405a transmits the estimated image to the control device 404. That is, the estimated image is set as the output image without adjusting the sharpening intensity.

[0101] In step S412, the communication unit 404a acquires the transmitted composite image or estimated image.

[0102] In the present embodiment described above, only when the blur of a defocused subject image has been sharpened is the image processing using the distance map performed to combine the captured image and the estimated image (adjust the sharpening strength), as in the first embodiment. This makes it possible to prevent a decrease in the sharpening effect of the blur of a focused subject image and an unnecessary increase in the processing load.

[0103] The above embodiments include the following methods. (Method 1) a first step of generating a first image by sharpening blur in a captured image; a second step of controlling intensity adjustment to generate a second image in which the intensity of the sharpening is adjusted for the first image based on information about the distance to the subject included in the captured image, An image processing method, characterized in that in the second step, the control related to the intensity adjustment is performed based on information related to the sharpening in a defocus region of the first image. (Method 2) The image processing method described in Method 1, characterized in that the control regarding the intensity adjustment is a process of generating the second image by combining the captured image and the first image according to a weight based on information about the distance. (Method 3) 3. The image processing method according to method 1 or 2, wherein in the second step, a selection is made as to whether or not to perform the intensity adjustment. (Method 4) 4. The image processing method according to method 3, wherein in the second step, the selection is made depending on whether the information relating to sharpening satisfies a predetermined condition. (Method 5) The image processing method described in Method 3, characterized in that in the second step, the intensity adjustment is performed if the information regarding sharpening indicates that the sharpening has been performed in the defocused area, and the intensity adjustment is not performed if the information regarding sharpening indicates that the sharpening has not been performed in the defocused area. (Method 6) 6. The image processing method according to any one of Methods 1 to 5, wherein the control relating to the intensity adjustment is a process of changing the degree of the intensity adjustment. (Method 7) 7. The image processing method according to method 6, wherein in the second step, the degree is changed depending on whether the information relating to sharpening satisfies a predetermined condition. (Method 8) 8. The image processing method according to method 6 or 7, characterized in that in the second step, if the information regarding sharpening indicates that the sharpening has not been performed in the defocus area, the degree is lowered compared to when the information indicates that the sharpening has been performed. (Method 9) 9. The image processing method according to any one of methods 1 to 8, wherein the information regarding sharpening is information regarding a correction component that is applied to the captured image for the sharpening. (Method 10) An image processing method according to any one of methods 1 to 8, characterized in that the information regarding sharpening is information regarding the optical performance of an optical system used in capturing the captured image. (Method 11) 11. The image processing method according to claim 10, wherein the optical performance is at least one of the focal length, aperture diameter, and subject distance of the optical system at the time of capturing the image. (Method 12) 12. The image processing method according to Method 10 or 11, wherein the information regarding optical performance is information for each image height. (Method 13) 13. The image processing method according to any one of methods 1 to 12, wherein the information regarding sharpening is information regarding edges of the captured image or the first image.

[0104] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0105] The embodiments described above are merely representative examples, and various modifications and changes can be made to each embodiment when implementing the present invention. [Explanation of symbols]

[0106] 102 Imaging device 103 Image processing device 103b Acquisition Department 103c Sharpening section

Claims

1. a first step of generating a first image by sharpening blur in a captured image; a second step of controlling intensity adjustment to generate a second image in which the intensity of the sharpening is adjusted for the first image based on information about the distance to the subject included in the captured image, An image processing method, characterized in that in the second step, control relating to the intensity adjustment is performed based on information relating to the sharpening in a defocus region of the first image.

2. The image processing method according to claim 1, characterized in that the control regarding the intensity adjustment is a process of generating the second image by combining the captured image and the first image according to a weight based on information regarding the distance.

3. 2. The image processing method according to claim 1, wherein in the second step, a selection is made as to whether or not the intensity adjustment is to be performed.

4. 4. The image processing method according to claim 3, wherein in the second step, the selection is made depending on whether the information relating to sharpening satisfies a predetermined condition.

5. 4. The image processing method according to claim 3, wherein in the second step, the intensity adjustment is performed if the information regarding sharpening indicates that the sharpening has been performed in the defocused region, and the intensity adjustment is not performed if the information regarding sharpening indicates that the sharpening has not been performed in the defocused region.

6. 2. The image processing method according to claim 1, wherein the control relating to the intensity adjustment is a process of changing the degree of the intensity adjustment.

7. 7. The image processing method according to claim 6, wherein in the second step, the degree of sharpening is changed depending on whether or not the information relating to the sharpening satisfies a predetermined condition.

8. 7. The image processing method according to claim 6, wherein in the second step, if the information regarding sharpening indicates that the sharpening has not been performed in the defocus area, the degree is lowered compared to when the information indicates that the sharpening has been performed.

9. 2. The image processing method according to claim 1, wherein the information regarding sharpening is information regarding a correction component that is applied to the captured image for the sharpening.

10. 2. The image processing method according to claim 1, wherein the information relating to sharpening is information relating to optical performance of an optical system used in capturing the captured image.

11. 11. The image processing method according to claim 10, wherein the optical performance is at least one of a focal length of the optical system, an aperture diameter, and a subject distance at the time of capturing the image.

12. 11. The image processing method according to claim 10, wherein the information regarding the optical performance is information for each image height.

13. 2. The image processing method according to claim 1, wherein the information regarding sharpening is information regarding edges of the captured image or the first image.

14. A program causing a computer to execute a process according to the image processing method of any one of claims 1 to 13.

15. a first means for sharpening a captured image to generate a first image; and second means for controlling intensity adjustment to generate a second image in which the intensity of the sharpening is adjusted for the first image based on information about the distance to the subject included in the captured image, The image processing device is characterized in that the second means controls the intensity adjustment based on information related to the sharpening in a defocus area of ​​the first image.

16. 16. The image processing apparatus according to claim 15, wherein the second means selects whether or not to perform the intensity adjustment as the control related to the intensity adjustment.

17. 16. The image processing apparatus according to claim 15, wherein the second means changes the degree of the intensity adjustment as the control relating to the intensity adjustment.

Images